oversight

National Accrediting Commission of Cosmetology Arts and Sciences' Accreditation Standards for Student Achievement and Measures of Program Length.

Published by the Department of Education, Office of Inspector General on 2002-12-19.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                                  U.8. Department of Education
                                              Office of Inspector General

                                               501 I Street, Suite 9-200 

                                            Sacramento, California 95814 

                                      Phone (916) 930-2388 • Fax (916) 930-2390 




                                                   December 19, 2002


MEMORANDUM

TO: 	                      Sally Stroup
                           Assistant Secretary
                           Office of Postsecondary EdU~                                  .     .

FROM: 	                    GIOriaPilotti~ ~
                           Regional Inspector General for Audit

SUBJECT: 	                 MANAGEMENT INFORMATION REPORT
                           National Accrediting Commission ofCosmetology Arts and Sciences'
                           Accreditation Standards for Student Achievement and Measures of
                           Program·Length
                           Control No. ED-OIG/A09-C0019

Attached is oursubject report providing infonnation on the results of our review of the National
Accrediting Commission of Cosmetology Arts and Sciences' standards for student achievement
and program length as required by the Higher Education Act, Section 496(a)(5). The report does
not contain recommendations for Department action, but it does provide infonnation that you
may find useful in making management decisions related to Federal recognition of accrediting
agencIes.

If you have any questions or wish to discuss the contents of this report, please contact me at
(916) 930-2399. Please refer to the above control number in all correspondence relating to this
report.


Attachment


Electronic cc: John Barth, Accreditation and State Liaison Staff Director, OPE
               Carol Griffiths, Accrediting Agency Evaluation Chief, OPE
               David Bergeron, Audit Liaison Officer, OPE




                                       501 I Street, Suite 9-200, SACRAMENTO, CA 95814
          Our mission is to promote the efficiency. effectiveness. and integrity ofthe Department's programs and operations
                                 U.S. Department of Education 

                                            Office of Inspector General


                                        501 I Street, Suite 9-200
                                     Sacramento, California 95814
                           Phone (916) 930-2388 • Fax (916) 930-2390


                                                   December 19, 2002



                                                                                                          ED-OIG/A09-C0019


Mark Gross, Chief Executive Officer
National Accrediting Commission of
Cosmetology Arts and Sciences
4401 Ford Avenue, Suite 1300
Alexandria, Virginia 22302

Dear Mr. Gross:

This is the Office of Inspector General’s Final Management Information Report, entitled
National Accrediting Commission of Cosmetology Arts and Sciences’ Accreditation Standards
for Student Achievement and Measures of Program Length.

The purpose of our review of the National Accrediting Commission of Cosmetology Arts and
Sciences (NACCAS) was to (1) identify NACCAS’ standards for success with respect to student
achievement and measures of program length as required by the Higher Education Act (HEA),
Section 496(a)(5), and (2) evaluate NACCAS’ management controls for ensuring that institutions
adhere to its standards for student achievement and measures of program length and that
consistent enforcement action is taken when institutions are not in compliance with the
standards. This management information report, describing NACCAS’ standards and its
monitoring and enforcement policies and procedures, is being provided to assist the
U.S. Department of Education (Department) in its oversight of accrediting agencies. We also
provide suggestions for NACCAS to consider for enhancing its standards and management
controls.

We received written comments from NACCAS on a draft of this report. Other than some
suggested changes, which we made in this final report, NACCAS concurred that the information
contained in the report is accurately presented. NACCAS described its plans to implement or
study several of our suggestions, but it expressed concern with our suggestion to utilize and
reconcile conflicting findings of on-site evaluation teams participating in the agency’s
double-teaming process. NACCAS’ comments and our response are summarized in the report.
The full text of NACCAS’ comments is included as an attachment to the report.




          Our mission is to ensure equal access to education and to promote educational excellence throughout the Nation.
ED-OIG/A09-C0019	                                                                      Page 2 of 18


                                   REVIEW RESULTS 

The Secretary of the Department recognizes NACCAS as a national accrediting agency. To
achieve this recognition, the HEA requires accrediting agencies to, among other requirements,
establish standards for student achievement and measures of program length. NACCAS has
established standards for student achievement that include minimum completion, licensure, and
placement rates. For program length, NACCAS’ standards require institutions to adhere to state
licensure requirements or justify the basis for exceeding those requirements. We suggested that
NACCAS could strengthen its standards by (1) conducting a periodic evaluation of its minimum
rates for completion, licensure, and placement, (2) revising the definition of exempt students in
its completion rate formula, (3) revising its definition of students ineligible for employment or
establishing a minimum placement rate for all students who complete the programs, and
(4) defining the type and period of employment for an acceptable placement.

NACCAS has policies and procedures in place for monitoring adherence and enforcing its
standards for student achievement and program length. Due to the limited documentation
available in NACCAS’ files, we were unable to confirm that the site visit teams followed
NACCAS’ procedures for site visit reviews. We provided suggestions that NACCAS might
consider for enhancing its management controls.

STANDARDS FOR STUDENT ACHIEVEMENT
AND MEASURES OF PROGRAM LENGTH

Section 496 (a)(5) of the HEA requires accrediting agencies to establish accreditation standards
that assess the institution’s—

       (A) 	success with respect to student achievement in relation to the institution’s
            mission, including, as appropriate, consideration of course completion, State
            licensing examinations, and job placement rates;... [and]
       (H) 	measures of program length and the objectives of the degrees or credentials
            offered;....

The regulations that address accrediting agencies’ standards for student achievement and
measures of program length mirror the statutory language.

NACCAS’ Standard for Student Achievement

NACCAS addresses student achievement in criteria 4 and 5 of its standard on “Educational
Objectives and Institutional Evaluation.”

       4. The institution meets or exceeds the following outcomes:

               Completion rate - 50% 

               Pass rate on state licensing examination - 70% 

               Placement rate of eligible graduates - 60% 

ED-OIG/A09-C0019	                                                                              Page 3 of 18


        5. The school must document that it evaluates the effectiveness of its programs
        using but not limited to
               a. 	 pass rates on examination for licensure to practice;
               b. 	 rates of employment and/or placement;
               c. 	 institutional completion rates;
               d. 	 comments and suggestions regularly solicited from students,
                    graduates, and employers of graduates.

According to a NACCAS official, the minimum rates were established in the early 1990s using
data collected from its accredited institutions and the minimum outcomes proposed by the former
State Postsecondary Review Entities.

The formulas used to calculate each institution’s actual completion, licensure, and placement
rates are incorporated into NACCAS’ database.1 The database contains the outcomes and other
data submitted by institutions in their Annual Reports.

                                           Completion Rate
                              Total number of students who completed the course
                                within 150 percent of normal completion time

                             Total number of students who were scheduled to
                          complete the course minus the number of exempt students
                                     who did not complete the course

The number of exempt students used in the formula is limited to 25 percent of the total number
of students who actually completed the course. Exempt students are students who (1) transferred
to another course within the school, (2) did not complete the course due to health reasons,
(3) had a military or out-of-state transfer, (4) failed to maintain satisfactory attendance or
academic progress, or (5) completed the program outside 150 percent of the normal completion
time.

NACCAS’ formula overstates completion rates since students who failed to maintain satisfactory
attendance and academic progress, and students who completed the program outside 150 percent
of the normal completion time are excluded from the formula’s denominator. These students had
the opportunity to complete the program within 150 percent of the normal completion time.

                                            Licensure Rate
                       Total number of students who passed the licensing examinations

                        Total number of students who took the licensing examinations




1
  As discussed later in the report, we found that the completion formula had been incorrectly entered in
the database used to calculate rates for the 2000-2001 Annual Report data.
ED-OIG/A09-C0019	                                                                          Page 4 of 18


                                         Placement Rate

                           Total number of students who completed the course 

                 within 150 percent of the normal completion time and found employment 


                Total number of students who completed the course within 150 percent of
                   normal completion time minus students ineligible for employment


Students ineligible for employment are those who moved out of state, died, or failed to take the
state licensing examination.

NACCAS’ placement rate overstates the percentage of students who completed the program and
obtained appropriate employment since students who failed to take the state licensing
examination are excluded from the formula’s denominator.2 Also, NACCAS has not defined the
type and period of employment for an acceptable placement. Thus, employment of one-day
duration could be considered a placement for purposes of calculating the placement rate.

NACCAS’ Standard for Measures of Program Length

NACCAS requires institutions to meet State minimum requirements for program length and
specifies a model to be used to assess program lengths in excess of the State minimum.
Specifically, NACCAS’ standard on “Curriculum and Criteria” states—

       Courses provide instruction in theory and are designed to develop practical skills
       required for licensure and employment, or (in the absence of licensure
       requirements) as required by industry standards for employment.

       In the absence of state minimum requirements or if the school wants to exceed the
       state minimums, the institution in developing the course must include the four
       points in the following model to assess effective program length for quality
       education:

               a. 	 State minimum standards;
               b. 	 Industrial needs as determined by the institution’s Advisory
                    Committee;
               c. 	 Public safety and sanitation requirements established by the State
                    Board of Cosmetology or other state and local regulatory agencies; and
               d. 	 Special academic needs of the students served, and in accordance with
                    the mission of the institution.

NACCAS’ standard on “Participation in Student Financial Assistance Programs” limits the
length of programs offered at institutions participating in HEA, Title IV programs to 150 percent
of the state minimum requirement.


2
 In its 1999-2000 Annual Report Verification Study, NACCAS recognized that its placement rate
formula needed to be changed to provide useful information to consumers and the NACCAS Board of
Commissioners. We discuss the Annual Report Verification Study later in the report.
ED-OIG/A09-C0019                                                                         Page 5 of 18


NACCAS requires a program in credit hours to satisfy the following requirements. For
programs measured on a non-standard term or semester basis, one credit hour equals at least
30 clock hours of theory, practice/demonstration, and/or clinical experience. For programs
measured on a quarter basis, one credit hour equals at least 20 clock hours of theory,
practice/demonstration, and/or clinical experience. If a State mandates a specific clock-to-credit
hour formula, the institution is required to use that formula. NACCAS does not require outside
preparation time for each credit hour.

Systematic Program of Review

The regulations at 34 C.F.R. § 602.21 address the requirements for an accrediting agency’s
systematic review of its standards. Paragraph (a) states “[t]he agency must maintain a systematic
program of review that demonstrates that its standards are adequate to evaluate the quality of the
education or training provided by the institutions and programs it accredits and relevant to the
educational or training needs of students.” Paragraph (b) lists the following required attributes of
the review—

       [T]he agency must ensure that its program of review—
          (1) Is comprehensive;
          (2) Occurs at a regular, yet reasonable, intervals or on an ongoing basis;
          (3) Examines each of the agency’s standards and the standards as a whole; and
          (4) Involves all of the agency’s relevant constituencies in the review and
              affords them a meaningful opportunity to provide input into the review.

NACCAS established an Advisory Committee on Validity and Reliability to assess the standards
and criteria over a five-year cycle—two of the ten standards every year. The Advisory
Committee’s procedures state that the Committee will conduct a preliminary review of the two
standards under review and send surveys to schools, site evaluators, practicing cosmetologist,
cosmetology educators, graduates, employers, and government regulatory agencies, including the
Department. The Advisory Committee also reviews the six criteria that were most frequently
cited in findings by site evaluation teams during the prior year.

In 2000, the Advisory Committee reviewed NACCAS’standard on “Educational Objectives and
Institutional Evaluation,” which addresses student achievement. As a result of the review, the
Advisory Committee recommended and the Commission adopted revisions to the criteria under
the standard. For example, the introductory wording for criteria 5 was changed from
“[t]he school assess[es] the achievements of its programs and students on the basis of....” to
“[t]he school must document that it evaluates the effectiveness of its programs using but not
limited to the following items....” Criteria 4, which contains the minimum rates for completion,
licensure, and placements, was not changed.
ED-OIG/A09-C0019                                                                       Page 6 of 18


Suggestions for Enhancing the
Standard for Student Achievement

To enhance its established standards for student achievement, we suggest that NACCAS
consider—

   ƒ   Requiring its Advisory Committee on Validity and Reliability to conduct quantitative
       analyses of institutions’ past completion, licensure, and placement rates as part of its
       evaluation to ensure that the minimum levels provide an adequate and relevant measure
       of educational quality.

   ƒ   Revising the completion rate formula by removing from the list of exempt students those
       students who failed to maintain satisfactory attendance and academic progress, and
       students who completed the program outside 150 percent of the course.

   ƒ   Either revising the placement rate formula by removing from the list of ineligible
       students those students who failed to take the state licensing examination or including a
       separate minimum rate that measures placement for all students who complete the
       program.

   ƒ   Defining the type and period of employment for placements reported in Annual Reports.

Based on our review, we have no suggested enhancements for NACCAS’ standard for measures
of program length.

NACCAS Comments

In its comments on the draft report, NACCAS stated that its Advisory Committee on Standards
reviewed the student achievement standard in 2000 and did not recommend changes in the
quantitative minimum rates for completion, licensure, and placement. NACCAS stated that the
Advisory Committee is scheduled to again review the student achievement standard in 2005.
NACCAS plans to discuss the OIG suggested revision to the completion and placement rate
formulas at its next Commission policy meeting. NACCAS stated that it already defined
employment for purposes of the placement rate: “Employment, to count, must be related to the
program from which the student graduated.” NACCAS plans to assess the availability of
information on length of employment by including questions on employment periods in future
Annual Report Verification Studies. NACCAS also provided three proposals for consideration
by the Department and State agencies that could result in more accurate employment data for the
Department, accrediting agencies, or institutions.

OIG Response

We revised our suggestion on the assessment of minimum rates to focus on the importance of
including quantitative analyses as part of the Advisory Committee’s review of the student
achievement standard. The employment definition, provided in NACCAS’ comments, needs to
be included in the published standards and instructions for the Annual Report.
ED-OIG/A09-C0019                                                                            Page 7 of 18


MANAGEMENT CONTROLS FOR ENSURING
ADHERENCE TO STANDARDS

The regulations at 34 C.F.R. § 602.18 require an accrediting agency to “consistently apply and
enforce its standards to ensure that the education or training offered by an institution or
program... is of sufficient quality to achieve its stated objective for the duration of any
accreditation or preaccreditation period granted by the agency.” In addition, 34 C.F.R. § 602.19
requires the agency to (a) “reevaluate, at regularly established intervals, the institutions or
programs it has accredited or preaccredited... [and] (b)... monitor institutions or programs
throughout their accreditation or preaccreditation period to ensure that they remain in compliance
with the agency's standards.”

NACCAS’ Monitoring Activities and Procedures

NACCAS has policies and procedures that address its effort to ensure adherence to the standards
for student achievement and program length, which include Annual Reports submitted by
institutions, an Annual Report Verification Study, institutional self-studies, and site visit reviews.
Due to the limited documentation available in NACCAS’ files, we were unable to confirm that
the site visit teams followed NACCAS’ procedures for site visit reviews. Also, as discussed
below, we identified an error in the database formula used to compute completion rates from the
institutional Annual Report data.

Annual Reports. Accredited institutions are required to submit an Annual Report, which
includes student outcomes data. The Annual Report also includes program information, which
NACCAS compares to the prior year data to identify changes in the programs offered and
program length.

Student outcome data, which is recorded into a database, is used by NACCAS to calculate each
institution’s completion, licensure, and placement rates. Our review found that the
10 institutions in our sample submitted 2000-2001 Annual Reports that contained the required
student outcome data and information on program length. Nine of the ten institutions met
NACCAS’ minimum completion, licensure, and placement rates.

From the 2000-2001 Annual Report data, NACCAS identified 99 institutions that did not meet
one or more of the minimum rates.3 NACCAS requires institutions, which do not meet the
minimum rates, to register within 45 days for a workshop on student outcomes. Within six
weeks of attending the workshop, the institutions must submit a written plan for improving
student achievement. These institutions are required to submit a follow-up report. Beginning
with the 2000-2001 Annual Report, NACCAS implemented a policy requiring a limited site
review for institutions that do not improve their deficient outcome rate by five percent in the first
year following the Annual Report, do not meet the minimum rate in the second year, or fails one




3
 Twenty of the 99 institutions that failed to meet minimum rates for the 2000-2001 Annual Report, also
failed to meet one or more of the minimum rates based on 1999-2000 Annual Report data.
ED-OIG/A09-C0019                                                                              Page 8 of 18


of the other minimum rates in either the first or second year.4 For the one institution in our
sample that did not meet the minimum rates for the 2000-2001 Annual Report, we confirmed that
institutional representatives attended an outcomes workshop and submitted the required
improvement plan.

Our review found that the formula used in the database for the completion rate was incorrect.5
NACCAS immediately revised the database formula and recalculated the completion rates for
the 2000-2001 Annual Report. NACCAS recalculation identified 20 additional institutions that
did not meet the minimum rates under the corrected formula. NACCAS’ Chief Executive
Officer decided not to notify the 20 institutions that failed the minimum rates because it was too
late for the institutions to attend a workshop or take corrective action before the end of the next
reporting period.

Annual Report Verification Study. NACCAS’policy requires that the agency conduct an Annual
Report Verification Study to confirm information provided in institutional Annual Reports. The
agency randomly selects accredited institutions for the verification study. NACCAS’ current
policy is to exclude from the selection pool those institutions that participated in a verification
study conducted in the prior three years or are scheduled for a site visit in the current year.

For the 1999-2000 Annual Report Verification Study, NACCAS selected a total of
76 institutions. The selected institutions were required to submit a listing of students scheduled
to graduate in the report year. NACCAS staff confirmed that the listing supported the totals
reported in the institutional Annual Report. For a sample of students, the institutions were
required to submit enrollment agreements, proof of graduation, proof of employment, and other
documentation. NACCAS staff reviewed the supporting documentation and conducted phone
interviews with students and employers to confirm the sampled students’ completion, licensure,
and placement, but did not review support for “exempt students” used in the completion rate
formula. NACCAS informed the individual institutions of problems identified in their Annual
Reports.

Based on the review, NACCAS recalculated the institutions’ completion, licensure, and
placement rates. For purposes of the verification study, NACCAS did not include in the rate
recalculations the numbers of “exempt students” and students’ “ineligible for employment.”
The Final Report for the 1999-2000 Annual Report Verification Study, issued in September
2001, states “[u]nadjusted rates were used for some purposes in this study as they are a fair basis




4
 Based on outcomes information in the report from the limited site review and other information, the
Commission takes one or more of the following actions: continue accreditation, place the institution on
“Accreditation with Reporting Requirements,” require an early renewal of accreditation, or withdraw
accreditation.
5
 The database formula was “total number of students who actually completed the course plus the number
of exempt students” divided by the “total number of students scheduled to complete the course within
150 percent of the normal completion time.”
ED-OIG/A09-C0019                                                                              Page 9 of 18


for direct comparisons.” Schools with recalculated rates below the minimum rates may be
directed to submit additional information or an improvement plan.6

The Final Report disclosed the following for the 76 institutions in the sample:

    ƒ   About half of the institutions improperly included students in the count of students who
        were scheduled to complete in the report year.
    ƒ   Nine institutions changed from meeting to failing the minimum completion rate based on
        the recalculations.
    ƒ   Six institutions changed from meeting to failing the minimum licensure rate based on the
        recalculations.
    ƒ   Fifteen institutions changed from meeting to failing the minimum placement rate based
        on the recalculations.
    ƒ   Fourteen institutions did not provide any acceptable documentation on placements.
    ƒ   The number of graduates claimed as “ineligible for employment” allowed 10 institutions
        to change from failing to meeting the minimum placement rate. 

    ƒ   Licensure rates could not be verified for 13 institutions. 


The report recommended that NACCAS continue to cover finding areas in workshops and
publications, provide technical assistance to schools, work with software developers to improve
record keeping systems, and expend effort to have States make licensure data available to
schools. The report also recommended that NACCAS adopt a placement rate formula that
“provides consumers with useful pre-enrollment information” and “will alert the Commission to
take appropriate action when a school is not fulfilling the accreditation goal of preparing students
for licensure and employment in the field.” The Commission did not accept the recommendation
to change the placement formula. The Commission concluded that the existing formula was an
appropriate measure.

Institutional Self-Study and Site Visits. NACCAS requires institutions seeking renewal of
accreditation to conduct an institutional self-study assessing strengths and weaknesses in relation
to the institutional mission, educational objectives, and accreditation standards. After receiving
the self-study report, an evaluation team, comprised of a representative from a cosmetology
school, the salon industry, the public, and NACCAS’ staff, conducts a site visit and issues a
Team Report. The purpose of the site visit is to evaluate the institution’s adherence to the
accreditation standards. Institutions are required to renew their accreditation every one to five
years.

The institutional self-study consists of responses to a series of questions and requests for
supporting exhibits that relate to NACCAS’ accreditation standards and criteria. The
institutional self-study is required to include the following areas related to student achievement
and measures of program length:



6
  According to the Director for Government Relations and Legal Department, NACCAS did not require
institutions with recalculated rates that were slightly below the minimum rate to submit documentation or
a plan. In making the decision whether institutions need to submit additional documentation or an
improvement plan, the Director stated that she considers the impact that the number of reported “exempt
students” and students “ineligible for employment” would have on the recalculated rates.
ED-OIG/A09-C0019                                                                               Page 10 of 18


              Student Achievement                               Measures of Program Length
ƒ Provide a summary of the comments and                  ƒ Describe any significant changes that have
  suggestions received through surveys from                been made in the institution’s course offerings
  students, graduates and employers of graduates.          or in the curriculum since its last grant of
                                                           accreditation.
ƒ Describe the frequency of and how the institution
  assesses its outcome rates and the information         ƒ Describe how the institution ascertains that
  received through surveys of current students,            relevant and current practices of each
  graduates and employers of graduates. List who           occupation are continually being taught.
  participated in the assessment process, their role,    ƒ Describe how the institution ensures that each
  and the frequency of assessment.                         course is designed so that students acquire the
ƒ Provide a copy of the Annual Report and the              skills needed to make a smooth transition into
  worksheets used to compile the information to the        entry-level employment.
  evaluation team for verification.                      ƒ Cite external validations and opinions of the
ƒ Submit your institution’s improvement plan.              institution’s curriculum, such as by State
  Describe how this plan was formulated in light of        regulatory agencies or the institution’s advisory
  the institution’s assessment of its outcomes. If the     committee.
  institution’s outcomes fall below its expectations     ƒ Provide a copy of the course outline for each
  or NACCAS’ minimum rates, describe the steps             course offered by the institution cross-
  taken to improve the rates. If the institution’s         referenced to the list of subject areas required
  rates meet or exceed NACCAS’ minimum rates,              by NACCAS, unless prohibited by State law or
  describe specific actions taken to maintain or           regulation.
  improve the rates.


NACCAS provides the evaluation team with a handbook for conducting and reporting on site
visits. The handbook provides a series of questions/instruction related to NACCAS’ standards of
accreditation, including the standards for student achievement and measures of program length.

                             Site Team Questions and Instruction Related to
                         Student Achievement and Measures of Program Length

Conduct a phone survey of 5% of the school’s graduates who have been placed within the last calendar
year, in order to verify employment in the cosmetology field.
When does the school evaluate its achievements using the outcomes listed in the criterion? Who studies
the outcomes and performs the evaluations? Describe the type of documentation reviewed as part of the
evaluation and the results maintained by the school.
Describe the method used by the school to seek feedback from students, graduates and employers. How
often, and what kind of information is sought? What type of documentation does the school maintain to
show that the information received is used for the evaluation of its achievements?
Does the school count only those students who have completed all requirements within 150% of the
published course length in compiling its completion, placement, and licensure rates?
Does EACH course meet or exceed the state’s minimum standards? If a course exceeds the state’s
minimum requirements, explain in what areas. Were there any findings related to the course length in the
last review by the State Board? If yes, what was the outcome?
List any course Title IV eligible, which exceeds 150% of the state’s required hours.
ED-OIG/A09-C0019                                                                                 Page 11 of 18


The NACCAS staff member of the evaluation team conducts the review of the Annual Report.
According to a NACCAS program manager, the NACCAS staff member reconciles the student
counts reported on each line of the Annual Report to the institution’s student lists. The
NACCAS staff member reviews supporting documentation for 10 percent of the students listed
for each report line, including the line for “exempt student” and “student ineligible for
employment.” The evaluation team calls employers to verify placements for the selected
students reported as employed on the Annual Report.

NACCAS’files for the 10 sampled institutions contained an institutional self-study report, which
addressed the student achievement and program length areas. Of the nine institutions with
Team Reports,7 three reports disclosed no institutional limitations (deficiencies) related to
student achievement and program length. The other six reports disclosed institutional limitations
related to student achievement in one or more of the following areas: advisory committee’s
representation, meeting frequency, review of outcomes rates, program for assessing the
effectiveness of its programs, use of the assessment results or outcomes rates in developing an
improvement plan, and submission of lesson plans. For one institution, the site visit team
reported that the Annual Report was inaccurate. We found that the Team Reports generally did
not include information on the results of the NACCAS staff member’s review of the Annual
Report and some reports did not include explanations of how the institution met NACCAS
standards and criteria.

Due to the limited documentation available in NACCAS’ files, we were unable to confirm that
the site visit teams followed NACCAS’ procedures for site visit reviews. NACCAS’ files did
not contain completed questionnaires, evaluation checklists, or other documentation showing the
results of the teams’ evaluation of institutional compliance with the accreditation standards.
Also, the files did not contain the results of the NACCAS staff member’s verification of
Annual Report data.

NACCAS uses a double-teaming process to assess whether evaluation teams are consistently
applying the accreditation standards. For a selected institution, two evaluation teams are
assigned to conduct the site visit—the designated team and the testing team. NACCAS requires
the institution to respond only to the findings identified by the designated team. NACCAS does
not utilize or reconcile conflicting information from the testing team in making accrediting
decisions. A report on the site visit double-teaming, which NACCAS provided for our review,
contained areas where the two teams reached different conclusions on the areas of
noncompliance with the accreditation standards.




7
 For one institution, the site visit for renewal of accreditation was scheduled to occur after our file review
at NACCAS.
ED-OIG/A09-C0019	                                                                        Page 12 of 18


Substantive Change

The regulations at 34 C.F.R. § 602.22 address the requirements for an accrediting agency’s
substantive change policy. Paragraph (a) of the section states—

       If the agency accredits institutions, it must maintain adequate substantive change
       policies that ensure that any substantive change to the educational mission,
       program, or programs of an institution after the agency has accredited or
       preaccredited the institution does not adversely affect the capacity of the
       institution to continue to meet the agency’s standards . . . .

Paragraph (a)(2) lists the types of changes that must be addressed by the agency’s policy.
The following listed changes could impact an institution’s adherence to the accrediting
agency’s standards on student achievement and measures of program length:

       (i) 	 Any change in the established mission or objectives of the institution.
       (iii) 	The addition of courses or programs that represent a significant departure, in
              either content or method of delivery, from those that were offered when the
              agency last evaluated the institution.
       (iv) 	The addition of courses or programs at a degree or credential level above that
             which is included in the institution’s current accreditation or
             preaccreditation.
       (v) 	 A change from clock hours to credit hours.
       (vi) 	A substantial increase in the number of clock or credit hours awarded for
             successful completion of a program.

We found that NACCAS’ substantive change policies complied with the Federal regulation.
Two of the ten institutions in our sample had substantive changes—one institution increased its
program length and the other added a program. We found that the institutions properly notified
NACCAS and NACCAS followed its procedures for approving the changes.

To ensure that institutions have reported substantive changes in program length, NACCAS uses
the program information provided in the Annual Reports. Also, in applications for renewal of
accreditation, institutions are required to disclose whether “any of the course(s) offered at your
institution exceed the requirements set by the state regulatory agency?” The institution is
required to supplement its application with a detailed rationale if the course’s length is greater by
20 percent (or 200 hours, whichever is less) the length established by the state regulatory
agency.
ED-OIG/A09-C0019                                                                      Page 13 of 18


Suggestions for Enhancing Management Controls
For Ensuring Adherence to Standards

To enhance its management controls for evaluating and monitoring institutions’ adherence to
standards, we suggest that NACCAS consider—

   ƒ   Including, as part of the Annual Report Verification Study, a verification of supporting
       documentation for the number of “exempt students” reported by institutions.

   ƒ   Having site visit teams include in their Team Reports the results of the NACCAS staff’s
       verification of the Annual Report data and the basis for concluding that institutions met
       the standards for student achievement and measure of course length.

   ƒ   Having the designated and testing evaluation teams utilize and reconcile any conflicting
       findings to ensure that the institution addresses all identified problems.

NACCAS Comments

In its comments on the report, NACCAS stated that its policy is to not retain the completed
questionnaires, evaluation checklists, and other working documents used by on-site evaluation
teams because the Team Report represents the consensus of the team. NACCAS stated that the
Team Report format is being changed to include the results of the NACCAS staff’s verification
of the Annual Report data and the basis for concluding that institutions met the standards for
student achievement and measures of program length. To further confirm that site visit teams
follow its procedures, NACCAS plans to add items to the questionnaire that evaluation team
members use to evaluate each other and the questionnaire given to school owners to comment on
the process. NACCAS also stated it is revising procedures to include verification of “exempt
students” during site reviews.

NACCAS was concerned that our suggestion, regarding utilization and reconciliation of
conflicting findings of the designated and testing evaluation teams, could jeopardize NACCAS’
ability to use the double-teaming process. NACCAS stated that if a few schools are required to
respond to the findings of two teams, the schools are not being treated consistently in accordance
with accreditation procedures. An institution may raise “due process” issues if it is not provided
an opportunity to respond to the teams’ findings.

OIG Response

Adding items to the team and school evaluations and adding the OIG suggested information in
Team Reports would improve NACCAS’ documentation of the procedures actually used by
on-site evaluation teams, but they would not provide the level of detail contained in documents
completed by individual team members during the site visit. We suggest that NACCAS
reconsider its record retention policy. We also encourage NACCAS to reconsider our suggestion
concerning conflicting findings identified during the double-teaming process. Accrediting
agencies are required by Federal regulations to take enforcement action when they become aware
that institutions have failed to meet accreditation standards. NACCAS should be able to develop
procedures that allow it to retain the benefits of double-teaming, while ensuring that the due
process rights of its members are protected and that it acts on all available adverse information.
ED-OIG/A09-C0019                                                                                Page 14 of 18


MANAGEMENT CONTROLS FOR ENSURING
ENFORCEMENT ACTION

When an institution or program is not in compliance with the accrediting agency’s standards, the
regulations at 34 C.F.R. § 602.20 require the agency to either “immediately initiate adverse
action against the institution... or... require the institution... to take appropriate action to bring
itself into compliance” within established timeframes.

NACCAS has policies and procedures that address its efforts to ensure enforcement of its
standards for student achievement and program length. The time allowed by NACCAS’ policies
and procedures for an institution to remedy a deficiency, together with the time for the
Commission to make a final decision, is limited to timeframes that parallel the regulations at
34 C.F.R. § 602.20(a).

NACCAS has three types of enforcement actions: accreditation with reporting requirement,
accreditation on probation, and withdrawal of accreditation. Also, NACCAS’ Chief Executive
Officer is authorized by the Commission to issue an administrative show cause order8 to an
institution that does not submit an annual report or fails to attend an outcomes workshop.

The status of “accreditation with reporting requirements” is granted to institutions that have not
met the minimum rates for graduation, licensure, or placement, but have submitted improvement
plans showing the institution can come into compliance with the minimum rates. NACCAS may
withdraw its accreditation of the institution, if the institution does not meet the minimum rates
within the established timeframes in 34 C.F.R. § 602.20 (a). NACCAS’ Institutional
Effectiveness: Outcomes Assessment Policy states—

        Pursuant to NACCAS’ [accreditation standards] and federal regulations, the
        Commission must withdraw accreditation from any institution which does not
        bring its standards within NACCAS minimums within established time lines. If
        the Commission finds the school is providing a quality education not reflected in
        its low outcomes due to special circumstances, the Commission shall include in
        the school record a written explanation of those circumstances on which it bases
        an action other than withdrawal of accreditation.

NACCAS’ policy does not define “special circumstances.”




8
  Once institutions have received a show cause order, they are required to provide NACCAS with a
response. The response must be accompanied by evidence that demonstrates the institution is in
compliance with the standards and criteria or other requirements cited in the order. If the institution does
not provide the proper documents, the Chief Executive Officer may withdraw the institution’s
accreditation.
ED-OIG/A09-C0019	                                                                        Page 15 of 18


For the 99 institutions that did not meet one or more of the minimum rates for the 2000-2001
Annual Report, NACCAS reported taking the following actions:

   ƒ   Fifty-three institutions attended a workshop and submitted or are scheduled to submit an
       improvement plan.
   ƒ   Thirteen institutions submitted additional data showing their outcome rates met the
       minimum standards.
   ƒ   Twelve institutions were sent show cause orders for failure to provide an improvement
       plan within six weeks after attending a workshop.
   ƒ   Eleven institutions had their accreditation involuntarily terminated.
   ƒ   Seven institutions met minimum outcome rates after NACCAS corrected data entry
       errors. 

   ƒ   Two institutions had previously approved plans and are being monitored by the 

       Commission. 

   ƒ   One institution closed. 


Our testing found that NACCAS followed its policies and procedures for enforcing its
accreditation standards. We reviewed five institutions that had their accreditation withdrawn
during the period January 1 to March 13, 2002. NACCAS withdrew accreditation for three of
the institutions for failing to submit their annual reports and two for failure to pay accreditation
fees. NACCAS terminated the institutions’ accreditation within the timeframes specified in the
regulations. We also reviewed the file of an institution that failed to submit a required
Application for Addition or Change of a Course. NACCAS required the institution to undergo
an early renewal of accreditation.

NACCAS established accreditation decision guidelines and uses a double-teaming process to
ensure that the Commission’s file review teams consistently enforce the accreditation standards.
The file review team reviews the Team Report, institution’s response, and other documents;
applies the decision guidelines; and makes a recommendation to the Commission. The
double-teaming process is similar to that used for the site reviews. Two file review teams are
assigned to conduct the review and develop a recommendation—the designated team and the test
team. The full NACCAS Board of Commissioners only hears the recommendation of the
designated team. NACCAS does not utilize and reconcile areas where there are differences in
the conclusions reached by the two teams. The report on the file review double-teaming, which
NACCAS provided for our review, contained areas where the two teams reached different
conclusions on areas of noncompliance with the accreditation standards.

Suggestions for Enhancing Enforcement Procedures

While our review found that NACCAS has management controls in place for ensuring that
enforcement action is taken when institutions are not in compliance with the standards,
NACCAS could enhance the controls by:

   ƒ   Defining the “special circumstances” that NACCAS would consider when deciding
       whether to withdraw accreditation for institutions not meeting its minimum completion,
       licensure, and placement rates.
ED-OIG/A09-C0019                                                                       Page 16 of 18


    ƒ   Utilizing and reconciling areas where there are differences in the conclusions reached by
        designated and testing file review teams when making accrediting decisions.

NACCAS Comments

In its comments to the report, NACCAS described its on-going study to determine if variable
outcomes standards were needed for institutions serving different student populations, and for
possibly defining special or mitigating circumstances that might be a basis for accepting lower
outcome rates at some institutions. NACCAS stated that the OIG suggestion to utilize and
reconcile conclusions reached by designated and testing file review teams did not raise the same
concern discussed earlier for on-site evaluation teams.

OIG Response

NACCAS did not state whether it would or would not consider implementation of the OIG’s
suggestions. We encourage NACCAS to implement the OIG suggestions by publishing a
definition of “special circumstances” and implementing procedures for utilizing and reconciling
differences in conclusions reached by file review teams.


                                       BACKGROUND
NACCAS is an independent, non-profit corporation. The Board of Commissioners, which
oversees the agency, is comprised of 13 elected representatives from cosmetology schools,
cosmetology salons, and educational administration. The Board of Commissioners sets policy
and approves institutions for accreditation. A Chief Executive Officer administers the policy set
forth by the Board of Commissioners and directs the activities of the agency’s administrative
staff.

The Department most recently renewed NACCAS’ recognition in 2000 for a period of five years.
To be accredited by NACCAS, an institution must meet the agency’s standards and related
criteria published in the NACCAS Handbook.9 NACCAS accredits about 1,000 institutions that,
in total, have about 100,000 students enrolled in their programs.


                    PURPOSE, SCOPE, AND METHODOLOGY
The purpose of the review was to (1) identify NACCAS’ standards for success with respect to
student achievement and measures of program length as required under the HEA, Section 496,
and (2) evaluate NACCAS’ management controls for ensuring that institutions adhere to its
standards for student achievement and measures of program length and that consistent
enforcement action is taken when institutions are not in compliance with the standards.



9
 For purposes of our review, we used NACCAS’ Annotated Rules of Practice and Procedures -
Annotated Standards and Criteria, dated December 2001. The agency’s current procedures were not
fully described in the 2001-2002 edition of the NACCAS Handbook.
ED-OIG/A09-C0019                                                                         Page 17 of 18


Our review focused on the agency’s current standards related to student achievement and
program length and the procedures that NACCAS used for monitoring and enforcing those
standards. We reviewed applicable Federal laws and regulations; reviewed NACCAS’ policies,
procedures, and guidance; and interviewed the Chief Executive Officer and several staff.

To confirm that NACCAS adhered to its monitoring policies and procedures, we reviewed
agency files for 10 institutions. The 10 institutions were randomly selected from the
918 NACCAS-accredited institutions. We reviewed the 2000-2001 Annual Report submitted
by the institutions, which was the most recent report for which NACCAS had calculated the
institutions’ completion, licensure, and placement rates.

To confirm that NACCAS adhered to its enforcement policies and procedures, we reviewed
agency files for the five institutions for which NACCAS withdrew accreditation during the
period January 1 to March 13, 2002.

We relied on student achievement and course length data contained in NACCAS’ database to
achieve our purpose. We also interviewed the System Administrator regarding NACCAS’
controls over the Annual Report data that was entered into its database. To determine the
completeness of the database, we compared the number of institutions in the database to the
number of NACCAS-accredited institutions in the Department’s Postsecondary Education
Participation System database. In addition, we tested the accuracy of the data entered in the
NACCAS’ database by tracing the information to the files for the 15 institutions we reviewed.
Based on these assessments, we concluded that the data were sufficiently reliable for our
purpose.

We conducted work at the NACCAS’ office in Alexandria, Virginia, from March 11 through
March 21, 2002. We held an exit briefing with NACCAS officials on August 28, 2002. Our
review was performed in accordance with generally accepted government auditing standards
appropriate to the scope of the review.


                 STATEMENT ON MANAGEMENT CONTROLS
We assessed the system of management controls, policies, procedures, and practices applicable
to NACCAS’ process for monitoring and enforcing accreditation standards for student
achievement and program length. We performed our assessment to determine whether
NACCAS’ processes provided a reasonable level of assurance that the agency ensured that
institutions adhered to established standards and, when institutions were noncompliant,
NACCAS took consistent enforcement action.

For the purpose of this report, we assessed and classified significant controls related to
NACCAS’ accreditation standards into the following categories:

       ƒ   Monitoring institutions’ adherence to the standards, and
       ƒ   Taking enforcement action for noncompliant institutions.

The management of NACCAS is responsible for establishing and maintaining a management
control structure. In fulfilling this responsibility, judgments by management are required to
ED-OIGIA09-COO19                                                                       Page 18 of 18


assess the expected benefits and related costs of control procedures. The objectives ofthe
system are to provide management with reasonable, but not absolute, assurance that institutions
adhere to accreditation standards and that enforcement action is taken when institutions are
found to be noncompliant with the standards.

Because of inherent limitations in any management control structure, errors and irregularities
may occur and not be detected. Also, projection of any evaluation ofthe system to future
periods is subject to risk that procedures may become inadequate because of changes in
conditions, or that the degree of compliance with the procedures may deteriorate.

As we discussed in the REVIEW RESULTS section, we concluded that NACCAS could enhance
its management controls by (1) verifying the "exempt students" as part of the Annual Report
Verification Study and on-site review process, (2) including in the Team Report the results of the
review of the Annual Report, (3) having the designated and testing evaluation teams utilize and
reconcile any conflicting findings to ensure that the institution addresses all identified problems,
(4) defining "special circumstances" for an action other than withdrawal of accreditation for
institutions not meeting the minimum outcome, and (5) utilizing and reconciling areas where
there are differences in the conclusions reached by designated and testing Commission file
review teams when making accrediting decisions.


                            ADMINISTRATIVE MATTERS

Statements that managerial practices need improvements, as well as other conclusions and
suggestions in this report represent the opinions of the Office of Inspector General. In
accordance with the Freedom of Information Act (5 U.S.C. § 552), reports issued by the Office
of Inspector General are made available, if requested, to members of the press and general public
to the extent information contained therein is not subject to exemptions under the Act.




                                           ~~ Gloria Pilotti
                                              Regional Inspector General
                                                 for Audit




                                                                               -wla.
                               ATTACHMENT 

                 NACCAS’ Comments on the Draft Report 





                                    ED-OIG NOTE

In the introductory paragraphs of the attachment, NACCAS referred to the Department’s
Data Quality Standards. We clarified in a letter to NACCAS, dated March 19, 2002, that
our conclusions regarding adherence to Federal requirements would be based on the
applicable HEA provisions, Federal regulations, Secretary’s comments issued in the
Federal Register, and any guidance or correspondence between the Department and the
accrediting agencies.
Z. . . .

Q
~
~
~
8
\II
           ~
            .
      \..ACCREDITINGCO-t.~
             -'.




             -
 "'~TOLOGY ARTS &. SO
                        m
                        6
                        ~
                        i!l
                        ~
                        IU
                              NATIONAL ACCREDITING COMMISSION OF COSMETOLOGY ARTS & SCIENCES
                                    4401 Ford Avenue, Suite 1300 Alexandria, VA 22302· (703) 600-7600· FAX (703) 379-2200
                                                            http://www.naccas.org naccas@naccas.org




                       October 20, 2002

                       Ms. Gloria Pilotti
                       Regional Inspector General for Audit
                       U.S. Department of Education
                       501 I Street - Suite 9-200
                       Sacramento, CA 95814

                       Dear Ms. Pilotti,

                       We have received the draft report ED-OIG/A09-C0019 following the audit by two or
                       your special agents. We value the high level of professionalism, objectivity, and care
                       taken by Mr. Stan Karpinski and Ms. Beverly Dalman in their conduct of the study on our
                       premises and through subsequent requests for supplemental information. Their
                       thoroughness is appreciated.

                       One element of the review, that surprised NACCAS was application to NACCAS of
                       Data Quality Standards that were developed in 1999 to assist managers of the U.S.
                       Department ofEducation. NACCAS had no knowledge of these Data Quality Standards
                       or that accrediting agencies would be evaluated based on them, until the letter setting up
                       the audit was received ..

                       Our other comments are organized under the suggestions made in the report. A few
                       "housekeeping" comments are included.

                       Responses to OIG Suggestions

                       The OIG suggests that NACCAS consider ­
                          • 	 #1. Assessing at regular intervals the minimum rates for completion, l;icensure,
                              andplacement to ensure the minimums are set at levels that provide an adequate
                              and relevant measure ofeducation quality (p. 6)

                        The report correctly states that in the year 2000 NACCAS' Advisory Committee on
                        Standards reviewed the NACCAS outcomes standards, made recommendations to the
                        Commission, and changes were adopted. The Advisory Committee did not recommend
                        any changes in the quantitative outcomes standards. According to the Committee's
                        schedule, the outcomes standards will undergo comprehensive review again in 2005.
Ms. Gloria Pilotti                                                                      2
Response to ED-OIG/A09-COOI9
October 20,2002

   • 	 #2. Revising the completion rate formula by removingfrom the list ofexempt
       students those students who failed to maintain satisfactory attendance and
       academic progress, and students who completed the program outside 150 percent
       ofthe course. (p. 6)

NACCAS has performed a very preliminary analysis of the impact adoption ofthis
suggestion would have on completion rates at NACCAS-accredited schools. Taking the
sample of74 annual reports selected for the 2000-2001 annual report verification study,
NACCAS noted 18 institutions, or about one fourth of the institutions, did not claim any
exemptions in either of these categories. In seven cases, these exemptions may have
pulled the completion rate above the 50% minimum standard. In the other 49 cases both
the unadjusted rate and the rate calculated using these two adjustments were either above
50% or below. More in-depth analysis of the impact of these exemptions will be a basis
for discussions at the Commission's next policy meeting, in May 2003 ..

   • 	 #3. Either revising the placement rate formula by removing from the list of
       ineligible students those students who failed to take the state licensing
       examination or including a separate minimum rate that measures placement for
       all students who complete the program. (p. 6)

This suggestion will be on the agenda of the Commission's next policy meeting.


   • 	 #4. Defining the type andperiod ofemploymentfor placements reported in
       annual reports. (p. 6

   Action: NACCAS has added two questions to the survey questionnaire used in the
annual report verification study. To the questions asked offormer students who are
graduates, and who have been employed in the field, the following question has been
added: "How many months have you worked in the cosmetology field? _ _ (# of
months)." To the questions asked of employers of graduates the following question has
been added: "How many months did the graduate work there? _ _ (# of months)."

The responses to this survey will give the Commission an idea of how available
information on length of employment may be.

The type of employment is already defined by the Commission.. Employment, to count,
must be related to the program from which the student graduated.

    Proposal: For a decade the Commission has been grappling with the question of how
to obtain accurate, meaningful employment data. It would be easy to cite a litany of
reasons why cosmetology schools, whose average enrollment is 83 students, are limited
in the types of tracking systems and procedures they can install. Instead, we suggest an
approach that would result in more accurate employment data than has previously been
available to the Department, accreditors, or the institutions they accredit.
Ms. Gloria Pilotti                                                                                3
Response to ED-OIG/A09-COOI9
October 20, 2002


Proposal 1: Provide access to the Department ofHealth and Human Services National
Directory ofNew Hires database. 1 The United States Department ofEducation already
carries out data matches with this database as a tool for collecting on student loans. 2
Accreditors could obtain lists ofgraduates with social security numbers and submit these
or a sample of these for employment verification against the national database. This
would allow for tracking graduates over time and could be used to determine whether or
not a graduate'S job is related to the training received.

Proposal 2: State agencies which license cosmetology professionals could require
applicants for the initial license or license renewal to indicate where they are working. If
state law could allow the Department, schools, accrediting agencies, or schools access to
this information by matching social security numbers employment information could be
obtained.

Proposal 3: State agencies could require applicants for salon licenses to list employees,
booth renters, and independent contractors working in the facility. Again, if access were
available to the Department, accreditors, or schools, this would be a source ofgood
employment information.

With any of these proposals, a number ofworking graduates would not be caught:
   • 	 Unlicensed graduates who are working
   • 	 Graduates working in unlicensed salons
   • 	 Unreported booth renters and independent contractors
   • 	 Graduates who have moved out of state (state data bases)

Despite the fact that some employed would not be counted through these proposals,
NACCAS believes access to state or federal databases is the best way to track
employment for a number of reasons:

I.      Accuracy of data
2.      Independence of data
3.      Cost efficiency
4.      Ability to track employment over longer periods oftime

     • 	 #5. Including as part ofthe Annual Report Verification Study, a verification of
         supporting documentation for the number of "exempt students" reported by the
         institutions. (p. 12)




 1 Required by   the Work Opportunity Act of 1996.

 2"Accountability for Results Works: Default Rate - Lowest Ever" e-mail from OPE ASL Partners dated
 21 Sep 2001.
Ms. Gloria Pilotti                                                                           4
Response to ED-OIGIA09-COO 19
October 20, 2002

The on-site evaluation procedure for checking back-up documents is being revised to
include verification for the number of"exempt students" reported on the annual report.

   • 	 #6. Having site visit teams include in their Team Reports the results ofthe 

       NACCAS staff's verification ofthe Annual Report data and the basis for 

       concluding that institutions met the standards for student achievement and 

       measure ofcourse length. (p.12) 


The Team Report format is being adjusted to consistently include this information.

   • 	 # 7. Having the designated and testing evaluation teams utilize and reconcile any
       conflicting findings to ensure that the institution addresses all identified
       problems. (p. 12

    Comment: "Double teaming" of on-site evaluation teams is a research method used
by the Advisory Committee on Standards to gather information on reliability and
consistency in application of the standards. This need must be balanced with the
requirement that NACCAS carry out its service while providing due process to the
institutions which submit to accreditation procedures. On-site evaluation procedures
require a single on-site team to go in to an institution. The school has an opportunity to
respond to the teams findings and so on. If a few schools are required to respond to the
findings of two teams, the schools are not being treated consistently in accordance with
accreditation procedures. In order to avoid "due process" issues raised by this, the
Commission might find this most useful research avenue blocked off

    • 	 #8. Defining the "special circumstances" that NACCAS would consider when
        deciding whether to withdraw accreditation for institutions not meeting its
        minimum completion, licensure, andplacement rates. (p.14)

    Comment. This suggestion already is being addressed through a research project the
Commission contracted with an outside educational research firm. In January 2001 the
Commission solicited bids to conduct a study of the links between student and
community characteristics and graduation, licensure, and employment rates (Links
Project). The purpose was to see ifvariable outcomes standards were needed for
institutions serving different student populations, and for possibly defining special or
mitigating circumstances based on which the Commission might accept lower outcomes
at some institutions.

Funds for the study were included in the 2001-2002 budget and the research began in
December 2001. The OIG auditors learned of the study but the initial final report was not
received until August, 2002, several months after the OIG audit. The Links Project
researchers grouped schools into quintiles according to various characteristics: length of
program, ownership structure ofthe institution, percentage of students receiving Pell
grants, age of students, dependent or independent status of students, length of course,
percent ofPell applicants that were single parents, percent ofPell applicants receiving
Ms. Gloria Pilotti                                                                          5
Response to ED-OIG/A09-COOI9
October 20, 2002

Aid to Families with dependent Children, and percent ofPell applicants that had an
expected family contribution (EFC) of zero. Correlations between characteristics and
outcomes were run. The outcomes NACCAS could expect at a school given the student
characteristics were calculated. Then schools that exceeded expectations were identified.
These were based on adjusted completion, placement and licensure rates.

In October, 2002 the Commission requested that the quintiles and expected outcomes be
re-calculated using unadjusted completion, placement and licensure rates. The initial
purpose of the re-calculation is to identify schools exceeding expectations so the
researchers can find out what "best practices" they use. The Links Project will be
completed before the next policy meeting ofthe Commission (May 2003).

   • 	 #9. Utilizing and reconciling areas where there are differences in the conclusions
       reached by designated and testingfile review teams when making accrediting
       decisions. (p. 14)

   Comment: In the case offile review teams, that "double team," since teams merely
make a recommendation, and the decision is of the full Commission, there is not the same
"due process" issue in adopting this suggestion from the OIG as in adopting suggestion
number 7.

"Housekeeping" :

         A.       (page 3) "NACCAS has not evaluated the minimum rates since their
initial establishment." (page 6) Suggestion that NACCAS assess at regular intervals the
minimum rates for completion, licensure, and placement.

See comment under Suggestion # 1.

         B        (page 6) "Due to the limited documentation available in NACCAS files,
we were unable to confirm that the site visit teams followed NACCAS' procedures for
site visit reviews." (page 10) Due to the limited documentation available in NACCAS
files, we were unable to confirm that the site visit teams followed NACCAS' procedures
for site visit reviews .... "

        Action: About twelve years ago, a decision was made not to preserve the
completed questionnaires, evaluation checklists, and other working documents used by
on-site evaluation teams because the Team Report represents the consensus of the team.
However, as part of each on-site evaluation, the members of the evaluation team are
given a questionnaire to evaluate each other as well as the NACCAS staff person. The
school owner is given a questionnaire to comment on adherence to the process, as well as
to rate the evaluators and staff person. NACCAS is including questions on these
evaluation forms to gather additional information to confirm (or not) that site visit teams
follow NACCAS procedures during on-site evaluations.
 Ms. Gloria Pilotti                                                                                  6
 Response to ED-OIG/A09-COOI9
 October 20, 2002

         C.      (page 8) List offindings from the Final Report (1999-2000 annual report
 verification study)

         Comment: Besides using overall findings for policy purposes, each institution in
 the study receives an individual school analysis to provide technical assistance to help it

 understand how the report should be filled out, and the sorts of back-up documents that
 are acceptable.

         D.          (p. 9) "Institutions are required to renew their accreditation every three to
 five years."

         Correction: Three years is the maximum grant of accreditation the Commission
 may grant an applicant for initial accreditation, or an institution granted accreditation
 following an appeal. Five years is the maximum grant of accreditation the Commission
 may grant to an institution seeking renewal of accreditation. Institutions may be required
 to renew their accreditation every one to five years.

          E.     (p. 10) "Of nine institutions with Team Reports .... " And footnote "One
 institution was scheduled for a site visit at a future date.

           Correction: This comment is misleading as it implies NACCAS accredited an
 institution that had not been visited. The OIG Report does not state the time period
 looked at by the auditors, but seemed to concentrate on two or three recent years Chart 1
 I lists the on-site evaluation visits carried out at the institutions in the OIG sample since
 1997.

Chart 1.
KEY: CON = Consultation Visit COV = change of ownership visit
!NT = initial visit IVS = unannounced visit
LOV = Low Outcomes Visit           REN = renewal visit

 Visits Since 1997
/N(¢.CA8~~f#i ·····Ins1:ltutioti .J,"Q;l~""'\.i~.
                  Addre$.s         .............................
 1074/028012-00   Stevenson's Acad
                  New Orleans, LA 70116
 1736/014041-00 Marinello                                          007476   IVS 1997, REN 1998
                  San Bernardino, CA 92410
 1587/042037-00 Jon Louis                                          009578   REN 1998 

                  Jamaica NY 11432 

 0634/053064-00 
 Ogle                                             013016   REN2001 

                  Arlington, TX 76013 

 1635/053073-00 
 International #4                                 012932   CON 1998, REN 2002
                  Irving, TX 75061
Ms. Gloria Pilotti                                                                       7
Response to ED-OIG/A09-COOI9
October 20, 2002



                   The Academy                      025049      REN 1997, REN 2001
                   W           WA98815
0268/012044-00     Charles ofItaly                  022805      REN 1999
                   Lake Havasu     AZ 86403
1925/030035-00     New Creation                     03327300 !NT 1998, REN 2001
                            MD20746
1909/053179-00                                      032773      !NT 1997, REN 2001
                               TX 77701
0149/015024-00     Artistic Beauty Colleges         021411      COVIREN 2001
                   Westminster CO 80030


       F.     (p. 14) "To be accredited byNACCAS, an institution must meet the
agency's standards and related criteria contained in its publication, titled Annotated
Rules ofPractice and Procedures - Annotated Standards and Criteria, dated December
2001."

        Correction: The schools must comply with the standards, rules, procedures, and
policies published each July in the much more extensive NACCAS Handbook. The
annotated rules and standards are only for use by the Commission and staff to help ensure
consistency in decisions. It was provided to the OIG Auditors because the new "low
outcomes" procedure adopted by the Commission to go into effect in October 2001, had
not been published in the 2001-2002 edition of the NACCAS Handbook.

The OIG report is a third-party review of standards and procedures which can point the
Commission to areas needed improvement or greater control. Again, we appreciate the
work of the audit team.




Mark C. Gross
Chief Executive Officer
                             REPORT DISTRIBUTION LIST

                                 ED-OIG/A09-C0019 



Auditee                                                ED Action Official
 Mark Gross, Chief Executive Director                   Sally Stroup, Assistant Secretary
 National Accrediting Commission of                     Office of Postsecondary Education
  Cosmetology Arts and Sciences
 4401 Ford Avenue, Suite 1300
 Alexandria, Virginia 22302

                     Other Departmental Officials/Staff (electronic copy)
 Audit Liaison Officer                                     Assistant General Counsel
 Office of Postsecondary Education                         Office of the General Counsel

 Accreditation and State Liaison Staff Director              Deputy Secretary
 Office of Postsecondary Education                           Office of the Deputy Secretary

 Accrediting Agency Evaluation Chief                         Chief of Staff
 Office of Postsecondary Education                           Office of the Secretary

 Assistant Secretary                                         Under Secretary
 Office of Legislation and Congressional Affairs             Office of the Under Secretary

 Assistant Secretary                                         Director
 Office of Intergovernmental and Interagency Affairs         Office of Communications

 Director                                                    Chief Financial Officer
 Financial Improvement and Post Audit Operations             Office of the Chief Financial Officer
 Office of the Chief Financial Officer

 Post Audit Group Supervisor                                Chief Operating Officer
 Financial Improvement and Post Audit Operations            Federal Student Aid
 Office of the Chief Financial Officer

                                                             Correspondence Control
                                                             Office of General Counsel