Job Corps: Participant Selection and Performance Measurement Need to Be Improved

Published by the Government Accountability Office on 1997-10-23.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                             United States General Accounting Office

GAO                          Testimony
                             Before the Subcommittee on Human Resources,
                             Committee on Government Reform and Oversight, House
                             of Representatives

For Release on Delivery
Expected at 10 a.m.
Thursday, October 23, 1997
                             JOB CORPS

                             Participant Selection and
                             Performance Measurement
                             Need to Be Improved
                             Statement of Cornelia M. Blanchette, Associate Director
                             Education and Employment Issues
                             Health Education, and Human Services Division

Job Corps: Participant Selection and
Performance Measurement Need to Be
              Mr. Chairman and Members of the Subcommittee:

              We are pleased to be here today to discuss Job Corps, a $1 billion program
              administered by the Department of Labor that serves youths aged 16 to 24
              who are economically disadvantaged, in need of additional education or
              training, and living under disorienting conditions, such as in a disruptive
              homelife. The 109 Job Corps centers are operated under contracts with
              Labor, which also issues contracts for outreach and admissions and
              placement services. Job Corps is the nation’s most expensive job training
              program with an average cost of more than $15,000 per participant.
              Reasons for this high cost include that the program serves a severely
              disadvantaged clientele with multiple barriers to employment and that it
              provides a comprehensive array of services in a residential setting. In spite
              of this considerable investment, about 25 percent of the participants in the
              program drop out within 60 days, and 57 percent of those who leave the
              program do so without completing their vocational training. In contrast,
              those who remain in the program and complete their vocational training
              obtain higher-skill jobs at better wages.

              My testimony today focuses on how Labor can improve the selection of
              Job Corps participants and how it can improve the information it has
              available on program placements in order to adequately manage the
              program and assess placement contractor performance. Having the right
              information is particularly important if Labor is to improve program
              performance as envisioned by the Government Performance and Results
              Act of 1993 (the Results Act). Much of my testimony is drawn from a study
              being released today that we conducted at the request of the Chairman as
              well as from several other Job Corps reports we have issued over the past
              2 years.1

              For the report being released today, we met with Labor officials and
              reviewed Labor’s eligibility policy guidance in relation to applicable
              legislation and regulations. We analyzed national data on the
              characteristics of program participants and early dropouts enrolled during
              program year 1995.2 We also analyzed program retention data and
              placement results for each outreach and admissions and placement
              contractor during program years 1994 and 1995 to identify contractors that

               Job Corps: Need for Better Enrollment Guidance and Improved Placement Measures
              (GAO/HEHS-98-1, Oct. 21, 1997).
               A program year begins on July 1 of a year and ends on June 30 of the following year. A program year is
              designated by the year in which it begins. Thus, program year 1995 began on July 1, 1995, and ended on
              June 30, 1996.

              Page 1                                                                           GAO/T-HEHS-98-37
             Job Corps: Participant Selection and
             Performance Measurement Need to Be

             had higher and lower retention or placement performance. From among
             these, we selected 14 contractors to visit—2 that do only outreach and
             admissions, 1 that provides only placement services, and 11 that perform
             outreach and admissions functions as well as placement functions. We
             selected them in order to obtain detailed information on the processes the
             contractors use to admit applicants into Job Corps and place them upon
             their leaving the program.

             In summary, Job Corps needs to improve the selection of program
             applicants in order to decrease the early dropout rate for program
             participants. It needs to identify participants, from among its eligible
             population of about 6 million, who have the commitment, attitude, and
             motivation to complete the training and benefit from Job Corps’
             comprehensive and intensive services. We found that the procedures that
             the more successful outreach and admissions contractors use include
             commitment checks as well as preenrollment tours and briefings, which
             give applicants a more realistic basis for deciding whether to enroll in the
             program. We also found that Job Corps’ policy guidance for 2 of the 11
             eligibility criteria is ambiguous and incomplete, leading to an eligibility
             determination process that fails to follow the requirements of the
             legislation and program regulations.

             Further, although Labor uses performance measures to make decisions
             about renewing placement contractors, we found that two of the four
             measures Labor uses do not provide information meaningful for assessing
             the performance of placement contractors. In addition, related measures
             regarding overall program performance are flawed, thus inaccurately
             assessing program performance. Although Job Corps reports that about
             65 percent of its participants are placed in jobs and about 46 percent of
             these placements are related to the training participants receive, our work
             has raised questions about the accuracy and relevancy of both of these
             figures. These flawed measures jeopardize the ability of Job Corps, the
             Employment and Training Administration (ETA) (which administers Job
             Corps), and Labor in general to determine whether the goals stated in
             Labor’s strategic plan are being achieved.

             Job Corps currently operates 109 centers throughout the United States,
Background   including centers in Alaska and Hawaii, the District of Columbia, and
             Puerto Rico. Most states have at least one center, and several states have
             four or more centers. Private corporations and nonprofit organizations,
             selected through competitive procurement, operate the majority of the

             Page 2                                                       GAO/T-HEHS-98-37
Job Corps: Participant Selection and
Performance Measurement Need to Be

centers. However, the departments of Agriculture and the Interior operate
28 centers, called civilian conservation centers, under interagency

To address the needs of students with multiple employment barriers, Job
Corps provides a comprehensive range of services, primarily in a
residential setting.3 It provides services 24 hours a day, 7 days a week,
including basic education, vocational skill training, social skill instruction,
counseling, health care, room and board, and recreation. Job Corps also
offers training in several vocational areas, such as business occupations,
automotive repair, construction trades, and health occupations. Because
of the comprehensive services Job Corps provides, it is a relatively
expensive program. According to Labor’s program year 1995 figures, the
average cost per Job Corps student was more than $15,000.4 In contrast,
the cost per participant in the Job Training Partnership Act title II-C
year-round program for youths is $1,673.5 Cost varies according to how
long Job Corps participants remain in the program—we estimated that, at
the six centers included in a previous study, those who completed
vocational training stayed an average of 13 months with an associated cost
of about $26,200.6

Considerable Job Corps’ resources are spent on participants who drop out
early and others who fail to complete their training. Although the length of
time students stay in Job Corps can vary substantially—from 1 day up to 2
years—a large number of Job Corps participants leave within a short time
after enrollment.7 In program year 1995, about 15 percent of the enrollees
left within 30 days of entering the program and more than one-fourth left
within 60 days. Still others who stayed longer failed to complete their
training. In total, 57 percent of program year 1995 terminees did not
complete their vocational training. We estimated in our 1995 report that

 In an earlier study, we compared the characteristics of Job Corps terminees with comparable youths
in the Job Training Partnership Act (JTPA) youth training program and found that a greater proportion
of Job Corps youths (about 68 percent) than JTPA youths (39 percent) had multiple barriers to
employment. See Job Corps: High Costs and Mixed Results Raise Questions About Program’s
Effectiveness (GAO/HEHS-95-180, June 30, 1995).
 On average, students spend about 7 months in the program.
 The average length of stay for JTPA title II-C is about 10 months. However, JTPA does not provide the
comprehensive array of services provided by Job Corps nor is it a residential program.
 Job Corps: High Costs and Mixed Results Raise Questions About Program’s Effectiveness
(GAO/HEHS-95-180, June 30, 1995).
 Job Corps participants may be enrolled in the program for up to 2 years and may enroll for an
additional year to attend advanced career training.

Page 3                                                                           GAO/T-HEHS-98-37
Job Corps: Participant Selection and
Performance Measurement Need to Be

about 40 percent of program funds at the six centers we visited was spent
on participants who did not complete vocational training.

Outreach and admissions services and placement services are provided by
private contractors, the centers, or state employment service agencies
under contract. During program year 1995, Job Corps spent about
$60 million on outreach and admissions and placement contracts.8

Job Corps is a performance-driven program and outreach and admissions
contractors and placement contractors must meet certain levels of
achievement in order to continue to participate and to receive program
funding. Two performance standards have been established for outreach
and admissions contractors for the enrollment of certain numbers of male
and female youths and for the proportion of enrollees who remain in the
program for more than 30 days. A third standard relates to the percentage
of participants who are eventually placed in jobs, the military, schools, or
other training programs following program termination. Similarly,
placement contractors are required to meet standards related to the
percentage of participants placed. Additional placement contractor
standards are applied to participants who are placed in jobs, such as the
percentage of participants obtaining full-time jobs, jobs directly related to
the vocational training they receive, and the average wage they receive at

Individuals enroll in Job Corps by submitting applications through
outreach and admissions contractors. Enrollment in the program is open
entry and its training courses are self-paced, allowing students to enroll
throughout the year and to progress at their own pace. Students leave Job
Corps for a variety of reasons, including successful completion of the
program objectives, voluntary resignation, disciplinary termination, and
being absent without leave (AWOL) for 10 consecutive training days. With a
few exceptions, participants terminating from Job Corps are assigned to a
placement contractor for assistance in finding a job or enrolling in other
education or training programs. Placement contractors are to give priority
to finding full-time, training-related jobs for participants.

Participation in Job Corps can lead to placement in a job or enrollment in
further training or education. It can also lead to educational achievements
such as attaining a high school diploma and gains in reading or
mathematics skills. According to Labor data, 75 percent of the more than

 About $8 million of this amount was for media support contracts. According to Labor, this high level
of media expenditure should be regarded as a one-time but necessary cost to counteract a decline in
Job Corps enrollments in program year 1994 and early program year 1995.

Page 4                                                                           GAO/T-HEHS-98-37
                          Job Corps: Participant Selection and
                          Performance Measurement Need to Be

                          60,000 program terminees in program year 1995 were placed—65 percent
                          in jobs and 10 percent in education or other training—and 46 percent of
                          the placements were in training-related jobs. The average wage for all
                          placements was $5.98; for training-related placements, $6.44.

                          Labor has long recognized that participants who complete their vocational
                          training courses tend to do better after program termination—that is, they
                          have significantly higher placement rates. Information we developed
                          during our 1995 study of Job Corps verified this conclusion. We found that
                          students who completed vocational training at the six centers we visited
                          were 50-percent more likely to obtain a job than students who did not
                          complete it (76 percent versus 49 percent, respectively). Furthermore,
                          those who completed such training were five times more likely to obtain a
                          training-related job at wages 25-percent higher than students who did not
                          complete their training.9 In contrast, about two-thirds of the jobs obtained
                          by students who did not complete their training were in low-skill positions
                          such as fast-food worker, cashier, laborer, assembler, and janitor.

                          The eligibility guidance that Labor provides to its outreach and admissions
Better Eligibility        contractors and the screening procedures these contractors follow need to
Guidance and              be improved. Labor has not provided adequate guidance on 2 of the 11
Screening Are Needed      eligibility criteria—living in an environment characterized by disorienting
                          conditions and having the capability and aspiration to complete and
to Select Participants    secure the full benefits of Job Corps. Without complete and unambiguous
                          guidance, outreach and admissions contractors may not be enrolling the
                          applicants who are the most appropriate for the program and, thus, may
                          potentially be contributing to the dropout rate. We found that contractors
                          with lower dropout rates follow procedures aimed at identifying
                          applicants with the commitment and motivation to remain in and benefit
                          from the program. However, others not following such procedures have
                          higher dropout rates.

Guidance for Two          Job Corps’ policy guidance for 2 of the 11 eligibility criteria is ambiguous
Eligibility Criteria Is   and incomplete, which has led to an eligibility determination process that
Inadequate                fails to follow the requirements of the legislation and regulations. One of

                           Because of the wide latitude Labor guidance permits for determining whether jobs participants
                          obtained were training-related, we analyzed each of the more than 400 placements at the six centers
                          and, using information contained in the Dictionary of Occupational Titles, compared the job with the
                          vocational training provided to determine whether the job was related to the training received. This
                          resulted in a stricter interpretation of a job-training match for both those who completed and those
                          who did not complete vocational training.

                          Page 5                                                                          GAO/T-HEHS-98-37
                            Job Corps: Participant Selection and
                            Performance Measurement Need to Be

                            these requirements is that, to be eligible for Job Corps, program
                            participants must be from an environment so characterized by cultural
                            deprivation, disruptive homelife, or other disorienting conditions as to
                            impair their ability to successfully participate in other education and
                            training programs. However, regarding this environmental requirement,
                            Job Corps’ Policy and Requirements Handbook (1) does not define key
                            terms used to describe “other disorienting conditions,” such as “limited job
                            opportunities,” and (2) limits eligibility to a set of factors that does not
                            include “cultural deprivation,” an environmental factor specified in the
                            law. Further, Labor has not provided adequate guidance regarding another
                            eligibility requirement—that participants have the capability and
                            aspiration to complete and secure the full benefits of Job Corps. Without
                            complete and unambiguous guidance, outreach and admissions
                            contractors may not be enrolling the most appropriate applicants for the
                            program under the law.

Contractors With Lower      In our most recent study, we found that placement contractors with lower
Dropout Rates Have Better   dropout rates differ discernibly in outreach and assessment approaches
Screening Procedures        and practices when compared with contractors having higher dropout
                            rates.10 We noted that admissions contractors with lower dropout
                            rates—10 percent or less—tend to have better procedures for identifying
                            applicants with the commitment and motivation to remain in and benefit
                            from the program. These contractors emphasize making sure that
                            applicants have the capability and aspiration to complete and secure the
                            full benefit of the program, which is one of the program’s statutory
                            eligibility criteria. These more-successful contractors’ procedures include
                            “commitment checks” and preenrollment tours and briefings that give
                            applicants a more realistic basis for deciding whether they want to enroll.
                            This emphasis by these contractors is consistent with the finding we
                            reported in a May 1996 report on successful training programs—namely,
                            that a key job-training strategy shared by successful programs is a focus
                            on ensuring that participants are committed to their training and to getting
                            a job.11

                            The “commitment checks” that contractors use to test Job Corps
                            applicants’ initiative take a variety of forms. For example, several
                            admissions counselors require individuals interested in Job Corps to set

                             Job Corps: Need for Better Enrollment Guidance and Improved Placement Measures
                            (GAO/HEHS-98-1, Oct. 21, 1997).
                             Employment Training: Successful Projects Share Common Strategy (GAO/HEHS-96-108, May 7,

                            Page 6                                                                    GAO/T-HEHS-98-37
Job Corps: Participant Selection and
Performance Measurement Need to Be

up application appointments. Four admissions counselors also mentioned
that they require applicants to arrive for their meetings dressed in proper
attire; otherwise, they have to schedule another appointment. In addition,
three admissions counselors require applicants to submit written
statements explaining why they want to participate in the program and
what they hope to accomplish. Several admissions counselors require
applicants to call weekly between the date of application and the
enrollment date to determine the status of their application and to
demonstrate their continued interest. Finally, one contractor uses a
nine-point checklist of documents that all interested persons have to
acquire before they set up their application appointment.

Some outreach and admissions contractors consider preenrollment tours
of Job Corps centers and briefings to be extremely useful, although they
are not practical in every situation. They provide applicants with a
firsthand opportunity to obtain a thorough understanding of Job Corps’
rules and requirements, observe the living conditions, erase false
expectations, and determine whether they are suited for the regimented
life of Job Corps. Some preenrollment briefings occur before application;
others take place afterward. For example, one contractor requires all
interested individuals to participate in a prearranged tour, briefing, and
question-and-answer session, after which those still interested must set up
an appointment to complete an application. Another contractor requires
potential enrollees to take a tour after the application process. After the
tour, applicants attend a briefing and a question-and-answer session,
followed by one-on-one interviews with center staff. The value of
preenrollment tours and briefings was also confirmed at two of the centers
we visited by Job Corps participants who thought the tours and briefings
were definitely worthwhile and by two regional directors who said that
preenrollment tours and briefings are very effective in preparing
applicants for Job Corps and in improving the prospect of retention.

Several regional directors commented on the importance of identifying
applicants who are ready for Job Corps and can benefit from its training.
For example, one regional director stated that because the program
cannot afford to squander its resources on applicants who do not really
want to be in the program, admissions counselors should ensure that
applicants are ready and can benefit from the investment. Another
regional director noted that because so many people are eligible (more
than 6 million), it was important to provide the opportunity of Job Corps
to those most likely to benefit and that commitment should be “first and
foremost” when assessing applicants.

Page 7                                                      GAO/T-HEHS-98-37
                          Job Corps: Participant Selection and
                          Performance Measurement Need to Be

                          To identify other factors that might be related to program retention, we
                          analyzed the relationship between participant characteristics and the
                          likelihood of remaining in the program for at least 60 days. We found that
                          the participants who are more likely to leave the program within the first
                          60 days included those who are younger (15 to 17 years old), have less
                          than 12 years of education, have a dependent child, or reside more than 50
                          miles from the assigned center. The most clear-cut use of this information
                          on participant characteristics may be for designing efforts to improve the
                          retention of participants whose characteristics are associated with leaving
                          the program early.

                          While Job Corps is a performance-driven program, and Labor uses
Improved Measures         performance measures in evaluating program performance and in making
Are Needed to             contract renewal decisions, we found that Labor does not have the
Evaluate Placement        information it needs to accurately assess either the program or placement
                          contractor performance. Two of the four measures Labor uses in assessing
Contractor                placement contractor performance are not meaningful. One of the
Performance               measures —placement in jobs—holds contractors accountable for placing
                          participants who are realistically unemployable and, therefore, could lead
                          to an understatement of actual placement performance. At the same time,
                          this measure could overstate placement performance because, as our
                          previous work has shown, many reported placements cannot be
                          confirmed. A second measure—placement in training-related
                          occupations—probably overstates performance for two reasons. First, it
                          includes participants who received little vocational training. Second, it
                          gives placement contractors too much latitude in deciding whether
                          placements are training-related. Problems in these measures also result in
                          flawed assessments of overall program performance. Thus, we are
                          uncertain about how well the program is performing.

Job Placement Measures    The job placement measure Labor uses for assessing contractor
Are Flawed and Many       performance has flaws that could lead to both understatement and
Reported Placements Are   overstatement of actual performance. Labor’s current methodology for
                          calculating a placement contractor’s performance may have resulted in an
Unverifiable              understatement of the placement rate at the contractors we visited by an
                          average of 8 percentage points. Labor’s calculation includes participants
                          who remained in the program for as little as 1 day, those who were AWOL,
                          and those who were expelled after 30 days for using drugs or committing
                          violent acts—all individuals a placement contractor would have difficulty
                          recommending for employment. During program year 1995, about

                          Page 8                                                     GAO/T-HEHS-98-37
Job Corps: Participant Selection and
Performance Measurement Need to Be

one-third of the participants leaving Job Corps were in these categories.
We recognize that determining what happens to every program participant
is an important indicator of how well Job Corps is performing but not
necessarily an appropriate measure of placement contractor performance.
If Labor’s methodology were modified to include only participants who
were in the program long enough to obtain at least minimal benefits (that
is, stayed for at least 30 days) and were employable (that is, were not
terminated for drug and violence violations and were not AWOL), the
average placement rate for the 12 placement contractors we visited would
have been from 2.6 percentage points to 13.6 percentage points higher.

Job placement data may also be overstated. Although Labor reports that
65 percent of Job Corps participants leaving the program are placed in
jobs, our work has raised questions about the validity of this figure. In our
June 1995 report, we questioned the validity of about 15 percent of the
reported placements at six locations that we visited. We attempted to
contact the employers of more than 400 randomly selected placements and
found that, in more than 7 percent of the cases, employers reported either
that they had never hired the participant or that the individual had never
shown up for work, and we were unable to locate the employer of record
for about 8 percent of the placements.

Although Job Corps has procedures for verifying contractors’ placements,
Labor may need to take steps to ensure that it fully implements such
procedures and that it provides adequate oversight. In a recent report,
Labor’s Inspector General pointed out that Job Corps had not adequately
managed or controlled the resolution of questionable placements
identified in a sample of Job Corps terminees by the Job Corps’ placement
verification contractor.12 Discrepancies in data reported by placement
contractors are referred to appropriate Job Corps regional offices for
review. The Inspector General’s report found a backlog of questionable
placements stemming from Job Corps’ not adequately monitoring the
resolution of such placements. In addition, the report said that the
verification process had limited benefit because Job Corps did not take
timely corrective actions.

In a memorandum to Job Corps regional directors, the Director of the
Office of Job Corps pointed out that program guidance on what
constitutes a valid placement is clear. In addition, placement contractors
are required to verify and document 100 percent of their placements. She

 Office of Inspector General, Job Corps Needs to Improve Its Followup on Questionable Placements
(Washington, D.C.: U.S. Department of Labor, Sept. 22, 1997).

Page 9                                                                       GAO/T-HEHS-98-37
                       Job Corps: Participant Selection and
                       Performance Measurement Need to Be

                       stated further that “if this is being done, there is really no excuse for a
                       reported placement to be found invalid at a later date.” The Director
                       pointed out that the federal responsibility in this area is oversight. We
                       agree and strongly encourage that oversight be adequate to ensure that
                       placement data are accurately reported.

Training-Related       The value of the current job-training match data is questionable. The
Placement Measure Is   job-training match measure is used to evaluate the effectiveness of
Flawed                 vocational training programs and placement contractors by determining
                       the percentage of jobs participants obtain that match the training they
                       receive while in Job Corps. Labor allows placement contractors wide
                       discretion in deciding whether a job placement they obtain for a
                       participant is related to the training he or she receives. At the same time,
                       Labor requires that participants who receive little vocational training be
                       included in the calculation of this measure. Labor is developing a new
                       system to determine job-training matches that, it believes, will be more

                       Placement contractors are responsible for recording whether or not
                       participants are placed in jobs requiring skills similar to those included in
                       their training. Labor’s guidance for such decisions consists of 16 broad
                       categories of training programs, and within each category are a number of
                       detailed occupations in which Job Corps participants could have received
                       training. In addition, each of the 16 broad categories contains a list of jobs
                       that would be considered a match with the training a participant receives.
                       To illustrate, the broad training category of construction trades includes
                       47 detailed training occupations and 357 placement occupations. An
                       individual who was trained in any one of the 47 training occupations and
                       was then placed into any one of the 357 placement occupations would be
                       counted as a job-training match. For example, an individual trained as a
                       carpenter (1 of the 47 training occupations) who was placed as a plumber,
                       janitor, or cable television installer (3 of the 357 placement occupations)
                       would be considered to have obtained a training-related placement.

                       Among the wide range of jobs that are considered to be training matches
                       under each of the broad training categories, Labor’s guidance includes
                       jobs that appear to bear little, if any, relationship to Job Corps training.
                       For example, a position as a key cutter would be considered a training
                       match for any of the 51 training categories under the broad category of
                       mechanics and repairers, which includes automobile mechanic,
                       electronics assembler, and parts clerk. A position as a general laborer

                       Page 10                                                       GAO/T-HEHS-98-37
                                        Job Corps: Participant Selection and
                                        Performance Measurement Need to Be

                                        would be considered a job-training match for any of the 30 training
                                        occupations under the precision production category, which includes
                                        mechanical drafter, sheet metal worker, and welder. Table 1 lists examples
                                        of some possible matches under Labor’s guidance.

Table 1: Some Occupations
Considered Job-Training Matches for     Instructional category                  Occupation
Selected Vocational Training Programs   Automobile mechanic                     Band attacher (attaches wrist bands to
                                                                                Feeder (stacks paper in offset presses)
                                                                                Key cutter
                                                                                Washer (clock parts)
                                        Cook                                    Bar attendant
                                                                                Housecleaner (hotels)
                                                                                Fast-food worker
                                        Cosmetologist                           Hot-room attendant (gives patrons towels)
                                                                                Sales person for weed eradication services
                                                                                Shaver (brushes suede garments after
                                                                                they have been cleaned)
                                                                                Shaver (shaves hog carcasses)
                                        Heavy-equipment operator                Baggage checker
                                                                                Freight elevator operator
                                                                                Ticket seller
                                        Medical secretary                       Coin counter and wrapper
                                                                                General cashier
                                                                                Hand packager
                                                                                Linen-room attendant
                                        Welder                                  Antisqueak filler (shoes)
                                                                                Casket liner
                                                                                General laborer
                                                                                Hacker (lifts bricks of clay tiles from
                                                                                conveyor belt and stacks them)

                                        Many of the positions that are considered to be related to Job Corps
                                        training require relatively little training to perform. The job placement
                                        occupational categories contained in Labor’s guidance for job-training
                                        match come from its Dictionary of Occupational Titles. The dictionary
                                        includes, for each occupation, the average time required to learn the
                                        techniques, acquire information, and develop a facility for average
                                        performance in a specific job situation. For more than 700 of the jobs in
                                        Labor’s guidance, the average training time is indicated as requiring either
                                        only a short demonstration or training up to and including 1 month. Thus,
                                        Labor is allowing job-training match credit for occupations such as
                                        fast-food worker, cashier, and laborer that require relatively short training

                                        Page 11                                                           GAO/T-HEHS-98-37
Job Corps: Participant Selection and
Performance Measurement Need to Be

time, even though participants spend, on average, about 7 months in the
program. While we recognize that some of these positions provide entry
into an occupational area that may lead to a better job, it is questionable in
our view to consider such positions to be a job-training match until a
participant advances into a job commensurate with the training he or she
has received.

Further, Labor guidance encourages placement contractors to search
among the allowable jobs for a job-training match. Its policy handbook
states that, if a job-training match is not generated when a job-placement
code is entered in its automated system, the placement contractor is
allowed to enter a different code that may generate a job-training match,
“so long as integrity of data is maintained.” We found that the placement
contractors’ practice of recording job-training matches does indeed raise
questions about the integrity of the data. One contractor told us that if a
placement specialist obtains a job for a participant that is not a
job-training match under Labor’s guidance, then the manager and
placement specialist meet to determine how to make it a match. This same
contractor claimed that it is possible to get a job-training match for
participants who were trained as bank tellers, secretaries, and welders and
were subsequently placed in fast-food restaurants. For the most part, the
placement contractors we visited similarly indicated that they use
creativity when entering the code for the placement job in order to obtain
a job-training match and raised concerns about the validity of reported
job-training match statistics.

The job-training match performance measure may also unfairly hold
placement contractors accountable for placing certain participants in
training-related jobs. All individuals placed in a job or the military are
included in the calculation of job-training match, regardless of how long
they received vocational training. Thus, individuals who were in the
program for a few days or weeks and had little chance to participate in
vocational skill-training would be included in the calculation of the
job-training match measure. Most of the placement contractors and
regional staff we spoke with agreed that it would be more meaningful to
include only participants who entirely or substantially completed their
vocational skills training when this measure was calculated.

According to Labor officials, they are revising the methodology for
determining job-training matches, which is currently based on the
Dictionary of Occupational Titles. The proposed methodology will use a
system that the Bureau of Labor Statistics uses to collect occupational

Page 12                                                      GAO/T-HEHS-98-37
                        Job Corps: Participant Selection and
                        Performance Measurement Need to Be

                        employment data by various industry classifications, with about 830
                        five-digit codes rather than the 5,700 nine-digit codes taken from the
                        dictionary. According to Labor, the proposed system will be more accurate
                        and easier to maintain and monitor in terms of egregious job-training
                        matches. Labor hopes to have the new methodology in place by July 1,
                        1998. In addition, Labor stated that the job-training match issue is primary
                        on the agenda of a committee established by Job Corps to improve the
                        quality of vocational outcomes.

                        Labor’s strategic plan with regard to Job Corps would be more useful if the
Job Corps Measures      measures for the two performance goals it articulated were based on valid
Under the Results Act   information and included existing performance indicators. Further, relying
Could Be Improved       on Job Corps’ invalid placement performance data compromises one of
                        the basic purposes of the Results Act—measuring the extent to which
                        goals are achieved—and jeopardizes Labor’s ability to effectively manage
                        the program.

                        The Results Act requires virtually every executive agency to develop a
                        strategic plan, covering a period of at least 5 years from the fiscal year in
                        which it is submitted. The act is aimed at improving program performance.
                        It requires that agencies, in consultation with the Congress and other
                        stakeholders, clearly define their missions and articulate comprehensive
                        mission statements that define their basic purpose. It also requires that
                        they establish long-term strategic goals, as well as annual goals linked to
                        them. Agencies must then measure their performance against the goals
                        they have set and report publicly on how well they are doing. In addition
                        to monitoring ongoing performance, agencies are expected to evaluate
                        their programs and to use the results from these evaluations to improve
                        the programs.

                        The strategic plan Labor submitted under the Results Act consists of a
                        department-level strategic plan overview supplemented by strategic plans
                        for 15 of its offices or units, including ETA. The ETA strategic plan includes
                        six goals, one of which is to increase the number of youths, particularly
                        at-risk youths, who successfully make the transition into the workforce
                        resulting in self-sufficiency. Job Corps is one of several programs aimed at
                        addressing this goal. In that regard, ETA’s plan includes two performance
                        goals for Job Corps: (1) increase the number of youths retaining jobs and
                        increase their earnings and (2) enhance their employability and increase
                        their educational attainment levels.

                        Page 13                                                       GAO/T-HEHS-98-37
Job Corps: Participant Selection and
Performance Measurement Need to Be

While ETA’s strategic plan structure appropriately defines the role of Job
Corps within its mission and strategic goal of increasing the participation
of at-risk youths in the workforce, resulting in self-sufficiency, we have
concerns about the validity of one of the measures articulated for one of
its performance goals. As previously noted, our past work and that of
Labor’s Inspector General has questioned the validity of placement
information. Thus, one of the basic measures with which Labor proposes
to assess the performance of Job Corps under the Results Act does not
provide accurate and meaningful information.

ETA’s articulation of performance measures could be improved. As shown
in table 2, its plan identifies proposed performance measures for one of its
goals but not the other. The plan states that performance indicators for the
goal of increasing job retention and earnings will eventually include
postprogram job retention and postprogram earnings gains. Job Corps will
be developing these indicators by collecting data over the next 2 years to
develop baseline measures. In the interim, its proposed measures for
program years 1998 and 1999 include placement rates and placement
wages. As we noted previously, we question the validity of these data. In
addition, while wages provide some measure of program success, an
additional indicator of program quality that would be useful is an
improved measure of job-training match. ETA’s strategic plan has no
related performance indicators for the second goal—enhancing
employability and increasing educational attainment. Nonetheless, Job
Corps’ current performance measurement system contains measures
directly related to this goal, including measures of functional literacy,
functional numeracy, the attainment of a general equivalency diploma, and
completing vocational training. In our opinion, measures such as these
would be useful as performance indicators for this second goal.

Page 14                                                     GAO/T-HEHS-98-37
                                             Job Corps: Participant Selection and
                                             Performance Measurement Need to Be

Table 2: Job Corps’ Performance Goals and Measures
ETA strategic goal                       Performance goal                               Performance measures
Increase the number of America’s youths,     Increase the number of youths retaining    Existing:
particularly at-risk youths, who make a      jobs and increase earnings, resulting in
successful transition into the labor force   greater self-sufficiency                   — number of terminees entering
resulting in self-sufficiency                                                           employment or further education

                                                                                        — at an average placement wage

                                                                                        To be developed:

                                                                                        — postprogram job retention

                                                                                        — postprogram earnings gains
                                             Enhance employability and raise            No measure identified
                                             educational attainment of program

                                             Labor also needs accurate information to effectively manage Job Corps.
                                             This is particularly important given the program’s complex structure,
                                             involving three independent functions—recruiting, training, and
                                             placement—that are often contracted for separately. Without accurate
                                             information on contractor performance, Labor does not have the data for
                                             making proper decisions on contractor renewal.

                                             Job Corps is the nation’s most expensive job-training program, with an
Conclusions and                              average cost of more than $15,000 per participant. And, although there are
Recommendations                              reasons for this high cost, a considerable amount of this investment is
                                             being spent on participants who drop out early and who fail to complete
                                             their vocational training. It has been recognized that those who complete
                                             their vocational training do better—that is, they get higher-skill jobs at
                                             better wages. To ensure that Job Corps operates the most effectively and
                                             that benefits accrue to the greatest number of eligible youths, it is
                                             imperative that the program identify, from among its eligible population,
                                             the applicants who both need Job Corps’ intensive services and have the
                                             commitment, attitude, and motivation to complete the training and benefit
                                             from the program. Furthermore, without meaningful and accurate
                                             program performance information, Labor’s ability to effectively manage
                                             the program is jeopardized.

                                             In the report we are releasing today, we make several recommendations to
                                             the Secretary of Labor to help ensure that Job Corps uses its resources to
                                             serve the most appropriate participants. We recommend that the Secretary

                                             Page 15                                                            GAO/T-HEHS-98-37
           Job Corps: Participant Selection and
           Performance Measurement Need to Be

           provide clear and complete guidance on program eligibility criteria and
           provide better guidance to ensure that outreach and admissions
           contractors assess each applicant’s capability and aspirations to complete
           training and obtain a positive outcome. We also recommend
           improvements in the measures Labor uses to assess placement contractor
           performance, to make them more meaningful as tools for improving the
           selection and retention of contractors.

           In commenting on a draft of our report, Labor disagreed with our
           recommendation that it clarify and expand its program eligibility guidance
           in order to ensure that it is consistent with the law and gave no indication
           of any formal action it planned to take on this recommendation. Labor
           expressed concern with our characterization of program eligibility
           guidance as inadequate. It commented that guidance on one eligibility
           factor—limited job opportunity—was provided to all admissions
           counselors during training conducted in program year 1995. Labor also
           stated that another eligibility factor—cultural deprivation—was not
           included in its policy handbook because other specific factors were more
           useful. We disagree that sufficient policy guidance was provided on both
           factors. Providing guidance on the term “limited job opportunity” during a
           training program was not adequate because, even if all admissions
           counselors at that time attended this training, contractors and staff have
           since turned over. And, as mentioned in our report, the admissions
           counselors we interviewed interpreted this term in different ways, as they
           did “cultural deprivation,” thus indicating that contractors need clearer
           guidance for interpreting these terms consistently. In addition, Labor fails
           to explain how its guidance on either term satisfies other specific
           provisions contained in the legislation and program regulations.

           Although Labor expressed some concern with our remaining
           recommendations, it acknowledged that they had merit and warranted
           consideration, and the agency identified actions that it would take.

           Mr. Chairman, this concludes my prepared statement. We would be happy
           to answer any questions that you or Members of the Subcommittee may

(205357)   Page 16                                                     GAO/T-HEHS-98-37
Ordering Information

The first copy of each GAO report and testimony is free.
Additional copies are $2 each. Orders should be sent to the
following address, accompanied by a check or money order
made out to the Superintendent of Documents, when
necessary. VISA and MasterCard credit cards are accepted, also.
Orders for 100 or more copies to be mailed to a single address
are discounted 25 percent.

Orders by mail:

U.S. General Accounting Office
P.O. Box 37050
Washington, DC 20013

or visit:

Room 1100
700 4th St. NW (corner of 4th and G Sts. NW)
U.S. General Accounting Office
Washington, DC

Orders may also be placed by calling (202) 512-6000
or by using fax number (202) 512-6061, or TDD (202) 512-2537.

Each day, GAO issues a list of newly available reports and
testimony. To receive facsimile copies of the daily list or any
list from the past 30 days, please call (202) 512-6000 using a
touchtone phone. A recorded menu will provide information on
how to obtain these lists.

For information on how to access GAO reports on the INTERNET,
send an e-mail message with "info" in the body to:


or visit GAO’s World Wide Web Home Page at:


United States                       Bulk Rate
General Accounting Office      Postage & Fees Paid
Washington, D.C. 20548-0001           GAO
                                 Permit No. G100
Official Business
Penalty for Private Use $300

Address Correction Requested