United States General Accounting Office GAO Testimony Before the Subcommittee on Human Resources, Committee on Government Reform and Oversight, House of Representatives For Release on Delivery Expected at 10 a.m. Thursday, October 23, 1997 JOB CORPS Participant Selection and Performance Measurement Need to Be Improved Statement of Cornelia M. Blanchette, Associate Director Education and Employment Issues Health Education, and Human Services Division GAO/T-HEHS-98-37 Job Corps: Participant Selection and Performance Measurement Need to Be Improved Mr. Chairman and Members of the Subcommittee: We are pleased to be here today to discuss Job Corps, a $1 billion program administered by the Department of Labor that serves youths aged 16 to 24 who are economically disadvantaged, in need of additional education or training, and living under disorienting conditions, such as in a disruptive homelife. The 109 Job Corps centers are operated under contracts with Labor, which also issues contracts for outreach and admissions and placement services. Job Corps is the nation’s most expensive job training program with an average cost of more than $15,000 per participant. Reasons for this high cost include that the program serves a severely disadvantaged clientele with multiple barriers to employment and that it provides a comprehensive array of services in a residential setting. In spite of this considerable investment, about 25 percent of the participants in the program drop out within 60 days, and 57 percent of those who leave the program do so without completing their vocational training. In contrast, those who remain in the program and complete their vocational training obtain higher-skill jobs at better wages. My testimony today focuses on how Labor can improve the selection of Job Corps participants and how it can improve the information it has available on program placements in order to adequately manage the program and assess placement contractor performance. Having the right information is particularly important if Labor is to improve program performance as envisioned by the Government Performance and Results Act of 1993 (the Results Act). Much of my testimony is drawn from a study being released today that we conducted at the request of the Chairman as well as from several other Job Corps reports we have issued over the past 2 years.1 For the report being released today, we met with Labor officials and reviewed Labor’s eligibility policy guidance in relation to applicable legislation and regulations. We analyzed national data on the characteristics of program participants and early dropouts enrolled during program year 1995.2 We also analyzed program retention data and placement results for each outreach and admissions and placement contractor during program years 1994 and 1995 to identify contractors that 1 Job Corps: Need for Better Enrollment Guidance and Improved Placement Measures (GAO/HEHS-98-1, Oct. 21, 1997). 2 A program year begins on July 1 of a year and ends on June 30 of the following year. A program year is designated by the year in which it begins. Thus, program year 1995 began on July 1, 1995, and ended on June 30, 1996. Page 1 GAO/T-HEHS-98-37 Job Corps: Participant Selection and Performance Measurement Need to Be Improved had higher and lower retention or placement performance. From among these, we selected 14 contractors to visit—2 that do only outreach and admissions, 1 that provides only placement services, and 11 that perform outreach and admissions functions as well as placement functions. We selected them in order to obtain detailed information on the processes the contractors use to admit applicants into Job Corps and place them upon their leaving the program. In summary, Job Corps needs to improve the selection of program applicants in order to decrease the early dropout rate for program participants. It needs to identify participants, from among its eligible population of about 6 million, who have the commitment, attitude, and motivation to complete the training and benefit from Job Corps’ comprehensive and intensive services. We found that the procedures that the more successful outreach and admissions contractors use include commitment checks as well as preenrollment tours and briefings, which give applicants a more realistic basis for deciding whether to enroll in the program. We also found that Job Corps’ policy guidance for 2 of the 11 eligibility criteria is ambiguous and incomplete, leading to an eligibility determination process that fails to follow the requirements of the legislation and program regulations. Further, although Labor uses performance measures to make decisions about renewing placement contractors, we found that two of the four measures Labor uses do not provide information meaningful for assessing the performance of placement contractors. In addition, related measures regarding overall program performance are flawed, thus inaccurately assessing program performance. Although Job Corps reports that about 65 percent of its participants are placed in jobs and about 46 percent of these placements are related to the training participants receive, our work has raised questions about the accuracy and relevancy of both of these figures. These flawed measures jeopardize the ability of Job Corps, the Employment and Training Administration (ETA) (which administers Job Corps), and Labor in general to determine whether the goals stated in Labor’s strategic plan are being achieved. Job Corps currently operates 109 centers throughout the United States, Background including centers in Alaska and Hawaii, the District of Columbia, and Puerto Rico. Most states have at least one center, and several states have four or more centers. Private corporations and nonprofit organizations, selected through competitive procurement, operate the majority of the Page 2 GAO/T-HEHS-98-37 Job Corps: Participant Selection and Performance Measurement Need to Be Improved centers. However, the departments of Agriculture and the Interior operate 28 centers, called civilian conservation centers, under interagency agreements. To address the needs of students with multiple employment barriers, Job Corps provides a comprehensive range of services, primarily in a residential setting.3 It provides services 24 hours a day, 7 days a week, including basic education, vocational skill training, social skill instruction, counseling, health care, room and board, and recreation. Job Corps also offers training in several vocational areas, such as business occupations, automotive repair, construction trades, and health occupations. Because of the comprehensive services Job Corps provides, it is a relatively expensive program. According to Labor’s program year 1995 figures, the average cost per Job Corps student was more than $15,000.4 In contrast, the cost per participant in the Job Training Partnership Act title II-C year-round program for youths is $1,673.5 Cost varies according to how long Job Corps participants remain in the program—we estimated that, at the six centers included in a previous study, those who completed vocational training stayed an average of 13 months with an associated cost of about $26,200.6 Considerable Job Corps’ resources are spent on participants who drop out early and others who fail to complete their training. Although the length of time students stay in Job Corps can vary substantially—from 1 day up to 2 years—a large number of Job Corps participants leave within a short time after enrollment.7 In program year 1995, about 15 percent of the enrollees left within 30 days of entering the program and more than one-fourth left within 60 days. Still others who stayed longer failed to complete their training. In total, 57 percent of program year 1995 terminees did not complete their vocational training. We estimated in our 1995 report that 3 In an earlier study, we compared the characteristics of Job Corps terminees with comparable youths in the Job Training Partnership Act (JTPA) youth training program and found that a greater proportion of Job Corps youths (about 68 percent) than JTPA youths (39 percent) had multiple barriers to employment. See Job Corps: High Costs and Mixed Results Raise Questions About Program’s Effectiveness (GAO/HEHS-95-180, June 30, 1995). 4 On average, students spend about 7 months in the program. 5 The average length of stay for JTPA title II-C is about 10 months. However, JTPA does not provide the comprehensive array of services provided by Job Corps nor is it a residential program. 6 Job Corps: High Costs and Mixed Results Raise Questions About Program’s Effectiveness (GAO/HEHS-95-180, June 30, 1995). 7 Job Corps participants may be enrolled in the program for up to 2 years and may enroll for an additional year to attend advanced career training. Page 3 GAO/T-HEHS-98-37 Job Corps: Participant Selection and Performance Measurement Need to Be Improved about 40 percent of program funds at the six centers we visited was spent on participants who did not complete vocational training. Outreach and admissions services and placement services are provided by private contractors, the centers, or state employment service agencies under contract. During program year 1995, Job Corps spent about $60 million on outreach and admissions and placement contracts.8 Job Corps is a performance-driven program and outreach and admissions contractors and placement contractors must meet certain levels of achievement in order to continue to participate and to receive program funding. Two performance standards have been established for outreach and admissions contractors for the enrollment of certain numbers of male and female youths and for the proportion of enrollees who remain in the program for more than 30 days. A third standard relates to the percentage of participants who are eventually placed in jobs, the military, schools, or other training programs following program termination. Similarly, placement contractors are required to meet standards related to the percentage of participants placed. Additional placement contractor standards are applied to participants who are placed in jobs, such as the percentage of participants obtaining full-time jobs, jobs directly related to the vocational training they receive, and the average wage they receive at placement. Individuals enroll in Job Corps by submitting applications through outreach and admissions contractors. Enrollment in the program is open entry and its training courses are self-paced, allowing students to enroll throughout the year and to progress at their own pace. Students leave Job Corps for a variety of reasons, including successful completion of the program objectives, voluntary resignation, disciplinary termination, and being absent without leave (AWOL) for 10 consecutive training days. With a few exceptions, participants terminating from Job Corps are assigned to a placement contractor for assistance in finding a job or enrolling in other education or training programs. Placement contractors are to give priority to finding full-time, training-related jobs for participants. Participation in Job Corps can lead to placement in a job or enrollment in further training or education. It can also lead to educational achievements such as attaining a high school diploma and gains in reading or mathematics skills. According to Labor data, 75 percent of the more than 8 About $8 million of this amount was for media support contracts. According to Labor, this high level of media expenditure should be regarded as a one-time but necessary cost to counteract a decline in Job Corps enrollments in program year 1994 and early program year 1995. Page 4 GAO/T-HEHS-98-37 Job Corps: Participant Selection and Performance Measurement Need to Be Improved 60,000 program terminees in program year 1995 were placed—65 percent in jobs and 10 percent in education or other training—and 46 percent of the placements were in training-related jobs. The average wage for all placements was $5.98; for training-related placements, $6.44. Labor has long recognized that participants who complete their vocational training courses tend to do better after program termination—that is, they have significantly higher placement rates. Information we developed during our 1995 study of Job Corps verified this conclusion. We found that students who completed vocational training at the six centers we visited were 50-percent more likely to obtain a job than students who did not complete it (76 percent versus 49 percent, respectively). Furthermore, those who completed such training were five times more likely to obtain a training-related job at wages 25-percent higher than students who did not complete their training.9 In contrast, about two-thirds of the jobs obtained by students who did not complete their training were in low-skill positions such as fast-food worker, cashier, laborer, assembler, and janitor. The eligibility guidance that Labor provides to its outreach and admissions Better Eligibility contractors and the screening procedures these contractors follow need to Guidance and be improved. Labor has not provided adequate guidance on 2 of the 11 Screening Are Needed eligibility criteria—living in an environment characterized by disorienting conditions and having the capability and aspiration to complete and to Select Participants secure the full benefits of Job Corps. Without complete and unambiguous guidance, outreach and admissions contractors may not be enrolling the applicants who are the most appropriate for the program and, thus, may potentially be contributing to the dropout rate. We found that contractors with lower dropout rates follow procedures aimed at identifying applicants with the commitment and motivation to remain in and benefit from the program. However, others not following such procedures have higher dropout rates. Guidance for Two Job Corps’ policy guidance for 2 of the 11 eligibility criteria is ambiguous Eligibility Criteria Is and incomplete, which has led to an eligibility determination process that Inadequate fails to follow the requirements of the legislation and regulations. One of 9 Because of the wide latitude Labor guidance permits for determining whether jobs participants obtained were training-related, we analyzed each of the more than 400 placements at the six centers and, using information contained in the Dictionary of Occupational Titles, compared the job with the vocational training provided to determine whether the job was related to the training received. This resulted in a stricter interpretation of a job-training match for both those who completed and those who did not complete vocational training. Page 5 GAO/T-HEHS-98-37 Job Corps: Participant Selection and Performance Measurement Need to Be Improved these requirements is that, to be eligible for Job Corps, program participants must be from an environment so characterized by cultural deprivation, disruptive homelife, or other disorienting conditions as to impair their ability to successfully participate in other education and training programs. However, regarding this environmental requirement, Job Corps’ Policy and Requirements Handbook (1) does not define key terms used to describe “other disorienting conditions,” such as “limited job opportunities,” and (2) limits eligibility to a set of factors that does not include “cultural deprivation,” an environmental factor specified in the law. Further, Labor has not provided adequate guidance regarding another eligibility requirement—that participants have the capability and aspiration to complete and secure the full benefits of Job Corps. Without complete and unambiguous guidance, outreach and admissions contractors may not be enrolling the most appropriate applicants for the program under the law. Contractors With Lower In our most recent study, we found that placement contractors with lower Dropout Rates Have Better dropout rates differ discernibly in outreach and assessment approaches Screening Procedures and practices when compared with contractors having higher dropout rates.10 We noted that admissions contractors with lower dropout rates—10 percent or less—tend to have better procedures for identifying applicants with the commitment and motivation to remain in and benefit from the program. These contractors emphasize making sure that applicants have the capability and aspiration to complete and secure the full benefit of the program, which is one of the program’s statutory eligibility criteria. These more-successful contractors’ procedures include “commitment checks” and preenrollment tours and briefings that give applicants a more realistic basis for deciding whether they want to enroll. This emphasis by these contractors is consistent with the finding we reported in a May 1996 report on successful training programs—namely, that a key job-training strategy shared by successful programs is a focus on ensuring that participants are committed to their training and to getting a job.11 The “commitment checks” that contractors use to test Job Corps applicants’ initiative take a variety of forms. For example, several admissions counselors require individuals interested in Job Corps to set 10 Job Corps: Need for Better Enrollment Guidance and Improved Placement Measures (GAO/HEHS-98-1, Oct. 21, 1997). 11 Employment Training: Successful Projects Share Common Strategy (GAO/HEHS-96-108, May 7, 1996). Page 6 GAO/T-HEHS-98-37 Job Corps: Participant Selection and Performance Measurement Need to Be Improved up application appointments. Four admissions counselors also mentioned that they require applicants to arrive for their meetings dressed in proper attire; otherwise, they have to schedule another appointment. In addition, three admissions counselors require applicants to submit written statements explaining why they want to participate in the program and what they hope to accomplish. Several admissions counselors require applicants to call weekly between the date of application and the enrollment date to determine the status of their application and to demonstrate their continued interest. Finally, one contractor uses a nine-point checklist of documents that all interested persons have to acquire before they set up their application appointment. Some outreach and admissions contractors consider preenrollment tours of Job Corps centers and briefings to be extremely useful, although they are not practical in every situation. They provide applicants with a firsthand opportunity to obtain a thorough understanding of Job Corps’ rules and requirements, observe the living conditions, erase false expectations, and determine whether they are suited for the regimented life of Job Corps. Some preenrollment briefings occur before application; others take place afterward. For example, one contractor requires all interested individuals to participate in a prearranged tour, briefing, and question-and-answer session, after which those still interested must set up an appointment to complete an application. Another contractor requires potential enrollees to take a tour after the application process. After the tour, applicants attend a briefing and a question-and-answer session, followed by one-on-one interviews with center staff. The value of preenrollment tours and briefings was also confirmed at two of the centers we visited by Job Corps participants who thought the tours and briefings were definitely worthwhile and by two regional directors who said that preenrollment tours and briefings are very effective in preparing applicants for Job Corps and in improving the prospect of retention. Several regional directors commented on the importance of identifying applicants who are ready for Job Corps and can benefit from its training. For example, one regional director stated that because the program cannot afford to squander its resources on applicants who do not really want to be in the program, admissions counselors should ensure that applicants are ready and can benefit from the investment. Another regional director noted that because so many people are eligible (more than 6 million), it was important to provide the opportunity of Job Corps to those most likely to benefit and that commitment should be “first and foremost” when assessing applicants. Page 7 GAO/T-HEHS-98-37 Job Corps: Participant Selection and Performance Measurement Need to Be Improved To identify other factors that might be related to program retention, we analyzed the relationship between participant characteristics and the likelihood of remaining in the program for at least 60 days. We found that the participants who are more likely to leave the program within the first 60 days included those who are younger (15 to 17 years old), have less than 12 years of education, have a dependent child, or reside more than 50 miles from the assigned center. The most clear-cut use of this information on participant characteristics may be for designing efforts to improve the retention of participants whose characteristics are associated with leaving the program early. While Job Corps is a performance-driven program, and Labor uses Improved Measures performance measures in evaluating program performance and in making Are Needed to contract renewal decisions, we found that Labor does not have the Evaluate Placement information it needs to accurately assess either the program or placement contractor performance. Two of the four measures Labor uses in assessing Contractor placement contractor performance are not meaningful. One of the Performance measures —placement in jobs—holds contractors accountable for placing participants who are realistically unemployable and, therefore, could lead to an understatement of actual placement performance. At the same time, this measure could overstate placement performance because, as our previous work has shown, many reported placements cannot be confirmed. A second measure—placement in training-related occupations—probably overstates performance for two reasons. First, it includes participants who received little vocational training. Second, it gives placement contractors too much latitude in deciding whether placements are training-related. Problems in these measures also result in flawed assessments of overall program performance. Thus, we are uncertain about how well the program is performing. Job Placement Measures The job placement measure Labor uses for assessing contractor Are Flawed and Many performance has flaws that could lead to both understatement and Reported Placements Are overstatement of actual performance. Labor’s current methodology for calculating a placement contractor’s performance may have resulted in an Unverifiable understatement of the placement rate at the contractors we visited by an average of 8 percentage points. Labor’s calculation includes participants who remained in the program for as little as 1 day, those who were AWOL, and those who were expelled after 30 days for using drugs or committing violent acts—all individuals a placement contractor would have difficulty recommending for employment. During program year 1995, about Page 8 GAO/T-HEHS-98-37 Job Corps: Participant Selection and Performance Measurement Need to Be Improved one-third of the participants leaving Job Corps were in these categories. We recognize that determining what happens to every program participant is an important indicator of how well Job Corps is performing but not necessarily an appropriate measure of placement contractor performance. If Labor’s methodology were modified to include only participants who were in the program long enough to obtain at least minimal benefits (that is, stayed for at least 30 days) and were employable (that is, were not terminated for drug and violence violations and were not AWOL), the average placement rate for the 12 placement contractors we visited would have been from 2.6 percentage points to 13.6 percentage points higher. Job placement data may also be overstated. Although Labor reports that 65 percent of Job Corps participants leaving the program are placed in jobs, our work has raised questions about the validity of this figure. In our June 1995 report, we questioned the validity of about 15 percent of the reported placements at six locations that we visited. We attempted to contact the employers of more than 400 randomly selected placements and found that, in more than 7 percent of the cases, employers reported either that they had never hired the participant or that the individual had never shown up for work, and we were unable to locate the employer of record for about 8 percent of the placements. Although Job Corps has procedures for verifying contractors’ placements, Labor may need to take steps to ensure that it fully implements such procedures and that it provides adequate oversight. In a recent report, Labor’s Inspector General pointed out that Job Corps had not adequately managed or controlled the resolution of questionable placements identified in a sample of Job Corps terminees by the Job Corps’ placement verification contractor.12 Discrepancies in data reported by placement contractors are referred to appropriate Job Corps regional offices for review. The Inspector General’s report found a backlog of questionable placements stemming from Job Corps’ not adequately monitoring the resolution of such placements. In addition, the report said that the verification process had limited benefit because Job Corps did not take timely corrective actions. In a memorandum to Job Corps regional directors, the Director of the Office of Job Corps pointed out that program guidance on what constitutes a valid placement is clear. In addition, placement contractors are required to verify and document 100 percent of their placements. She 12 Office of Inspector General, Job Corps Needs to Improve Its Followup on Questionable Placements (Washington, D.C.: U.S. Department of Labor, Sept. 22, 1997). Page 9 GAO/T-HEHS-98-37 Job Corps: Participant Selection and Performance Measurement Need to Be Improved stated further that “if this is being done, there is really no excuse for a reported placement to be found invalid at a later date.” The Director pointed out that the federal responsibility in this area is oversight. We agree and strongly encourage that oversight be adequate to ensure that placement data are accurately reported. Training-Related The value of the current job-training match data is questionable. The Placement Measure Is job-training match measure is used to evaluate the effectiveness of Flawed vocational training programs and placement contractors by determining the percentage of jobs participants obtain that match the training they receive while in Job Corps. Labor allows placement contractors wide discretion in deciding whether a job placement they obtain for a participant is related to the training he or she receives. At the same time, Labor requires that participants who receive little vocational training be included in the calculation of this measure. Labor is developing a new system to determine job-training matches that, it believes, will be more accurate. Placement contractors are responsible for recording whether or not participants are placed in jobs requiring skills similar to those included in their training. Labor’s guidance for such decisions consists of 16 broad categories of training programs, and within each category are a number of detailed occupations in which Job Corps participants could have received training. In addition, each of the 16 broad categories contains a list of jobs that would be considered a match with the training a participant receives. To illustrate, the broad training category of construction trades includes 47 detailed training occupations and 357 placement occupations. An individual who was trained in any one of the 47 training occupations and was then placed into any one of the 357 placement occupations would be counted as a job-training match. For example, an individual trained as a carpenter (1 of the 47 training occupations) who was placed as a plumber, janitor, or cable television installer (3 of the 357 placement occupations) would be considered to have obtained a training-related placement. Among the wide range of jobs that are considered to be training matches under each of the broad training categories, Labor’s guidance includes jobs that appear to bear little, if any, relationship to Job Corps training. For example, a position as a key cutter would be considered a training match for any of the 51 training categories under the broad category of mechanics and repairers, which includes automobile mechanic, electronics assembler, and parts clerk. A position as a general laborer Page 10 GAO/T-HEHS-98-37 Job Corps: Participant Selection and Performance Measurement Need to Be Improved would be considered a job-training match for any of the 30 training occupations under the precision production category, which includes mechanical drafter, sheet metal worker, and welder. Table 1 lists examples of some possible matches under Labor’s guidance. Table 1: Some Occupations Considered Job-Training Matches for Instructional category Occupation Selected Vocational Training Programs Automobile mechanic Band attacher (attaches wrist bands to watches) Feeder (stacks paper in offset presses) Key cutter Washer (clock parts) Cook Bar attendant Carhop Housecleaner (hotels) Fast-food worker Cosmetologist Hot-room attendant (gives patrons towels) Sales person for weed eradication services Shaver (brushes suede garments after they have been cleaned) Shaver (shaves hog carcasses) Heavy-equipment operator Baggage checker Freight elevator operator Porter Ticket seller Medical secretary Coin counter and wrapper General cashier Hand packager Linen-room attendant Welder Antisqueak filler (shoes) Casket liner General laborer Hacker (lifts bricks of clay tiles from conveyor belt and stacks them) Many of the positions that are considered to be related to Job Corps training require relatively little training to perform. The job placement occupational categories contained in Labor’s guidance for job-training match come from its Dictionary of Occupational Titles. The dictionary includes, for each occupation, the average time required to learn the techniques, acquire information, and develop a facility for average performance in a specific job situation. For more than 700 of the jobs in Labor’s guidance, the average training time is indicated as requiring either only a short demonstration or training up to and including 1 month. Thus, Labor is allowing job-training match credit for occupations such as fast-food worker, cashier, and laborer that require relatively short training Page 11 GAO/T-HEHS-98-37 Job Corps: Participant Selection and Performance Measurement Need to Be Improved time, even though participants spend, on average, about 7 months in the program. While we recognize that some of these positions provide entry into an occupational area that may lead to a better job, it is questionable in our view to consider such positions to be a job-training match until a participant advances into a job commensurate with the training he or she has received. Further, Labor guidance encourages placement contractors to search among the allowable jobs for a job-training match. Its policy handbook states that, if a job-training match is not generated when a job-placement code is entered in its automated system, the placement contractor is allowed to enter a different code that may generate a job-training match, “so long as integrity of data is maintained.” We found that the placement contractors’ practice of recording job-training matches does indeed raise questions about the integrity of the data. One contractor told us that if a placement specialist obtains a job for a participant that is not a job-training match under Labor’s guidance, then the manager and placement specialist meet to determine how to make it a match. This same contractor claimed that it is possible to get a job-training match for participants who were trained as bank tellers, secretaries, and welders and were subsequently placed in fast-food restaurants. For the most part, the placement contractors we visited similarly indicated that they use creativity when entering the code for the placement job in order to obtain a job-training match and raised concerns about the validity of reported job-training match statistics. The job-training match performance measure may also unfairly hold placement contractors accountable for placing certain participants in training-related jobs. All individuals placed in a job or the military are included in the calculation of job-training match, regardless of how long they received vocational training. Thus, individuals who were in the program for a few days or weeks and had little chance to participate in vocational skill-training would be included in the calculation of the job-training match measure. Most of the placement contractors and regional staff we spoke with agreed that it would be more meaningful to include only participants who entirely or substantially completed their vocational skills training when this measure was calculated. According to Labor officials, they are revising the methodology for determining job-training matches, which is currently based on the Dictionary of Occupational Titles. The proposed methodology will use a system that the Bureau of Labor Statistics uses to collect occupational Page 12 GAO/T-HEHS-98-37 Job Corps: Participant Selection and Performance Measurement Need to Be Improved employment data by various industry classifications, with about 830 five-digit codes rather than the 5,700 nine-digit codes taken from the dictionary. According to Labor, the proposed system will be more accurate and easier to maintain and monitor in terms of egregious job-training matches. Labor hopes to have the new methodology in place by July 1, 1998. In addition, Labor stated that the job-training match issue is primary on the agenda of a committee established by Job Corps to improve the quality of vocational outcomes. Labor’s strategic plan with regard to Job Corps would be more useful if the Job Corps Measures measures for the two performance goals it articulated were based on valid Under the Results Act information and included existing performance indicators. Further, relying Could Be Improved on Job Corps’ invalid placement performance data compromises one of the basic purposes of the Results Act—measuring the extent to which goals are achieved—and jeopardizes Labor’s ability to effectively manage the program. The Results Act requires virtually every executive agency to develop a strategic plan, covering a period of at least 5 years from the fiscal year in which it is submitted. The act is aimed at improving program performance. It requires that agencies, in consultation with the Congress and other stakeholders, clearly define their missions and articulate comprehensive mission statements that define their basic purpose. It also requires that they establish long-term strategic goals, as well as annual goals linked to them. Agencies must then measure their performance against the goals they have set and report publicly on how well they are doing. In addition to monitoring ongoing performance, agencies are expected to evaluate their programs and to use the results from these evaluations to improve the programs. The strategic plan Labor submitted under the Results Act consists of a department-level strategic plan overview supplemented by strategic plans for 15 of its offices or units, including ETA. The ETA strategic plan includes six goals, one of which is to increase the number of youths, particularly at-risk youths, who successfully make the transition into the workforce resulting in self-sufficiency. Job Corps is one of several programs aimed at addressing this goal. In that regard, ETA’s plan includes two performance goals for Job Corps: (1) increase the number of youths retaining jobs and increase their earnings and (2) enhance their employability and increase their educational attainment levels. Page 13 GAO/T-HEHS-98-37 Job Corps: Participant Selection and Performance Measurement Need to Be Improved While ETA’s strategic plan structure appropriately defines the role of Job Corps within its mission and strategic goal of increasing the participation of at-risk youths in the workforce, resulting in self-sufficiency, we have concerns about the validity of one of the measures articulated for one of its performance goals. As previously noted, our past work and that of Labor’s Inspector General has questioned the validity of placement information. Thus, one of the basic measures with which Labor proposes to assess the performance of Job Corps under the Results Act does not provide accurate and meaningful information. ETA’s articulation of performance measures could be improved. As shown in table 2, its plan identifies proposed performance measures for one of its goals but not the other. The plan states that performance indicators for the goal of increasing job retention and earnings will eventually include postprogram job retention and postprogram earnings gains. Job Corps will be developing these indicators by collecting data over the next 2 years to develop baseline measures. In the interim, its proposed measures for program years 1998 and 1999 include placement rates and placement wages. As we noted previously, we question the validity of these data. In addition, while wages provide some measure of program success, an additional indicator of program quality that would be useful is an improved measure of job-training match. ETA’s strategic plan has no related performance indicators for the second goal—enhancing employability and increasing educational attainment. Nonetheless, Job Corps’ current performance measurement system contains measures directly related to this goal, including measures of functional literacy, functional numeracy, the attainment of a general equivalency diploma, and completing vocational training. In our opinion, measures such as these would be useful as performance indicators for this second goal. Page 14 GAO/T-HEHS-98-37 Job Corps: Participant Selection and Performance Measurement Need to Be Improved Table 2: Job Corps’ Performance Goals and Measures ETA strategic goal Performance goal Performance measures Increase the number of America’s youths, Increase the number of youths retaining Existing: particularly at-risk youths, who make a jobs and increase earnings, resulting in successful transition into the labor force greater self-sufficiency — number of terminees entering resulting in self-sufficiency employment or further education — at an average placement wage To be developed: — postprogram job retention — postprogram earnings gains Enhance employability and raise No measure identified educational attainment of program terminees Labor also needs accurate information to effectively manage Job Corps. This is particularly important given the program’s complex structure, involving three independent functions—recruiting, training, and placement—that are often contracted for separately. Without accurate information on contractor performance, Labor does not have the data for making proper decisions on contractor renewal. Job Corps is the nation’s most expensive job-training program, with an Conclusions and average cost of more than $15,000 per participant. And, although there are Recommendations reasons for this high cost, a considerable amount of this investment is being spent on participants who drop out early and who fail to complete their vocational training. It has been recognized that those who complete their vocational training do better—that is, they get higher-skill jobs at better wages. To ensure that Job Corps operates the most effectively and that benefits accrue to the greatest number of eligible youths, it is imperative that the program identify, from among its eligible population, the applicants who both need Job Corps’ intensive services and have the commitment, attitude, and motivation to complete the training and benefit from the program. Furthermore, without meaningful and accurate program performance information, Labor’s ability to effectively manage the program is jeopardized. In the report we are releasing today, we make several recommendations to the Secretary of Labor to help ensure that Job Corps uses its resources to serve the most appropriate participants. We recommend that the Secretary Page 15 GAO/T-HEHS-98-37 Job Corps: Participant Selection and Performance Measurement Need to Be Improved provide clear and complete guidance on program eligibility criteria and provide better guidance to ensure that outreach and admissions contractors assess each applicant’s capability and aspirations to complete training and obtain a positive outcome. We also recommend improvements in the measures Labor uses to assess placement contractor performance, to make them more meaningful as tools for improving the selection and retention of contractors. In commenting on a draft of our report, Labor disagreed with our recommendation that it clarify and expand its program eligibility guidance in order to ensure that it is consistent with the law and gave no indication of any formal action it planned to take on this recommendation. Labor expressed concern with our characterization of program eligibility guidance as inadequate. It commented that guidance on one eligibility factor—limited job opportunity—was provided to all admissions counselors during training conducted in program year 1995. Labor also stated that another eligibility factor—cultural deprivation—was not included in its policy handbook because other specific factors were more useful. We disagree that sufficient policy guidance was provided on both factors. Providing guidance on the term “limited job opportunity” during a training program was not adequate because, even if all admissions counselors at that time attended this training, contractors and staff have since turned over. And, as mentioned in our report, the admissions counselors we interviewed interpreted this term in different ways, as they did “cultural deprivation,” thus indicating that contractors need clearer guidance for interpreting these terms consistently. In addition, Labor fails to explain how its guidance on either term satisfies other specific provisions contained in the legislation and program regulations. Although Labor expressed some concern with our remaining recommendations, it acknowledged that they had merit and warranted consideration, and the agency identified actions that it would take. Mr. Chairman, this concludes my prepared statement. We would be happy to answer any questions that you or Members of the Subcommittee may have. (205357) Page 16 GAO/T-HEHS-98-37 Ordering Information The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. VISA and MasterCard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent. Orders by mail: U.S. General Accounting Office P.O. Box 37050 Washington, DC 20013 or visit: Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000 or by using fax number (202) 512-6061, or TDD (202) 512-2537. Each day, GAO issues a list of newly available reports and testimony. To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone. A recorded menu will provide information on how to obtain these lists. For information on how to access GAO reports on the INTERNET, send an e-mail message with "info" in the body to: email@example.com or visit GAO’s World Wide Web Home Page at: http://www.gao.gov PRINTED ON RECYCLED PAPER United States Bulk Rate General Accounting Office Postage & Fees Paid Washington, D.C. 20548-0001 GAO Permit No. G100 Official Business Penalty for Private Use $300 Address Correction Requested
Job Corps: Participant Selection and Performance Measurement Need to Be Improved
Published by the Government Accountability Office on 1997-10-23.
Below is a raw (and likely hideous) rendition of the original report. (PDF)