United States General Accounting Office GAO Report to the Chairman, Subcommittee on Human Resources, Committee on Government Reform and Oversight, House of Representatives October 1997 JOB CORPS Need for Better Enrollment Guidance and Improved Placement Measures GAO/HEHS-98-1 United States GAO General Accounting Office Washington, D.C. 20548 Health, Education, and Human Services Division B-272492 October 21, 1997 The Honorable Christopher Shays Chairman, Subcommittee on Human Resources Committee on Government Reform and Oversight House of Representatives Dear Mr. Chairman: Job Corps is an employment and training program that is aimed at providing severely disadvantaged youths with a comprehensive array of services, generally in a residential setting. Job Corps is one of the few remaining federally administered training programs. The Department of Labor contracts with private and nonprofit organizations to (1) recruit and enroll individuals in the program, (2) operate its 109 centers throughout the nation, and (3) place program participants in jobs or additional training upon termination from the program.1 About $1 billion a year is appropriated for Job Corps, and it serves about 68,000 youths. However, about one-quarter of the participants leave the program after a short time—many of them expelled for disciplinary reasons. In your continued interest in the Job Corps program, you asked us to provide you with information on Job Corps recruitment and placement contractors. Specifically, the objectives of our study were to determine (1) whether Job Corps’ policy guidance regarding eligibility criteria is consistent with the legislation and regulations, (2) how the use of recruiting contractors could be improved to increase participant retention in the program, and (3) how the use of placement contractors could be improved to enhance positive outcomes. In carrying out our work, we met with Labor officials and reviewed Labor’s eligibility policy guidance in relation to applicable statutes and regulations. We analyzed national data on the characteristics of program participants and early dropouts enrolled during program year 1995.2 We also analyzed program retention data and placement results for each outreach, admission, and placement contractor during program years 1994 and 1995 to identify contractors that had higher and lower retention or 1 Placement is defined as getting a job, entering the military, returning to school, or entering another training program. 2 A program year begins on July 1 of a year and ends on June 30 of the following year. A program year is designated by the year in which it begins. Thus, program year 1995 began on July 1, 1995, and ended on June 30, 1996. Page 1 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process B-272492 placement performance. From among these, we selected 14 contractors to visit—2 that did only outreach and admissions, 1 that provided only placement services, and 11 that performed outreach and admissions functions and placement functions—to obtain detailed information on the processes used to admit applicants into Job Corps and place them upon their leaving the program. We also interviewed Job Corps participants at three centers to learn about their experiences when they were recruited for the program and to obtain their views about the enrollment process. (App. I contains a more detailed discussion of our scope and methodology.) Job Corps’ policy guidance for 2 of the 11 eligibility criteria was Results in Brief ambiguous and incomplete, which has led to an eligibility determination process that fails to follow the requirements of the law and program regulations. Under Job Corps’ enabling act and its regulations, program participants must be from an environment so characterized by cultural deprivation, a disruptive homelife, or other disorienting conditions as to impair the applicant’s ability to successfully participate in other education and training programs. However, regarding this environmental criterion, Job Corps’ Policy and Requirements Handbook (1) did not provide definitions of key terms to describe “other disorienting conditions,” such as “limited job opportunities,” and (2) limited eligibility to factors that do not include “cultural deprivation,” an environmental factor specified in the law. Further, Labor has not provided adequate guidance regarding another eligibility requirement—that participants have the capability and aspirations to complete and secure the full benefits of Job Corps. Without complete and unambiguous guidance, outreach and admissions contractors may not be enrolling those who are most appropriate, under the act and regulations, for the program. We used two ways to identify how outreach and admissions contractors could target the recruitment and selection of participants to those more likely to stay in and benefit from Job Corps. In our visits to several outreach and admissions contractors, we found that those with higher retention rates followed procedures aimed at identifying applicants with the commitment and motivation to remain in and benefit from the program. And in our analysis of participant characteristics, we identified certain characteristics significantly related to the likelihood of remaining in the program for at least 60 days. Labor could use some of these characteristics to design outreach efforts or to establish priorities among eligible applicants. In addition, this information may be useful to Labor Page 2 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process B-272492 should it decide to undertake an effort to improve the retention rate for participants with characteristics associated with leaving the program within 60 days of enrollment. Although Job Corps is a performance-driven program and Labor uses performance measures to make decisions on placement contractor renewal, we found that two of the measures Labor used were not meaningful and, thus, Labor did not have the information it needed to accurately assess the performance of placement contractors. We found that the placement measure held contractors responsible for placing individuals who may have received little or no benefit from the program or who demonstrated behavior that normally would be unacceptable to most employers. In addition, the job-training match measure did not accurately portray the extent to which participants obtained jobs related to their vocational training because of the wide latitude placement contractors have in deciding whether a job is related to the training received and the creativity contractors used in recording the occupational titles of the jobs obtained. One aspect of placement contractors’ operations associated with better performance was having staff solely responsible for placing Job Corps participants. The seven contractors we visited that had higher placement rates (over 73 percent) had staff solely responsible for placing Job Corps participants. Most of these contractors were also responsible for managing Job Corps centers or had placement staff located at Job Corps centers. In contrast, four of the five contractors having lower placement rates had the same staff responsible for performing outreach and assessment as well as placement; none had placement staff located at the Job Corps center. In addition, three of the contractors we visited were state employment service agencies that provided services to Job Corps participants similar to those provided to regular employment service clients. As a result of their concern about performance, in the past 2 years Labor has not renewed 12 of the 18 contracts with state employment service agencies. Job Corps was established as a national employment and training program Background in 1964 to mitigate employment barriers faced by severely disadvantaged youths. Job Corps enrolls youths aged 16 to 24 who are economically disadvantaged, in need of additional education or training, and living in Page 3 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process B-272492 disorienting conditions such as a disruptive homelife.3 Students may enroll in training programs throughout the year and progress at their own pace. Job Corps provides participants with a wide range of services, including basic education, vocational skills training, social skill instruction, counseling, health care, room and board, and recreation. The program offers vocational skills training in areas such as business occupations, automotive repair, construction trades, and health occupations. Participation in Job Corps can lead to placement in a job or enrollment in further training or education. It can also lead to educational achievements such as attaining a high school diploma and skills in reading or mathematics. Job Corps is unique in that, for the most part, it is residential. About 90 percent of the youths enrolled each year live at Job Corps centers and are provided services 24 hours a day, 7 days a week. The premise for boarding participants is that most come from a disruptive environment and, therefore, can benefit from receiving education and training in a different setting in which a variety of support services are available around the clock. Job Corps operates in a very structured and disciplined environment. For example, established daily routines must be followed, as must specific rules and regulations governing such areas as acceptable dress and behavior. Furthermore, Job Corps participants must have permission to leave the Job Corps center grounds, and participants “earn” home leave, which must be approved before being taken and can be denied for a number of reasons such as failure to follow a center’s rules of conduct. Job Corps typically employs residential staff to oversee dormitory living and security staff for the safety and well-being of its participants. The program recently implemented a “zero tolerance” policy for violence and drugs. This policy includes a “one-strike-and-you’re-out” provision for the most serious violent or criminal offenses as well as for drug violations. Job Corps currently operates 109 centers throughout mainland United States, Alaska and Hawaii, the District of Columbia, and Puerto Rico. Most states have at least one center, and several states have four or more 3 Although the act includes 14- and 15-year-old youths in the age criteria, Job Corps regulations provide that youths 14 and 15 years of age may be eligible “upon a specific determination by the program director to enroll them.” Page 4 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process B-272492 centers.4 Job Corps’ nine regional directors are responsible for the day-to-day administration of the Job Corps program at the centers within their geographic boundaries. Private corporations and nonprofit organizations, selected through competitive procurement, operate the majority of the centers. However, the departments of Agriculture and Interior directly operate 28 centers, called civilian conservation centers, under interagency agreements. The regional directors are also responsible for overseeing the recruitment of youths for program participation and the placement of participants after they leave Job Corps. Recruitment, referred to as outreach and admissions by program managers, and placement services are provided by private contractors, the centers, or state employment service agencies under contract with the regional offices. During program year 1995, Job Corps spent about $60 million on outreach and admissions as well as placement contracts.5 This included amounts paid contractors solely for outreach and admissions and placement services. In addition, a portion of the funding for some Job Corps center operation contracts was specifically designated for outreach and admissions and placement services. Job Corps contractors are expected to meet certain levels of achievement in order to continue to participate in the program and receive program funding. A performance standard has been established for outreach and admissions contractors with respect to “quotas” of male and female youths to be enrolled (as specified in the contract), and a second standard relates to the proportion of participants who are to remain in the program for more than 30 days (90 percent). A third standard relates to the percentage of participants who are eventually placed following termination from the program (70 percent). Similarly, placement contractors are required to meet established standards related to the percentage of participants placed in jobs, the military, schools, or other training programs (70 percent). Additional standards are applied to participants who are placed in jobs. These standards relate to the percentage obtaining full-time jobs (70 percent) and jobs directly related to the vocational training 4 Delaware, New Hampshire, Rhode Island, and Wyoming have no centers. California, Kentucky, New York, North Carolina, Oklahoma, Oregon, Pennsylvania, Texas, and Washington have four or more Job Corps centers. See Job Corps: Where Participants Are Recruited, Trained, and Placed in Jobs (GAO/HEHS-96-140, July 17, 1996). 5 About $8 million of this amount was for media support contracts. According to Labor, this high level of media expenditures should be regarded as a one-time but necessary cost to counteract a decline in Job Corps enrollments in program year 1994 and early program year 1995. Page 5 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process B-272492 received (42 percent). A fourth placement standard relates to the average wage received at placement.6 Individuals enroll in Job Corps by submitting applications through outreach and admissions contractors. The length of time students stay in Job Corps can vary substantially—from 1 day to 2 years.7 In program year 1995, about 15 percent of the enrollees left Job Corps within 30 days of entering the program and more than one-fourth left within 60 days. On the average, however, students spend about 7 months in the program. Students leave Job Corps for a variety of reasons, including successful completion of the program objectives, voluntary resignation, disciplinary termination, and being absent without leave (AWOL) for 10 consecutive training days. With a few exceptions, participants terminating from Job Corps are assigned to a placement contractor for assistance in finding a job or enrolling in other education or training programs. Placement contractors are to give priority to finding full-time, training-related jobs for participants. We found that Job Corps’ policy guidance on two of its eligibility criteria Job Corps Eligibility was ambiguous and incomplete. As a result, the program’s eligibility Guidance Is process was not following all the requirements of the law or program Inadequate regulations. The law specifies program eligibility requirements, including age, economic status, educational needs, medical condition, and behavioral condition—all defined in the legislation, implementing regulations, or Labor policy guidance.8 Another legislative requirement—living in an environment characterized by disorienting conditions—has not been clearly defined in the statute, regulations, or Labor’s guidance. Further, Labor has not provided adequate guidance regarding the requirement that participants have the capability and aspirations to complete and secure the full benefits of Job Corps. Contractors are required to follow Labor’s Policy and Requirements Handbook, which sets out 11 eligibility criteria for the program that all participants must satisfy: age, economically disadvantaged, requires additional education or training, environment, health history, behavioral 6 Because economic conditions vary by location, the standard for this measure is adjusted by a model that adjusts for local conditions. 7 Job Corps participants may be enrolled in the program for an additional year to attend advanced career training. 8 29 U.S.C. 1501. The law also allows the Secretary of Labor to prescribe other eligibility requirements for enrollment. Page 6 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process B-272492 adjustment history, capability and aspirations to participate, legal U.S. resident, child care, parental consent, and Selective Service registration (see app. II).9 The first seven are specified in the law. The policy handbook generally provides guidance on what is needed to meet most of these criteria. For example, to be eligible under the education or training criterion, an applicant must be a dropout or in need of additional education, training, or related support services in order to hold meaningful employment, participate in regular school work, qualify for other training, or satisfy armed forces requirements. However, guidance on two of the criteria (environment and capability and aspirations) is vague. Environmental Criterion Is One of Job Corps’ eligibility criteria specified in the law for participation Open to Interpretation in the program relates to environment: A participant must come from “an environment so characterized by cultural deprivation, a disruptive homelife, or other disorienting conditions as to substantially impair prospects for successful participation in other programs providing needed training, education, or assistance.” Program regulations go on to explain that the disorienting condition must be one that would impair the applicant’s chance of success in a nonresidential program rather than a residential Job Corps program. Job Corps legislation, Labor’s program regulations, and Job Corps’ policy handbook list environmental factors to be considered when assessing eligibility, but these sources of program guidance are not entirely consistent nor do they contain adequate definitions (see table 1). With the exception of the regulatory definition of disruptive homelife, program guidance does not define the factors that make up the environmental criterion. In the absence of specific definitions of the environmental criterion, admissions counselors applied their own interpretations. 9 Three of these criteria do not apply to all applicants. For example, child care applies only to those with a dependent child; parental consent, only to those who are minors; and Selective Service registration, only to male applicants. Page 7 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process B-272492 Table 1: Comparison of Elements of the Environmental Criterion in Job Labor’s policy Corps Program Guidance Criterion Statute Regulation handbook Currently living in an environment characterized by Cultural deprivation Cultural deprivation Disruptive home life Disruptive home lifea Disruptive home life, unsafe, overcrowded dwelling Other disorienting Other disorienting Limited job conditions conditions opportunities; disruptive community; high crime rates a Defined in the regulations as a homelife characterized by conditions such as (1) living in an orphanage or other institution, (2) suffering from parental or familial neglect or abuse, and (3) having parents or guardians who are chronic invalids, alcoholics, or drug addicts or have other serious health conditions. As shown in table 1, Labor includes “limited job opportunities” in its policy handbook as a disorienting condition that fulfills the environmental eligibility requirement. However, none of the sources of program guidance specifically defines this factor or gives any direction to assessment counselors to help them interpret it, nor do they explain how limited job opportunities affect the chance of success in a residential program compared to a nonresidential one. In prior Job Corps regulations, Labor included among “disruptive conditions” that could impair an applicant’s prospect to participate fully in nonresidential training “a neighborhood or community characterized by high crime rates, high unemployment rates, high school dropout rates, and similar handicaps.” Unlike the present regulations, the prior version made clear that applicants might be subject to more than one disruptive factor and that several factors in combination might satisfy this impairment criterion. Labor’s present guidance does not explain how “limited job opportunities” by themselves can satisfy this criterion. Nonetheless, limited job opportunities was the factor cited as fulfilling the environmental eligibility requirement for 92 percent of the 68,000 Job Corps enrollees in program year 1995. Because admissions counselors generally indicate only one environmental factor on the Job Corps application form, we have no way of knowing how many of these participants would have met the environmental criterion had limited job opportunities not been used to fulfill the requirement. Page 8 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process B-272492 Further, the admissions counselors we interviewed had varying interpretations of limited job opportunity. Some thought that it referred to the applicants’ lack of job skills or lack of education, whereas others thought that it referred to the economic condition of the geographic areas in which applicants resided or their being too young or lacking transportation. Cultural deprivation, another eligibility factor that could fulfill the environmental criterion, was not clearly defined—in fact, it is not even listed in Labor’s policy handbook—and was also interpreted differently by various admissions counselors. One contractor referred to persons who had never gone to a museum or the beach; another thought it applied to a situation such as raising a minority child in a nonminority family; a third referred to living in a housing project. Most admissions counselors we interviewed admitted that they had no idea what this term meant. Finally, Labor’s policy handbook restricts what can be considered under the environmental criterion, stating that to be eligible an applicant must be living in an environment characterized by • disruptive homelife; unsafe, overcrowded dwelling; • limited job opportunities; or • disruptive community; high crime rates. However, the handbook excludes cultural deprivation—specified in the statute and Labor’s own regulations—from permitted environmental factors. Inadequate Guidance on The Job Corps law states that to enroll in Job Corps, an applicant must, Capability and Aspirations after careful screening, have the present capability and aspirations to Criterion complete and secure the full benefit of the program. However, in determining whether applicants meet this requirement, Labor relied primarily on an evaluation form that assesses behavior that would be expected of any and all applicants. Without more detailed guidance on the use of this criterion, the program may not always be serving those who are most likely to benefit from it. In previous work, we found that ensuring that project participants are committed to training and getting a job is a key feature of successful employment training projects.10 10 Employment Training: Successful Projects Share Common Strategy (GAO/HEHS-96-108, May 7, 1996). Page 9 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process B-272492 The law does not define “capabilities and aspirations” but leaves to Labor the tasks of defining this term and providing guidance on how it is to be implemented. Labor has developed the “Capability and Aspirations Assessment Tool,” which admissions counselors must complete for each applicant (see app. III). This “tool” formulates four categories of factors—commitment, attitude, capability, and compatibility of applicant and program goals—that are used to assess capability and aspirations and to demonstrate suitability for the program. Factors under commitment include meeting scheduled appointments on time, providing requested documents such as birth certificates, and reacting favorably to program requirements such as following center rules and living away from home. Attitude includes willingly responding to questions and behaving respectfully during the interview. Capability involves obtaining documentation that supports an applicant’s ability to benefit from the program such as school, court, or medical records or a letter from a former employer. Compatibility of applicant and program goals relates to the admissions counselor’s opinion that an applicant’s expressed goals—for example, for job placement or vocational training—can be realistically achieved through Job Corps. The factors specified in Labor’s assessment tool include characteristics that if not displayed would be an appropriate basis for rejecting an application. However, the possession of these characteristics does not necessarily demonstrate that an applicant has the ability and motivation to benefit from Job Corps. Job Corps outreach and admissions contractors and regional staff whom we spoke with pointed out shortcomings in the current approach to assessing applicants’ capability and aspirations. Staff in one of Labor’s regional offices stated that admissions counselors have asked for additional guidance in making better decisions on capability and aspirations. An admissions contractor with statewide recruiting responsibility in one state said that there is a need for a valid assessment tool for this criterion because the current tool is inadequate. Another contractor stated that it filled out Labor’s assessment tool because it is a program requirement but did not use it in assessing the suitability of applicants. One of Labor’s regional offices has started to develop a more meaningful tool. Page 10 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process B-272492 A substantial number of Job Corps participants leave the program within a Recruitment and short time after enrollment—about one-fourth of program year 1995 Selection of Job Corps participants left within 2 months. Therefore, we believed that it would be Participants Could Be useful to identify ways contractors could target recruitment efforts and the selection of applicants to the eligible youths who are more likely to stay in Improved the program and, thus, more likely to benefit from it. To determine the factors that might be related to program retention, we visited a number of outreach and admissions contractors to examine their practices in assessing and screening applicants for the program. We also analyzed the characteristics of the more than 68,000 program year 1995 participants to determine the characteristics that were associated with remaining in Job Corps for at least 60 days.11 In our visits, we identified several procedures that distinguished outreach and admissions contractors with higher retention rates from other outreach and admissions contractors. In general, these procedures were aimed at identifying applicants with the commitment and motivation to remain in and benefit from the program. Our statistical analysis provides some information about characteristics significantly related to the likelihood of remaining in the program for at least 60 days that Labor could use to design outreach efforts, establish priorities among applicants, or improve the retention rate for those who might otherwise leave the program early. Contractors With Higher Of the 11 outreach and admissions contractors that we visited, those with Retention Rates Have higher retention rates (10 percent or fewer of their enrollees dropping out Better Assessment within the first 30 days) tended to have better procedures for identifying applicants with the commitment and motivation to remain in and benefit Procedures from the program. That is, these contractors emphasized making sure that applicants met the programs’ statutory eligibility criterion of having the capability and aspirations to complete and secure the full benefit of the program. These more-successful contractors’ procedures included “commitment checks” and preenrollment tours and briefings, which gave applicants a more realistic basis for deciding whether they wanted to enroll. The emphasis in these programs was consistent with the finding we reported in a May 1996 report on successful training programs—that a key job-training strategy shared by successful programs was a focus on ensuring that participants are committed to training and getting a job.12 It 11 We performed a multivariate logistic regression analysis to identify characteristics associated with individuals staying in Job Corps longer (at least 60 days). 12 Employment Training: Successful Projects Share Common Strategy (GAO/HEHS-96-108, May 7, 1996). Page 11 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process B-272492 was also consistent with the opinions expressed by several regional directors we interviewed. The “commitment checks” contractors’ used were designed to test Job Corps applicants’ initiative. For example, several contractors required individuals interested in Job Corps to set up application appointments. Admissions counselors at four contractors also mentioned that they required applicants to arrive for their meetings dressed in proper attire; otherwise, they had to schedule another appointment. In addition, three admissions counselors required applicants to submit written statements of why they wanted to participate in the program and what they hoped to accomplish. Several admissions counselors required applicants to call weekly between the date of application and the enrollment date to determine the status of their application and to demonstrate their continued interest in the program. Finally, one contractor also used a nine-point checklist of documents that all interested persons had to acquire before they set up their application appointment. Some outreach and admissions contractors considered preenrollment tours and briefings to be extremely useful, although they were not practical in every situation. They provided applicants with a firsthand opportunity to obtain a thorough understanding of Job Corps rules and requirements, observe the living conditions, erase false expectations, and determine whether they were suited for regimented life. In some instances, these preenrollment briefings were given prior to application while others took place afterward. For example, one contractor required that all interested individuals attend a prearranged tour and briefing. After taking the tour, attending the briefing, and participating in a question and answer session, those still interested had to set up an appointment to complete an application. Another contractor required potential enrollees to take a tour after the application process. Following the tour, applicants attended a briefing and question and answer session, followed by one-on-one interviews with center staff. The value of preenrollment tours and briefings was also confirmed by Job Corps participants at two of the centers we visited who thought the tours and briefings were definitely worthwhile and by two regional directors who agreed that the preenrollment tours and briefings were very effective in preparing applicants for Job Corps and in improving program retention. These tours and briefings would help meet the law’s requirements that applicants be given a full understanding of Job Corps as well as what is expected of them after enrollment. Page 12 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process B-272492 Several regional directors commented on the importance of identifying applicants who are ready for Job Corps and can benefit from its training. For example, one regional director stated that because the program cannot afford to squander its resources on applicants who do not really want to be in the program, admissions counselors should ensure that applicants are ready and can benefit from the investment. Another regional director noted that because so many people are eligible for Job Corps (over 6 million) it was important to provide this opportunity to those most likely to benefit and that commitment should be “first and foremost” when assessing applicants. Another regional director agreed that commitment was important but considered the program’s Capability and Aspirations Assessment Tool to be ineffective in measuring it. Characteristics Associated In our analysis, we identified several characteristics associated with With Program Retention program retention that Labor might consider in designing outreach efforts, establishing priorities among applicants, or improving participant retention rates. Some of these characteristics would be of limited value nationwide, however, because so few participants nationwide had those characteristics. In addition, when considering how to use the results from our analysis, Labor also needs to consider other factors. Two of the characteristics most strongly related to the likelihood of remaining in the program were need for bilingual education and years of education. Of the characteristics we examined, the need for bilingual education had the strongest relationship with the likelihood of remaining in the program. Participants needing bilingual training—Spanish as well as other languages—were much more likely than others to remain in the program for at least 60 days. Education was also an important factor—participants with 12 or more years of education were more likely to remain than participants with 8 or fewer years of schooling. Another characteristic with a strong relationship to retention was age. Our analysis indicated that older participants had a greater likelihood than younger participants of remaining in the program. Specifically, when compared to 15-17-year-old participants, those aged 18 to 20 and 21 to 25 were more likely to remain in the program for at least 60 days.13 This analysis supported the concern expressed by many of the admissions counselors we interviewed regarding enrollment, retention, and placement of 16- and 17-year-old youths, who make up nearly 40 percent of the 13 We obtained data for this analysis from Labor’s national database and they showed that less than 1 percent of program year 1995 enrollees were either 15 or 25 years old. Page 13 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process B-272492 program year 1995 enrollees. The concerns they expressed were that these younger youths are often victimized by older participants at the center, have a harder time adjusting to center life, are more likely to drop out, and are difficult to place. Labor program year 1995 outcome data showed that 16- and 17-year-old terminees were less likely to be placed once they left the program (see fig. 1). Because of the difficulty in placing 16- and 17-year-old participants, one regional Labor official believed that the minimum age for enrollment should be increased, while another thought that there should be separate standards for these participants. In contrast, a third regional Labor official thought that maturity, and not age, should be the deciding factor for enrollment. He acknowledged, however, that the program should probably have different expectations and performance standards for 16-year-old participants. Another Labor official told us that a work group has been established to look into the problem of serving 16- and 17-year-old participants. Figure 1: Percentage of Program Year 1995 Terminees Not Placed by Age Percentage Not Placed 50 40 30 20 10 0 16 17 18 19 20 21 22 23 24 25+ Age at Termination Page 14 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process B-272492 Appendix IV discusses our statistical analysis of characteristics related to remaining in the program at least 60 days, including limitations associated with the analysis. Table IV.3 in that appendix contains the final model and significance levels. For example, it shows that other factors that had a significant relationship to the likelihood of remaining in the program for at least 60 days included residing less than 50 miles from the assigned Job Corps center, being a nonresidential student, having no dependents, and having served in the military. Additionally, some of the factors that proved to be useful predictors of remaining in the program were characteristics of only small subsets of participants. For example, because relatively few participants had a need for bilingual education (less than 3 percent of the Job Corps population), that characteristic was limited in its value as a feature for nationwide use in screening. Because we found no large subgroups with great differences, the ability of the model we used in our analysis to predict 60-day retention for the program’s full population is limited. In deciding how to use the results of this analysis, Labor would need to consider more than the statistical results. For example, it would clearly be inappropriate to use these findings to exclude applicants who met the statutory eligibility requirements because they had characteristics associated with a low likelihood of completing the program. If Labor chose to consider these characteristics in designing outreach efforts or establishing priorities for eligible applicants, it would be faced with the complexity of integrating these results with existing eligibility requirements and program policy. For example, our results showed that participants with at least 12 years of education were more likely to remain for 60 days than those with less education. Many youths with that many years in school, however, might not meet the eligibility requirement of needing additional education or training to secure and hold meaningful employment, participate successfully in regular school work, qualify for other suitable training programs, or satisfy armed forces requirements. The most clear-cut use of this information on participant characteristics may be in designing efforts to improve the retention rate of participants with characteristics associated with leaving the program early. Page 15 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process B-272492 Labor uses performance measures in deciding whether contractors are to Performance continue to participate in the program. However, Labor does not have the Measures Are information it needs to accurately assess the performance of its placement Inadequate for contractors. We found that two of the four measures Labor used in assessing placement contractor performance were not meaningful. One of Assessing Placement the measures held contractors accountable for placing participants who Contractor were realistically unemployable. A second measure, relating to the placement of terminees in training-related occupations, included Performance terminees who received little vocational training and also gave placement contractors wide latitude in deciding whether placements were related to training. Job Corps requires placement contractors to assist all terminees with placement regardless of how long they were in the program or the reason they left, and it has established the following standards to measure contractor performance:14 • 70 percent of all terminees assigned to a contractor are to be placed, • 70 percent of all placements are to be in full-time jobs, • the average wage paid to participants placed in jobs is to be equal to or greater than a specified level, and • 42 percent of all job placements are to be in occupations related to the training received. Measurement of Job In calculating a contractor’s placement performance, Labor includes Placements Includes participants who remained in the program for as little as 1 day, those who Unemployable Terminees were AWOL, and those who were expelled from Job Corps after 30 days for using drugs or committing violent acts—all individuals a placement contractor would have difficulty recommending for employment. During program year 1995, about one-third of the participants leaving Job Corps were in these categories. If Labor’s methodology were modified to include only participants who were in the program for sufficient time to obtain at least minimal benefits (that is, stayed for at least 30 days) and were employable (that is, were not terminated for drug violations and violence and were not AWOL), the average placement rate for the 12 placement contractors we visited would be about 8 points higher—ranging from an increase of 2.6 points for one contractor to 13.6 points for another contractor—and the rank order among the 12 contractors would change somewhat. (See fig. 2.) 14 Job Corps contractors provide placement services to all program participants once they leave the program, except those who are terminated within the first 30 days for violating the program’s zero tolerance policy for drugs and violence and those found to be ineligible after enrollment. Page 16 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process B-272492 Figure 2: Program Year 1995 Placement Rates for Selected Contractors Using Existing and Modified Methodology Percentage Placed 100 80 60 40 20 0 A B C D E F G H I J K L Placement Contractors Existing Methodology Modified Methodology About half of the placement contractors we visited suggested that Labor should exclude certain individuals when calculating placement rates. For example, one contractor noted that it is unreasonable to expect contractors to recommend to an employer someone who was expelled for taking drugs or committing a violent act. Another contractor believed that it was a waste of resources to try to place participants who were AWOL because they were not only difficult to locate but also undependable to an employer. A third contractor suggested that Labor’s methodology include only participants who are truly employable. Similarly, a regional director stated that it is ridiculous to require placement specialists to be responsible for placing participants who stayed in the program a very short time, were expelled for drug use or violence, or were AWOL. He said that this responsibility asks the placement specialist to lie to employers by recommending they hire these people. Another regional director agreed that placement contractors should not be responsible for participants who Page 17 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process B-272492 received no benefit from the program or who were kicked out for violating the program’s drug and violence policies. Training-Related The job-training match measure is used to evaluate the effectiveness of Placement Measure Is vocational training programs and placement contractors by determining Flawed the percentage of jobs terminees obtain that matches the training they received while in Job Corps. Labor allows placement contractors wide discretion in deciding whether a job placement they obtain for a terminee is related to the training received—another measure of performance. At the same time, Labor requires that terminees who receive little vocational training be included in the calculation of this measure. As a result, the value of the current job-training match performance measure is questionable. Labor is developing a new system to determine job-training matches that, it believes, will be more accurate. Labor’s guidance gives placement contractors wide latitude in deciding whether a job placement was a job-training match. According to Labor guidance, a job-training match results when a participant is placed in a job requiring skills similar to those included in the participant’s training. Placement contractors are responsible for recording this information. Labor’s guidance for these decisions consists of 16 broad categories of training programs, and within each category are a varying number of detailed occupations in which Job Corps participants may be trained. In addition, each of the 16 broad categories contains a list of jobs that would be considered a match with the training received. To illustrate, the broad training category of construction trades includes 47 detailed training occupations and 357 placement occupations. An individual who was trained in any one of the 47 training occupations and then was placed into any one of the 357 placement occupations would be counted as having made a job-training match. Overall, Labor’s system includes nearly 300 detailed training occupations and more than 5,700 job placement occupations. In addition to the wide range of jobs that are considered to be training matches under each of the broad training categories, Labor’s guidance includes jobs that appear to bear little, if any, relationship to the training received. For example, a position as a key cutter would be considered to be a training match for any of the 51 training categories under the broad category of mechanics and repairers, which includes auto mechanic, electronics assembler, and parts clerk. A position as a general laborer would be considered to be a job-training match for any of the 30 training Page 18 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process B-272492 occupations under the precision production category, which includes mechanical drafter, sheet metal worker, and welder. Table 2 lists examples of some possible matches under Labor’s guidance. Table 2: Examples of Occupations Considered to Be Job-Training Instructional category Occupation Matches for Selected Vocational Automobile mechanic Band attacher (attaches wristbands to Training Programs watches) Feeder (stacks paper in offset press) Key cutter Washer (clock parts) Cook Bar attendant Car hop Housecleaner (hotel) Fast-food worker Cosmetologist Hot-room attendant (gives patrons towels) Sales person for weed eradication services Shaver (brushes suede garment after it has been cleaned) Shaver (shaves hog carcasses) Heavy equipment operator Baggage checker Freight elevator operator Porter Ticket seller Medical secretary Coin counter-and-wrapper General cashier Hand packager Linen-room attendant Welder Antisqueak filler (shoes) Casket liner General laborer Hacker (lifts bricks and clay tiles from conveyor belt and stacks them) Page 19 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process B-272492 Many of the positions that are considered to be related to Job Corps training require relatively little training to perform. The job placement occupational categories contained in Labor’s guidance for job-training match come from its Dictionary of Occupational Titles. The dictionary includes, for each occupation, the average time required to learn the techniques, acquire information, and develop the facility for average performance in a specific job situation. For more than 700 of the jobs included in Labor’s guidance, the average training time is indicated as either only a short demonstration or training up to and including 1 month. Thus, Labor is allowing job-training match credit for occupations requiring relatively short training time even though participants spend an average of about 7 months in the program at an average cost of about $15,300 each.15 While we recognize that some of these positions provide entry into an occupational area that may lead to a better job, in our view it is questionable to consider such positions to be a job-training match until the participant advances into a job commensurate with the training received. Further, Labor guidance encourages placement contractors to search among the allowable jobs for a job-training match. Its policy handbook states that, if a job-training match is not generated when a job placement code is entered in its automated system, the placement contractor is allowed to enter a different code that may generate a job-training match, “so long as integrity of data is maintained.” We found that placement contractors’ practice of recording job-training matches does indeed raise questions about the integrity of the data. One contractor told us that if a placement specialist obtained a job for a terminee that was not a job-training match under Labor’s guidance, then the manager and placement specialist would meet to determine how to make it a match. This same contractor claimed that it is possible to get a job-training match in fast-food restaurants for participants trained as bank tellers, secretaries, and welders. For the most part, the placement contractors we visited similarly indicated that creativity is used when entering the code for the placement job in order to obtain a job-training match and raised concerns about the validity of reported job-training match statistics. The job-training match performance measure may also unfairly hold placement contractors accountable for placing certain terminees in training-related jobs. All participants placed in a job or the military are included in the calculation of job-training match, regardless of how long they received vocational training. Thus, participants for a few days or 15 Job Corps: High Costs and Mixed Results Raise Questions About Program’s Effectiveness (GAO/HEHS-95-180, June 30, 1995). Page 20 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process B-272492 weeks who had little chance to participate to any extent in vocational skill training would be included in the calculation of the job-training match measure. Most of the placement contractors and regional staff we spoke with agreed that when calculating this measure it would be more meaningful to include only participants who completed their vocational skills training. Labor officials told us that they are revising the methodology for determining job-training matches. The proposed methodology will use an existing system used by the Bureau of Labor Statistics to collect occupational employment data by various industry classifications. This system uses about 830 five-digit codes rather than the 5,700 nine-digit codes used in the current methodology based on the Dictionary of Occupational Titles. In its comments on a draft of our report, Labor acknowledged that we made a legitimate point about the need to strengthen the job-training match process. According to Labor, the proposed system will be more accurate and easier to maintain and monitor in terms of egregious job-training matches. Labor hopes to have implemented the new methodology by July 1, 1998. In addition, Labor stated that the job-training match issue is one of the primary projects being addressed by a Job Corps committee to improve the quality of vocational outcomes. We found that a characteristic common to the contractors we visited that Characteristics of had higher placement rates was having staff solely responsible for Contractors With providing placement services to Job Corps participants. In addition, most Higher Placement of these placement contractors were either Job Corps centers or had staff located at the centers they served. In contrast, Labor regional officials Rates have been concerned with the performance of state employment service agencies and have not renewed many of their contracts during the past 2 years. We also noted that Labor and several of the Job Corps centers we visited were starting to improve links to the business community in an effort to increase placements. The placement contractors we visited had had varying success in placing Job Corps participants in program year 1995. Placement included getting a job, entering the military, or returning full-time to school. The seven contractors that had relatively high placement rates (over 73 percent) included four Job Corps centers and three private organizations. A common characteristic among these contractors was having staff who had only one responsibility—placing Job Corps participants. Other contractors Page 21 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process B-272492 that were not as successful used the same staff to perform outreach and admissions as well as placement. One contractor whose staff performed these functions noted that with the program’s emphasis on maintaining centers at full capacity, placement is often secondary to admissions. We also noted that most of the contractors with higher placement rates were either Job Corps centers or had staff at the center. Placement specialists at the Job Corps center contended that being at the center allowed them easy access to instructors, counselors, and participants. One Labor regional director also mentioned the importance of having a continuity of services from the time enrollees arrive at the center until they are placed, noting that it was no accident that every center in his region also has a placement contract. In contrast, the placement contractor we visited with the highest placement rate was not a Job Corps center and did not have staff at a center. The program manager of this private company viewed Job Corps placement as a business and ran the organization accordingly—either placement specialists produced jobs for Job Corps participants or else the program manager found someone who could. Thus, having a focus on the ultimate goal—placement in a job—is a strategy associated with a high placement rate. One type of contractor that generally has not had high placement rates is state employment service agencies. Between program years 1994 and 1996, Labor regional offices did not renew two-thirds (12 of 18) of the placement contracts they had with state employment service agencies (see table 3). Labor officials in three regional offices informed us that they cancelled the placement contracts with state employment service agencies because of poor performance. A Labor official in a fourth region stated that the agency had sent a letter of concern to the state employment service agency because it was the worst-performing placement contractor in the region. Five of the six remaining state employment service placement contractors had placement rates in program year 1995 below the national Job Corps standard of 70 percent. Page 22 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process B-272492 Table 3: State Employment Service Agencies Having Placement Contracts Program year With Job Corps, Program Years 1994 1995 1996 1994-96 State agency Missouri Missouri Missouri Nevada Nevada Nevada North Dakota North Dakota North Dakota Oklahoma Oklahoma Oklahoma South Dakota South Dakota South Dakota Texas Texas Texas Alabama Alabama Arkansas Arkansas Florida Florida Georgia Georgia Louisiana Louisiana Mississippi Mississippi Tennessee Tennessee Virgin Islands Virgin Islands Kansas Kentucky Washington Wyoming Officials from two of the three state employment service agencies we visited expressed reservations about continuing to contract with Job Corps for placement services. For example, one employment service official said that the agency might not seek contract renewal because of its strained relations with Labor’s regional office. An official from another employment service commented that its Job Corps contract was really “small potatoes” and insufficient to provide for adequate staffing and that the only reason it was still involved was that the employment service commissioner believed that Job Corps was worthwhile and wanted to assist disadvantaged youths. An official from the third employment service agency we visited noted that the Labor regional office threatened to cancel its placement contract 2 years ago for poor performance and gave the agency another 6 months to improve. The official noted that, under new management, performance did improve and Labor renewed the agency’s contract for another 2 years. Placement specialists at the three employment service offices we visited stated that they have no contact with Job Corps participants before their termination. It also appeared that the major placement emphasis was to register Job Corps participants in the employment service databank. While this did provide access to a major source of potential jobs, it was the same Page 23 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process B-272492 service provided to regular job seekers using the employment service and was not any kind of specialized assistance. As pointed out by the Chairman of the Senate Subcommittee on Employment and Training, Committee on Labor and Human Resources, during hearings on Job Corps in April 1997, a key to program success is the development of links to the business community. However, concerns were raised about whether Job Corps had developed such links. We noted that several of the centers we visited that had higher placement rates also had good relationships with local businesses. For example, one center had established a physical therapy program to meet the needs of local health facilities, and another center used temporary agencies as a springboard for their computer services trainees to gain access to the area’s computer industry. A third center was working on improving its work experience component to better match participants’ skills and abilities to the needs of local businesses so that more permanent hires would result. Labor regional offices are also exploring ways to improve links to the business sector. For example, one office has recently started a business roundtable of 18 employers in the region who discuss placement issues. Another regional office has begun a project to get local employers involved with training and placement. The idea is to have employers identify what they need in terms of training curriculum, equipment, and skills and then determine how the program can meet these needs. Recognizing the importance of employer links, Labor has launched a new school-to-work initiative within Job Corps to involve more employers in placing program terminees and to establish the basic framework for a school-to-work program. It started as a pilot program at three Job Corps centers and will be expanded to 30 centers this year. Further expansion will depend on the availability of funding. Labor’s program guidance to admissions counselors on two eligibility Conclusions requirements was ambiguous and incomplete. One of the program’s eligibility criteria—living in an environment characterized by disorienting conditions—has not been clearly defined in the statute, regulations, or Labor’s guidance. In addition, Labor has not provided adequate guidance regarding the requirement that participants have the capability and aspirations needed to complete and secure the full benefits of Job Corps. As a result, outreach and admissions contractors may not be enrolling the applicants who are most appropriate for the program. Page 24 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process B-272492 In the absence of specific Labor guidance, we noted that outreach and admissions practices varied among contractors. Those with higher participant retention rates tended to have better procedures to identify applicants who have the capability and aspirations to remain in and benefit from the program. A particularly effective tool in preparing applicants for Job Corps appeared to be preenrollment tours and briefings. Most admissions counselors expressed concern about the enrollment of 16- and 17-year-old applicants. Labor data confirm that these youths are more likely to drop out early for disciplinary reasons and less likely to be placed once they leave the program. Although Job Corps is a performance-driven program, the measures used to assess placement performance may not be meaningful and thus may not provide Labor with the information it needs to accurately assess placement contractor performance. Labor’s system for calculating a contractor’s placement performance included program terminees who were realistically unemployable. Determining what happens to every program participant is an important indicator of how well Job Corps is performing but not necessarily an appropriate measure of a contractor’s placement performance. Guidance related to another placement measure—the extent to which terminees were placed in training-related occupations—gave contractors such wide latitude when deciding whether a job was related to the training received that the validity of the measurement was questionable. In addition, the performance measure included terminees who received little vocational skills training and, therefore, were unlikely to be placed in jobs requiring an acquired skill. Labor is redesigning the methodology for determining job-training matches, which may help address some of these problems. However, any system would still be susceptible to manipulation by placement contractors without proper oversight and monitoring. We noted similarities in the procedures the placement contractors with higher placement rates used. One common characteristic was that they all had staff whose sole responsibility was placing program participants, whereas other contractors had the same staff performing outreach and admissions functions and providing placement services. In contrast, five of the six state employment service agencies were performing below Labor’s placement performance standard in program year 1995. We noted that between program years 1994 and 1996, Labor did not renew the contracts with 12 of the 18 state employment service agencies that had Job Corps placement contracts. None of the placement specialists we interviewed at the three employment service offices we visited had contact with students Page 25 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process B-272492 before termination, and it appeared that their primary effort was to register participants in the employment service databank. At these agencies, it appeared that Job Corps participants received similar services as regular employment service clients, raising questions as to why Job Corps is paying for services that could be obtained free of charge To help ensure that Job Corps’ resources serve the most appropriate Recommendations to participants, we recommend that the Secretary of Labor provide clear and the Secretary of Labor complete guidance on program eligibility criteria, ensuring that the guidance is consistent with the law, and provide better guidance to ensure that outreach and admissions contractors assess each applicant’s capability and aspirations to complete training and attain a positive outcome. Improvements are also needed to make the measures used to assess placement contractor performance more meaningful. Therefore, we recommend that the Secretary of Labor modify certain measures for placement contractors, including • eliminate from the placement pool participants whom contractors realistically could not or should not be expected to place, such as participants who were expelled for criminal or violent behavior; • replace the current job-training match system with one that captures realistic information and provide guidance to regional offices to ensure that the data are accurately recorded; • establish separate placement performance standards for participants with different levels of program accomplishment—for example, those who completed program requirements and those who dropped out early. In Labor’s comments on a draft of this report, the agency disagreed with Agency Comments our recommendation that it clarify and expand its program eligibility criteria in order to ensure that they are consistent with the law. Labor stated that our report lacked acknowledgment of the detailed specifications for eligibility requirements developed over the years in conjunction with the Office of Inspector General and that the eligibility, verification, and documentation requirements contained in its policy handbook are detailed and specifically related to guidance for Job Corps admissions counselors. Labor gave no indication of any formal action it planned to take on this recommendation. Although Labor expressed some concern with our remaining recommendations, it acknowledged that they Page 26 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process B-272492 have merit, warrant consideration, and identify actions that the agency would take in response to them. Labor’s specific concerns with our report are in three broad areas—adequacy of program eligibility guidance, potential effect of additional assessment procedures, and recommended changes to placement performance measures, including training-related placements. Labor also pointed out a number of items in the draft report that it believed should be modified or clarified, and we acted on these, where appropriate. For example, we clarified that our discussion of the ambiguity of program eligibility guidance related to only 2 of the 11 criteria. We also made a number of other technical changes to our report to respond to Labor’s comments. Following is a summary of Labor’s concerns and our responses. Labor’s full comments are printed in appendix VI. Eligibility Guidance Labor stated that our report lacked acknowledgment of the detailed specifications for eligibility requirements developed over the years in conjunction with the Office of Inspector General and that the eligibility, verification, and documentation requirements contained in its policy handbook are detailed and specifically related to guidance for Job Corps admissions counselors. Labor expressed concern with our characterization of the program eligibility guidance as inadequate. For example, regarding the lack of definition in Labor’s policy handbook for “limited job opportunities,” Labor commented that training conducted in program year 1995 for all admissions counselors included technical assistance material that defined this term as follows: “scarcity of jobs, commensurate with the skill levels of Job Corps-eligible youth and which has been designated as an area of substantial unemployment.” Labor added that “In essence, any applicant who lacks the specific skills required by the local labor market to obtain meaningful employment is a legitimate candidate for Job Corps.” Labor acknowledged that another eligibility factor—cultural deprivation—is not included in the policy handbook because more-specific factors—including (1) disruptive homelife, (2) unsafe or overcrowded dwelling, (3) disruptive community with high crime rates, and (4) limited job opportunities—were more useful to admissions counselors than the general term itself. Finally, Labor expressed concern with our discussion of the tool used in assessing another eligibility requirement—capability and aspirations. According to Labor, this Page 27 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process B-272492 assessment by its very nature must rely on the judgment of admissions counselors and determining aspirations is very difficult and challenging; Labor stated that the current assessment tool will be revisited and modified according to suggestions from regional offices and admissions counselors. We disagree that sufficient policy guidance defining “limited job opportunities” was provided to admissions counselors at a training seminar. Even if all admissions counselors at that time received such guidance, contractors and staff have since turned over. And, as mentioned in our report, admissions counselors we interviewed had different interpretations of “limited job opportunities,” indicating that something more is needed to ensure the consistent interpretation of limited job opportunities. Because Labor’s policy handbook was created to be “the single document containing all policy and requirements which would be: clear and concise, and up-to-date, and consistent with legislative provisions,” any definition of “limited job opportunities” that Labor develops should be incorporated into this policy handbook. In addition, the law states that environmental factors substantially impair an individual’s ability to succeed in training, not his or her ability to find employment. But Labor fails to explain the connection between its definition and the impairment of ability to succeed in training. And there is a separate eligibility requirement in the law that the applicant must “require additional education, training, or intensive counseling and related assistance in order to secure and hold meaningful employment . . . .” Labor’s interpretation of limited job opportunities appears to duplicate or at least overlap that separate requirement. Finally, Labor fails to explain how its definition satisfies the program regulations that stipulate that the environmental criteria are to be used in the context of residential versus nonresidential programs. Nowhere in its guidance does Labor mention this distinction. We also disagree that Labor provided adequate guidance regarding the term “cultural deprivation.” On the Job Corps application form, Labor not only lists each of the four factors it says define “cultural deprivation” as separate and distinct eligibility factors (any one of which would satisfy the eligibility requirement) but also adds the term “cultural deprivation” as a fifth factor that can be used to meet program eligibility. Guidance for completing the application form does not define this term and, as noted in our report, most of the admissions counselors we spoke with admitted that they did not know what the term meant. Furthermore, cultural Page 28 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process B-272492 deprivation cannot include disruptive homelife, as Labor says it does, because the law lists these as two separate environmental conditions. Regarding the eligibility requirement that participants have the capability and aspirations to complete and benefit from Job Corps, we agree with Labor that making such a determination is very difficult and challenging and, therefore, we believe that it is important that admissions counselors have guidance adequate to assist them in making these judgments. Furthermore, we agree with one regional official’s portrayal of the current assessment tool as a beginning step in providing guidance on this criterion. Accordingly, we support Labor’s decision to revisit this assessment tool and to obtain regional office and admissions contractors’ suggestions for improving it. Assessment Procedures With respect to assessment procedures, Labor agreed that Job Corps should not enroll youths who obviously have no desire to be in the program or capability to succeed and that assessing the interest and ability to benefit are important parts of the intake procedure. Labor also noted that participants’ leaving the program within the first 2 months is a cost that Job Corps must do whatever it can to minimize. However, Labor points out the need for a balance between this goal and the goal of serving youths who truly need the program, noting that overly strict assessment procedures could be a barrier to many severely disadvantaged youths. Furthermore, Labor states that the Congress clearly intended that Job Corps serve a severely at-risk population. Labor acknowledged that our report contained a number of positive suggestions (that is, “best practices”) that will be made available to outreach and admissions as well as placement contractors. Labor cautioned that the results of our analysis of characteristics associated with program retention could be misinterpreted because the report lacks the proper context. Labor further suggested that the detailed appendix related to this discussion be removed. Finally, Labor stated that the age data relating to participants who were 15 and 25 years old was inaccurate because Job Corps serves individuals aged 16 to 24. While we do not disagree that the program is to target persons most in need, the law states that the purpose is to assist youths who both need and can benefit from an intensive program. And the law requires that enrollees have the capabilities and aspirations to complete and secure the full benefits of the program. Several Labor regional directors commented on Page 29 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process B-272492 the importance of identifying applicants who are ready for Job Corps and can benefit from its training. For example, one director stated that with more than 6 million people eligible for Job Corps, admissions counselors have to identify those most likely to benefit from the program and that commitment should be first and foremost when they assess applicants. We also note that, in a previous report, we found that a key element of successful job-training projects was ensuring that participants are committed to training and to getting a job.16 Accordingly, we endorse Labor’s decision to make available to admissions contractors the procedures noted in our report that help identify the applicants who have the commitment and motivation to remain in and benefit from the program. We modified the report to provide our reasons for performing our analysis of characteristics associated with program retention and to highlight the limitations associated with our approach as well as the results. However, we do not believe the detailed appendix should be eliminated. In addition to describing our analysis and results in detail, it describes the related limitations. Regarding our mention of 15- and 25-year-old program participants being inaccurate, we obtained our data from Labor’s national database, which showed that less than 1 percent of program year 1995 enrollees were either 15 or 25 years old. We have added a relevant footnote. Placement Performance Labor expressed concern with our recommendation with respect to Measures placement performance measures that Job Corps eliminate from the contractors’ placement pool individuals who realistically could not or should not be expected to be placed, such as those expelled from the program for using drugs or engaging in violent behavior. Labor believes that the program has the responsibility to provide placement services to all participants and that it is not asking placement contractors to mislead or lie to employers during placement. Labor further commented that the current placement measure resulted from a recommendation by its Office of Inspector General that all participants who leave the program should be included in the placement pool, thus creating incentives to keep students as long as possible. Labor acknowledged that the points we made in this portion of the report merit serious consideration and, therefore, it will convene a workgroup to discuss our recommendations and examine the 16 Employment and Training: Successful Projects Share Common Strategy (GAO/HEHS-96-108, May 7, 1996). Page 30 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process B-272492 incentives and disincentives resulting from any proposed changes to the performance management system. Labor also acknowledged that our report contained “some good points” with respect to training-related placements but expressed concern about our use of hypothetical examples of questionable job-training matches and the lack of data to indicate the degree to which these occur. Labor also commented that the claim by a contractor about obtaining a job-training match for participants trained as bank tellers, secretaries, and welders and placed in fast-food restaurants is inaccurate, noting that the system does not permit such matches. Although Labor may not be asking its placement contractors to lie to or mislead employers when attempting to place individuals who realistically could not be placed, by holding contractors responsible for placing individuals expelled for criminal or violent behavior, the program may be encouraging such practices. We agree with Labor that determining what happens to every participant is an important indicator of program performance, but we do not believe that it is necessarily an appropriate measure of a contractor’s placement performance. We also acknowledge that establishing an effective performance management system is complex and agree with Labor that, before any changes are made to this system, the incentives and disincentives should be thoroughly examined, and we commend Labor for its proposed action. We used “hypothetical” examples of job-training matches to illustrate the wide latitude Job Corps permits. Labor data were not available to identify the extent of abuse, but as we mentioned in the report, most placement contractors we interviewed indicated that creativity is used when entering codes for placement jobs, and they expressed their concern about the validity of reported job-training match statistics. In response to Labor’s contention that the system does not permit job-training matches for participants trained as bank tellers, secretaries, and welders who obtain jobs in fast-food restaurants, we agree that if such jobs were reported as “fast-food workers,” the system would not permit a job-training match. But, as a contractor we spoke with pointed out, reporting such jobs in fast-food restaurants as “cashier” would be an allowable match for participants trained as bank tellers and secretaries, and reporting such placements as “machine cleaners” would be an allowable match for participants trained as welders. Page 31 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process B-272492 As arranged with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution of it until 15 days after its issue date. We will then send copies to the Secretary of Labor, the Director of the Office of Management and Budget, relevant congressional committees, and others who are interested. Copies will be made available to others on request. If you or your staff have any questions concerning this report, please call me at (202) 512-7014 or Sigurd R. Nilsen at (202) 512-7003. GAO contacts and staff acknowledgments are listed in appendix VII. Sincerely yours, Carlotta C. Joyner Director, Education and Employment Issues Page 32 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process Page 33 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process Contents Letter 1 Appendix I 38 Scope and Methodology Appendix II 44 Labor’s Job Corps Eligibility Criteria Appendix III 47 Labor’s Capability and Aspirations Assessment Tool Appendix IV 50 Analysis of the Relationship Between Participant Characteristics and the Likelihood of Remaining in Job Corps for at Least 60 Days Appendix V 59 Data Supporting Report Figures Page 34 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process Contents Appendix VI 61 Comments From the Department of Labor Appendix VII 72 GAO Contacts and Staff Acknowledgments Related GAO Products 76 Tables Table 1: Comparison of Elements of the Environmental Criterion 8 in Job Corps Program Guidance Table 2: Examples of Occupations Considered to Be Job-Training 19 Matches for Selected Vocational Training Programs Table 3: State Employment Service Agencies Having Placement 23 Contracts With Job Corps, Program Years 1994-96 Table I.1: Outreach, Admissions, and Placement Contractors We 39 Visited Table IV.1: Percentage of Participants Remaining in Job Corps for 51 at Least 60 Days by Selected Characteristics Table IV.2: Bivariate and Multivariate Effects of Various Factors 53 on the Odds of Remaining in Job Corps for at Least 60 Days Table IV.3: Final Multivariate Model of Effects of Various Factors 56 on the Odds of Remaining in Job Corps for at Least 60 Days Table V.1: Percentage of Program Year 1995 Terminees Not 59 Placed by Age Table V.2: Comparison of Program Year 1995 Placement Rates for 59 Selected Contractors Table V.3: Percentage of Program Year 1995 Enrollees Staying in 60 Program for Less Than 30 Days Table V.4: Program Year 1995 Placement Rates 60 Figures Figure 1: Percentage of Program Year 1995 Terminees Not Placed 14 by Age Figure 2: Program Year 1995 Placement Rates for Selected 17 Contractors Using Existing and Modified Methodology Page 35 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process Contents Figure I.1: Percentage of Program Year 1995 Enrollees Leaving 41 Within 30 Days for Selected Contractors Figure I.2: Program Year 1995 Placement Rates for Selected 42 Contractors Abbreviations AWOL absent without leave SPAMIS Student Pay, Allotment, and Management Information System Page 36 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process Page 37 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process Appendix I Scope and Methodology We designed our study to identify whether Labor’s policy guidance on eligibility was consistent with legislation and regulations and to collect information on contractors’ practices in enrolling individuals for the program and in placing them in jobs after they leave Job Corps. We reviewed Job Corps legislation as well as Labor’s program regulations and policy guidance on program eligibility, outreach and assessment of individuals for participation in the program, and placement of participants after termination. We also interviewed national and regional Job Corps officials and conducted site visits to 14 outreach, admissions, and placement contractors. We augmented the information we collected during the site visits with data from Labor’s Student Pay, Allotment, and Management Information System (SPAMIS), a database containing nationwide Job Corps data on all Job Corps participants as well as information on the outreach, admissions, and placement contractors for each participant. We analyzed program year 1995 enrollee data, the most recent full program year for which SPAMIS data were available. While we did not verify the accuracy of Labor’s SPAMIS data, we performed internal validity checks to ensure the consistency of the database. We performed our work between October 1996 and July 1997 in accordance with generally accepted government auditing standards. We visited 14 Job Corps outreach and assessment and placement Site Visits contractors. We selected the sites judgmentally to provide a mixture of contractors that were private contractors, Job Corps centers, and state agencies. We also selected contractors that provided both outreach and assessment services and placement services or that provided only one of these services. In addition, we considered past contractor performance in making our selections. We selected contractors located in 5 of Labor’s 10 regions to provide some regional management diversity and geographic dispersion and to allow us to visit multiple contractors during individual trips. In making our site selections, we identified contractors that had outreach and admissions or placement contracts with Labor during program years 1994 and 1995 and that were still under contract in program year 1996. This provided us with contractors that had multiyear program experience and were currently under contract with Job Corps. In order to select among the larger contractors, we included only contractors who enrolled or were responsible for placing at least 150 participants each year. We Page 38 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process Appendix I Scope and Methodology then ranked outreach and admissions contractors according to the percentage of program year 1995 enrollees who stayed in the program for more than 30 days and placement contractors according to the percentage of program year 1995 terminees placed in jobs, school, the military, or other training.17 We then selected contractors from among the top, middle, and bottom third of these rankings. Table I.1 lists the contractors we visited and their characteristics. Table I.1: Outreach, Admissions, and Placement Contractors We Visited Outreach and admissions Labor region Location Contractor Contractor type rankinga Placement rankingb IV Kittrell, N.C. Kittrell Job Corps Center Top third Top third Center Raleigh, N.C. North Carolina State Bottom third Not a placement Department of contractor Human Resources V St. Paul, Minn. Dynamic Private Top third Middle third Educational Systems, Inc./Hubert H. Humphrey Job Corps Centerc VI Dallas, Tex. Dynamic Private Did not meet Bottom third Educational selection criteria Systems, Inc. El Paso, Tex. Education Center Top third Top third Foundation/David L. Carrasco Job Corps Center New Orleans, La. New Orleans Job Center Middle third Bottom third Corps Center Oklahoma City, Okla. Oklahoma State Bottom third Bottom third Employment Security Commission Austin, Tex. Texas Workforce State Middle third Bottom third Commission IX Sacramento, Calif. Nero Support Private Not an outreach and Top third Services admissions contractor Sacramento, Calif. Sacramento Job Center Middle third Top third Corps Center San Jose, Calif. San Jose Job Corps Center Did not meet Top third Center selection criteria (continued) 17 One of Labor’s standards for measuring the performance of outreach and admissions contractors is the extent to which enrollees remain in Job Corps more than 30 days. The current standard is that 90 percent of enrollees arriving at the center will remain more than 30 days. Page 39 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process Appendix I Scope and Methodology Outreach and admissions Labor region Location Contractor Contractor type rankinga Placement rankingb Carson City, Nev. State of Nevada State Middle third Bottom third Department of Employment, Training, and Rehabilitation San Francisco, Calif. Women In Private Middle third Not a placement Community Service contractor X Seattle, Wash. Del Jen, Inc. Private Top third Top third a Ranking based on percentage of program year 1995 enrollees staying in the program for at least 30 days. b Ranking based on percentage of program year 1995 assigned terminees placed in job, school, military, or other training. c Hubert H. Humphrey Job Corps Center subcontracts with Dynamic Educational Systems, Inc., to perform outreach, admissions, and placement. We visited 11 outreach and admissions contractors from which varying percentages of program year 1995 enrollees left the program within the first 30 days. As shown in figure I.1, the percentages ranged from about 1 percent for one contractor’s enrollees to nearly 20 percent for another’s. Page 40 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process Appendix I Scope and Methodology Figure I.1: Percentage of Program Year 1995 Enrollees Leaving Within 30 Days for Selected Contractors Percentage of Enrollees 25 20 15 10 5 0 A B C D E F G H I J K Outreach and Admissions Contractors We also selected 12 placement contractors to visit that had varying success in placing Job Corps participants in program year 1995. As shown in figure I.2, placement rates ranged from about 54 percent to about 85 percent. Page 41 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process Appendix I Scope and Methodology Figure I.2: Program Year 1995 Placement Rates for Selected Contractors Percentage of Enrollees 100 80 60 40 20 0 A B C D E F G H I J K L Placement Contractors To obtain information on how contractors enroll individuals in Job Corps Interviews With and place them after their termination, we interviewed contractor Contractors personnel using a semistructured interview protocol. We asked outreach and admissions contractors questions related to their practices and procedures in attracting youths to Job Corps and in screening applicants. We also asked about their understanding and implementation of program eligibility criteria as specified by Labor and about their views on what affects program retention. We questioned placement contractors on their procedures in placing terminees in jobs, the military, or other training; the types of services they provided to terminees; and their practices when deciding whether a placement is a job-training match. We asked both groups of contractors about their views on current Labor performance standards related to recruitment and placements and their opinions on improvements needed in the Job Corps program. At three centers (David L. Carrasco, Kittrell, and Sacramento), we also interviewed Job Corps participants (approximately six from each center) to learn about their experiences when they were recruited for Job Corps and to obtain their views about the enrollment process. Page 42 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process Appendix I Scope and Methodology We interviewed Labor officials at national and selected regional offices to National and Regional obtain an overview of Job Corps enrollment, placement, and contracting. Job Corps Offices We also obtained information on Labor’s policy guidance on eligibility and how it relates to the Job Corps legislation; outreach, admissions, and placement contractors’ performance; and the program’s performance management system. In addition, we reviewed Labor’s Policy and Requirements Handbook, which was designed to include all program policy and requirements concerning eligibility criteria and policies and standards related to program enrollment and participant placement. We analyzed Job Corps participant retention data, reasons for termination, Data Analysis and placement information for program year 1995. We used 30-day retention data, part of Labor’s standard for evaluating outreach and admissions contractor performance, as a basis for selecting outreach and admissions contractors to visit. We expanded our analysis of retention beyond the 30-day standard and determined how many terminees left Job Corps within 60 days of enrollment in order to look at retention beyond the realm of outreach and admissions contractor performance. We also used one of Labor’s placement standards—the extent to which terminees are placed in jobs, the military, school, or other training—as a basis for selecting placement contractors to visit. Furthermore, we used the data from our analysis to supplement information obtained in discussions with admissions counselors and placement specialists. Page 43 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process Appendix II Labor’s Job Corps Eligibility Criteria Labor’s Policy and Requirements Handbook includes 11 eligibility requirements for enrollment in Job Corps. As noted below, three of these requirements—child care, parental consent, and Selective Service registration—do not apply to all applicants. 1. Age • at least 16 but not yet 25 years old at enrollment • no upper age limit for those who are physically or mentally disabled 2. Selective Service registration • all male applicants, who must sign a consent form authorizing the Selective Service System to register them automatically at age 18 3. Legal U.S. residency • a U.S. citizen or national, including naturalized citizens, or • a lawfully admitted permanent resident alien, refugee, parolee, or other alien permitted to accept permanent employment in the United States or • a resident of a U.S. territory or • a Canadian-born American Indian (“Jay Treaty Indian”) 4. Economic disadvantage • an individual receiving or member of a family receiving cash welfare payments, government-provided medical assistance, or food stamps or • a foster child for whom state or local government payments are made or a ward of the state or court or • an individual with physical or mental disabilities that present barriers to obtaining employment and whose own income meets the income criteria or • an individual or member of a family receiving total family income not in excess of the higher of the poverty level or 70 percent of the lower living standard income level 5. Requirement for additional education or training • a school dropout or • an individual in need of additional education, vocational training, or related support services in order to hold meaningful employment, Page 44 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process Appendix II Labor’s Job Corps Eligibility Criteria participate successfully in regular school work, qualify for other suitable training programs, or satisfy armed forces requirements 6. Environment • an individual living in an environment characterized by disruptive home life, unsafe, overcrowded dwelling; limited job opportunities; or disruptive community, high crime rates 7. Health history • a drug-free individual also free of any health condition (medical, mental, emotional, or dental) that represents a potentially serious hazard to youths or others, precludes participation in Job Corps with a reasonable expectation of successful completion followed by employment, or requires intensive or costly treatment • an individual with a number of other health conditions that Labor’s policy handbook specifies for consideration • all applicants with disabilities, who must be referred to the regional office for evaluation and determination of eligibility and assessed for appropriate assignment to centers equipped to handle each particular disability 8. Behavioral adjustment history • an individual free of behavioral problems so serious that the applicant cannot adjust to the standards of conduct, discipline, work, and training required or would prevent others from benefiting from the program or requires face-to-face court supervision or court-imposed financial obligations • a youth on probation or parole or under other supervision as a result of court action, who may be eligible only if the agency with jurisdiction states that the youth has responded positively to supervision, will permit the applicant to leave the local area or state, and will not require personal, face-to-face supervision during participation in the program • all applicants, who must sign the zero tolerance for violence certification 9. Child care • all applicants with dependent children who provide primary or custodial care, who must have established suitable child care arrangements 10. Parental consent Page 45 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process Appendix II Labor’s Job Corps Eligibility Criteria • youths who have not reached the age of majority as defined by state law, who must have parental or legal guardian consent to participate 11. Capability and aspirations to participate • all applicants, who must have the capability and aspirations to complete and secure the maximum benefits of Job Corps Page 46 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process Appendix III Labor’s Capability and Aspirations Assessment Tool Page 47 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process Appendix III Labor’s Capability and Aspirations Assessment Tool Page 48 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process Appendix III Labor’s Capability and Aspirations Assessment Tool Page 49 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process Appendix IV Analysis of the Relationship Between Participant Characteristics and the Likelihood of Remaining in Job Corps for at Least 60 Days In our analysis, we examined the relationship between the characteristics of Job Corps participants and the likelihood of their remaining in the program for at least 60 days. We used the data that were available from Labor’s Student Pay, Allotment, and Management Information System (SPAMIS) on the characteristics of the more than 68,000 participants enrolled in Job Corps during program year 1995. We performed a three-stage analysis resulting in a logistic regression model that used these characteristics to predict the odds of a participant’s remaining in the program for at least 60 days. While the information from our analysis provides some indication of whether participants with specific characteristics will remain in Job Corps for at least 60 days, we do not intend to imply that only individuals with these characteristics should be enrolled in the program or that outreach and assessment efforts should be focused on them. Rather, this information is a source of insight into early program attrition for Labor’s use in Job Corps management. We also recognize that being in the program for at least 60 days indicates only longevity, not necessarily success. For our initial exploration of the data, we selected the participant characteristics from SPAMIS that appeared to be conceptually relevant to the likelihood of remaining in the program for at least 60 days. These included age at enrollment, distance between a participant’s home and the assigned Job Corps center, and educational status. We first used crosstabulations to examine the relationship of these variables to whether the participant remained in the program for 60 days. The chi-square statistics from these analyses showed the variables that seemed to exhibit no relationship to 60-day retention and helped us eliminate certain characteristics and select a set of variables for further analysis.18 The set of variables we selected is shown in table IV.1. 18 In some cases, we suspected that variables that showed no relationship in bivariate analysis might be important in multivariate analysis. In these cases, we retained the variable for subsequent analysis. Page 50 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process Appendix IV Analysis of the Relationship Between Participant Characteristics and the Likelihood of Remaining in Job Corps for at Least 60 Days Table IV.1: Percentage of Participants Remaining in Job Corps for at Least 60 Remained at least 60Did not remain at least Significance Days by Selected Characteristics days 60 days of Characteristic Number Percentage Number Percentage chi-squarea Age at enrollment .00 15-17 19,148 72% 7,539 28% 18-20 21,650 75 7,109 25 21-25 9,815 78 2,829 22 Need for bilingual education .00 Spanish-English 1,067 87 163 13 Other-English 621 93 48 7 No need for bilingual education 48,941 74 17,272 26 Distance from home to center .00 Less than 50 miles 16,180 78 4,710 22 50-149 miles 10,979 73 4,108 27 150-299 miles 10,556 72 4,032 28 300 miles or more 10,301 72 4,029 28 High school diploma .00 No 38,663 73 14,591 27 Yes 11,966 80 2,892 20 Last school grade completed .00 0-8 7,324 68 3,368 32 9-11 31,148 74 11,177 26 12-15 12,157 80 2,938 20 Participant has dependents .92 No 44,380 74 15,325 26 Yes 6,210 74 2,139 26 Months out of school .00 0-2 13,026 75 4,245 25 3-6 9,425 74 3,383 26 7-12 9,659 74 3,436 26 Over 12 18,519 74 6,419 26 Prior conviction .00 No 48,671 74 16,665 26 Yes 1,958 70 818 30 (continued) Page 51 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process Appendix IV Analysis of the Relationship Between Participant Characteristics and the Likelihood of Remaining in Job Corps for at Least 60 Days Remained at least 60Did not remain at least Significance days 60 days of Characteristic Number Percentage Number Percentage chi-squarea Prior military service .00 No 50,217 74 17,385 26 Yes 412 81 98 19 Size of participant’s home city or town .00 Fewer than 2,500 5,361 74 1,937 26 2,500-9,999 5,627 72 2,238 28 10,000-49,000 10,301 74 3,667 26 50,000-249,000 8,781 74 3,160 26 250,000 or more 20,559 76 6,481 24 Resident at the center .00 No 7,384 80 11,814 20 Yes 43,245 73 15,669 27 a We used the chi-square test of independence to test for a relationship between remaining in the program at least 60 days and the student characteristic examined. The chi-square significance represents the probability that no relationship exists. For instance, the probability that no relationship exists between age and remaining at least 60 days is less than 1 in 100. With these variables, we then performed a bivariate logistic regression to estimate the effects of each individual factor on remaining in Job Corps for at least 60 days. The results from the regression models are stated as odds ratios, which tell us how much more likely participants with certain characteristics are to remain in Job Corps for at least 60 days than participants without those characteristics. We give a chi-square test of significance for each of these odds ratios. To calculate the odds of a specific group remaining in Job Corps for at least 60 days, the percentage remaining and not remaining must be determined. For example, 26,687 participants aged 15-17 enrolled in Job Corps during program year 1995. As shown in table IV.1, 19,148 of these participants remained in the program for at least 60 days, while 7,539 did not. The odds of 15-17-year-old participants remaining in the program for at least 60 days were calculated by dividing the number remaining (19,148) by the number not remaining (7,539). Therefore, the odds for this group’s remaining were 2.54, meaning that 2.54 individuals remained for every 1 who did not. Similar calculations for participants aged 18 to 20 and 21 to 25 yield higher odds of 3.04 and 3.47, respectively. Page 52 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process Appendix IV Analysis of the Relationship Between Participant Characteristics and the Likelihood of Remaining in Job Corps for at Least 60 Days The logistic regression model provides us with odds ratios that tell us how different the odds are for each group and whether the differences are statistically significant. For example, when the 15-17-year-old group is used as a benchmark for comparing the two other groups, the resultant odds ratios are 3.04/2.54 = 1.20 and 3.47/2.54 = 1.37 for participants aged 18 to 20 and 21 to 25, respectively. Thus, the odds of 18-20-year-old participants remaining in Job Corps at least 60 days are 1.20 times the odds of 15-17-year-old participants, and the odds of 21-25-year-old participants remaining are 1.37 times the odds of 15-17-year-old participants. Odds ratios that deviate from 1.0 the most, in either direction, represent the most sizable effects (for example, odds ratios of 0.5 and 2.0 represent effects that are similar in size, since 0.5 indicates that one group is half as likely as the other to remain in the program for at least 60 days, while 2.0 indicates that one group is twice as likely as the other to remain). We performed this type of bivariate analysis for each characteristic we selected. The resulting odds ratios are shown under the “bivariate results” column of table IV.2. Table IV.2: Bivariate and Multivariate Effects of Various Factors on the Odds Odds ratio of Remaining in Job Corps for at Least Bivariate Multivariate 60 Days Independent variable result result Age at enrollment 18-20 vs. 15-17 1.20* 1.16* 21-25 vs. 15-17 1.37* 1.27* Need for bilingual education Spanish-English vs. no need 2.31* 1.90* Other-English vs. no need 4.48* 3.13* Distance from home to center Less than 50 miles vs. 300 miles or more 1.34* 1.15* 50-149 miles vs. 300 miles or more 1.05 1.03 150-299 miles vs. 300 miles or more 1.02 1.00 High school diploma Yes vs. no 1.56* 1.02 Last school grade completed 0-8 vs. 12-15 0.53* 0.57* 9-11 vs. 12-15 0.67* 0.72* Participant has dependents No vs. yes 1.00 1.27* Months out of school 0-2 vs. over 12 1.06* 1.39* (continued) Page 53 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process Appendix IV Analysis of the Relationship Between Participant Characteristics and the Likelihood of Remaining in Job Corps for at Least 60 Days Odds ratio Bivariate Multivariate Independent variable result result 3-6 vs. over 12 0.97 1.17* 7-12 vs. over 12 0.97 1.06* Prior conviction Yes vs. no 0.82* 0.94 Prior military service Yes vs. no 1.46* 1.28* Size of participant’s home city or town 2,500-9,999 vs. under 2,500 0.91* 0.93 10,000-49,000 vs. under 2,500 1.02 0.95 50,000-249,000 vs. under 2,500 1.00 0.97 250,000 or over vs. under 2,500 1.15* 1.07* Resident at the center No vs. yes 1.47* 1.20* * Statistical significance = .05. Note: We also included two additional characteristics (race-ethnicity and gender) in our analysis to ensure that we had used all available and relevant data. However, for Labor to use these characteristics to distinguish between applicants would raise serious legal concerns because, in various rulings, the Supreme Court has made clear that using race or gender as a basis on which to treat people differently is unconstitutional unless stringent conditions are met. We have, therefore, not reported the coefficients for these characteristics. Although federal law generally prohibits discrimination in federally funded programs based on a third characteristic—age—this characteristic may be considered by Job Corps because the program legislation itself makes age a factor. After performing the bivariate analysis, we used the same set of variables in a multivariate logistic regression analysis, identical to the bivariate analysis except that it provides estimates of the effects of each characteristic on the likelihood of remaining in the program for at least 60 days while holding constant, or controlling for, the effects of the other characteristics. We included all factors (and levels of factors), even if their effects were not statistically significant in the bivariate analysis because, in some cases, effects that are suppressed in bivariate analysis emerge as significant in multivariate analysis. Similarly, effects that were significant in the bivariate analysis may be insignificant in the multivariate analysis. The results of the multivariate logistic regression are shown in column 2 of table IV.2 (“multivariate result”). As this column shows, when we entered all variables into the model, some variables and levels of variables had odds ratios that were not significantly Page 54 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process Appendix IV Analysis of the Relationship Between Participant Characteristics and the Likelihood of Remaining in Job Corps for at Least 60 Days different from the reference category.19 We dropped these variables or, in cases in which levels of variables were not significantly different from other levels within the same variable, we combined levels. For instance, in the multivariate model, the odds of remaining in Job Corps for at least 60 days for participants having a prior conviction were not significantly different from the odds of remaining for participants not having had a conviction. As shown in table IV.2, the odds ratio of .94 is not statistically significant. Therefore, we dropped this variable from subsequent analysis. Similarly, the odds of remaining for two levels of the variable “distance from home to center” (50-149 miles and 150-300 miles) were not significantly different from the odds of the reference category (over 300 miles). Therefore, we combined these two levels with the reference category to create a two-level variable for subsequent analysis. Thus, we included in the final model only the variables, and levels of variables, that were shown to be significant in the previous multivariate analysis. The results of this final model, as well as statistics related to how well the model performs, are shown in table IV.3. Model performance can be measured by the likelihood ratio method, which evaluates the probability of the observed results, given the parameter estimates. These results are shown under the –2 Log Likelihood (–2LL) entries in the note to table IV.3. As shown, the model containing the predictor variables shows an improved (smaller) –2LL compared with the model containing only the constant (that is, the model that assumes no differential effects resulting from individual variables). The model chi-square, which tests that the coefficients for all the terms in the model (except the constant) are 0 (that is, the null hypothesis), was significant at the .0000 level. 19 The reference category is the one category against which other categories are compared. For example, the reference category for age in table IV.2 is 15-17 years old. In this instance, 18-20-year-old participants are compared to those 15-17 years old, as are 21-25-year-old participants. Page 55 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process Appendix IV Analysis of the Relationship Between Participant Characteristics and the Likelihood of Remaining in Job Corps for at Least 60 Days Table IV.3: Final Multivariate Model of Effects of Various Factors on the Odds Independent variable Odds ratio of Remaining in Job Corps for at Least Age at enrollment 60 Days 18-20 vs. 15-17 1.15* 21-25 vs. 15-17 1.27* Need for bilingual education Spanish-English vs. no need 1.90* Other-English vs. no need 3.15* Distance from home to center Less than 50 miles vs. 50 miles or more 1.14* Last school grade completed 12-15 vs. 0-8 1.82* 12-15 vs. 9-11 1.41* Participant has dependents No vs. yes 1.27* Months out of school 0-2 vs. over 12 1.39* 3-6 vs. over 12 1.17* 7-12 vs. over 12 1.06* Prior military service Yes vs. no 1.28* Size of participant’s home city or town Over 250,000 vs. under 250,000 1.11* Resident at the center No vs. yes 1.20* * Statistical significance = .05. Note: We also included two additional characteristics (race-ethnicity and gender) in our analysis to ensure that we had used all available and relevant data. However, for Labor to use these characteristics to distinguish between applicants would raise serious legal concerns because, in various rulings, the Supreme Court has made clear that using race or gender as a basis on which to treat people differently is unconstitutional unless stringent conditions are met. We have, therefore, not reported the coefficients for these characteristics. Although federal law generally prohibits discrimination in federally funded programs based on a third characteristic—age—this characteristic may be considered by Job Corps because the program legislation itself makes age a factor. Model chi-square: chi-square, 1651.166; degree of freedom, 18; significance, .0000 Improvement: chi-square, 1651.166; degree of freedom,18; significance, .0000 Goodness-of-fit statistics –2 Log Likelihood initial model (constant only), 74294.86 –2 Log Likelihood final model, 72643.70 Goodness of fit, 64973.69 Page 56 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process Appendix IV Analysis of the Relationship Between Participant Characteristics and the Likelihood of Remaining in Job Corps for at Least 60 Days Results of Multivariate The results of our multivariate analysis revealed that older participants Analysis have greater odds of remaining in the program 60 days or more. When compared to 15-17-year-old participants, those aged 18 to 20 and 21 to 25 had odds of remaining that were 15-percent and 27-percent greater, respectively. In addition, we found that participants with 12 or more years of school had about 80-percent greater odds of remaining in Job Corps for at least 60 days than participants with 8 years or less of school. (See table IV.3.) We also found a relationship between the need for bilingual education and the likelihood of remaining in the program for at least 60 days. Of the variables we examined, the need for bilingual education yielded the highest odds ratio. Spanish-speaking participants needing bilingual training had odds of remaining that were almost twice the odds of those not needing bilingual education. Other non-English-speaking participants needing bilingual assistance had odds that were more than 3 times the odds of those not needing bilingual education. Limitations of the Analysis Our attempt to construct a model for predicting the characteristics of participants who are more likely to remain in the program for at least 60 days was limited by the variables available to us in Labor’s SPAMIS extracts. Most of these variables were demographic characteristics. We were unable to include in the analysis measures of such things as student ability, attitude, and motivation, as well as other characteristics that could potentially affect the likelihood of participants remaining in the program for at least 60 days.20 Additionally, the factors that proved to be the most useful predictors of remaining in the program for at least 60 days were characteristics of small subsets of participants. For example, there is evidence that participants in need of bilingual education are more likely to remain, but this group made up less than 3 percent of the Job Corps population. Similarly, participants who had completed 12 years or more of school had odds of remaining that were more than 80-percent greater than those of participants who completed 8 or fewer grades, but almost two-thirds of the participants were in neither of these groups. Consequently, while the model is very useful in predicting whether participants with specific characteristics will remain in Job Corps for at least 60 days, the model’s ability to predict 60-day retention for the program’s full population is limited because we 20 Although SPAMIS files include tests of adult basic educational skills, we were unable to include these scores in our analysis because of problems with the data in the files we received. Page 57 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process Appendix IV Analysis of the Relationship Between Participant Characteristics and the Likelihood of Remaining in Job Corps for at Least 60 Days found no large subgroups with great differences. Finally, in this analysis, we examined only main effects for the variables we investigated. An examination of the interactions among the variables might produce useful information and improve the predictive ability of the model. Page 58 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process Appendix V Data Supporting Report Figures Table V.1: Percentage of Program Year 1995 Terminees Not Placed by Age Age at termination Not placed (Data for Fig. 1) 16 44.9% 17 36.1 18 30.5 19 28.4 20 24.8 21 24.5 22 23.5 23 23.3 24 22.7 25+ 14.9 Table V.2: Comparison of Program Year 1995 Placement Rates for Placed using Placed using Selected Contractors (Data for Fig. 2) existing modified Placement contractor methodology methodology A 83.7% 87.3% B 65.1 74.3 C 64.3 76.3 D 54.2 65.6 E 76.5 86.6 F 63.4 69.2 G 73.5 87.1 H 78.3 82.9 I 73.2 82.3 J 80.1 84.5 K 59.6 62.2 L 84.7 89.5 Page 59 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process Appendix V Data Supporting Report Figures Table V.3: Percentage of Program Year 1995 Enrollees Staying in Program for Outreach and admissions contractor Enrollees Less Than 30 Days (Data for Fig. I.1) A 1.2% B 12.9 C 18.6 D 10.6 E 12.0 F 12.1 G 10.1 H 7.5 I 10.0 J 12.3 K 17.9 Table V.4: Program Year 1995 Placement Rates (Data for Fig. I.2) Enrollees Placement contractor placed A 83.7% B 65.1 C 64.3 D 54.2 E 76.5 F 63.4 G 73.5 H 78.3 I 73.2 J 80.1 K 59.6 L 84.7 Page 60 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process Appendix VI Comments From the Department of Labor Page 61 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process Appendix VI Comments From the Department of Labor Page 62 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process Appendix VI Comments From the Department of Labor Page 63 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process Appendix VI Comments From the Department of Labor Statement deleted. Now on p. 2. Now on p. 2. Page 64 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process Appendix VI Comments From the Department of Labor Statement deleted. Page 65 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process Appendix VI Comments From the Department of Labor Now on p. 3. Now on p. 5. Now on p. 6. Now on p. 6. Now on p. 7. Now on p. 8. Now on pp. 8-9. Now on p. 9. Page 66 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process Appendix VI Comments From the Department of Labor Now on p. 9. Now on p. 10. Now on pp. 11-15. Page 67 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process Appendix VI Comments From the Department of Labor Now on p. 13. Now on pp. 13-15. Page 68 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process Appendix VI Comments From the Department of Labor Now on pp. 16-17. Now on pp. 18-21. Page 69 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process Appendix VI Comments From the Department of Labor Now on p. 20. Now on pp. 24-26. Page 70 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process Appendix VI Comments From the Department of Labor Now on p. 38. See p. 50. Page 71 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process Appendix VII GAO Contacts and Staff Acknowledgments Sigurd R. Nilsen, Assistant Director, (202) 512-7003 GAO Contacts Wayne Sylvia, Evaluator-in-Charge, (617) 565-7492 In addition to the contacts named above, the following persons made Acknowledgments important contributions to this report: Thomas N. Medvetz, Wayne Dow, Deborah Edwards, Jeremiah Donoghue, Robert Crystal, and Sylvia Shanks. Page 72 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process Appendix VII GAO Contacts and Staff Acknowledgments Page 73 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process Appendix VII GAO Contacts and Staff Acknowledgments Page 74 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process Appendix VII GAO Contacts and Staff Acknowledgments Page 75 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process Related GAO Products Job Corps: Where Participants Are Recruited, Trained, and Placed in Jobs (GAO/HEHS-96-140, July 17, 1996). Employment Training: Successful Projects Share Common Strategy (GAO/HEHS-96-108, May 7, 1996). Job Corps: Comparison of Federal Program With State Youth Training Initiatives (GAO/HEHS-96-92, Mar. 28, 1996). Job Corps: High Costs and Mixed Results Raise Questions About Program’s Effectiveness (GAO/HEHS-95-180, June 30, 1995). (205329) Page 76 GAO/HEHS-98-1 Job Corps Recruitment and Placement Process Ordering Information The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. VISA and MasterCard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent. Orders by mail: U.S. General Accounting Office P.O. Box 37050 Washington, DC 20013 or visit: Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000 or by using fax number (202) 512-6061, or TDD (202) 512-2537. Each day, GAO issues a list of newly available reports and testimony. To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone. A recorded menu will provide information on how to obtain these lists. For information on how to access GAO reports on the INTERNET, send an e-mail message with "info" in the body to: email@example.com or visit GAO’s World Wide Web Home Page at: http://www.gao.gov PRINTED ON RECYCLED PAPER United States Bulk Rate General Accounting Office Postage & Fees Paid Washington, D.C. 20548-0001 GAO Permit No. G100 Official Business Penalty for Private Use $300 Address Correction Requested
Job Corps: Need for Better Enrollment Guidance and Improved Placement Measures
Published by the Government Accountability Office on 1997-10-21.
Below is a raw (and likely hideous) rendition of the original report. (PDF)