m aJ ul 0 CI 0 Z 2 Z u) n IL n 0 r 71 ul P 2 u m u m z 13 r 0 L( n- n ul . Personnel Appeals Board September 30,1999 The Honorable David M. Walker Comptroller General United States General Accounting Office Room 7000 441 G Street, N;W. -:.. Washington, D.C. 20543 > Dear Mr. Walker: Pursuant to the authority granted to it under the General Accounting Office Personnel Act of 1980,the Personnel Appeals Board has statutory responsibility to oversee equal employment opportunity at GAO. In exercise of that authority, the Board is issuing the attached report on. promotions of Banded employees at GAO. The Board’s report summarizes the findings of a study that examined the media time to promotion and rates of promotion for the five year period 1991-95. Employees in the study were differentiated by race, gender, national origin, age, and disability to discern whether there were any signZcant *parities amongthese groups in either the median time to promotion or rates of promotion at different levelsof the banding~system. We concluded that there were some disparities based on race, gender and age, but that the causes of these differences were not readily apparent from the statistics alone. Therefore, the Board has recommended that the Agency further investigate the disparities to determine whether additional steps need to be taken to ensure equal opportunity for its employees. Sincerely, Y(LLULQ Michael Wolf Chair attachment U. S. General Accounting Office . Suite560 l Union Center Plaza II l ; @ashington;D.C. 20548 l Phone (202) 512-6137 Contents .I. Letter Chapter I Introduction Background I. History of the Promotion Study II. Jurisdiction and Methodolo& . Chapter II 11 I. Bands 11 The Promotion II. Competitive Promotions (Banded Employees) at GAO 15 Process .- Chapter III 18 I. Time-In-Band 18 Findings II. Promotion Rates, Adjusted for Composition of “Best-Qualified” 20 Lists I 25 Chapter IV Conclusions- and Recommendations Appendixes Appendix I: Agency Comments 28 Appendix II: PAB Response toAgency Comments 32 Appendix IIk 6tatistics Lab, University of Maryland, Response to 35 Agency Comments Apperidix TV: Personnel Appeals Board 37 Tables Competition 1 9 Competition 2 9 Combined Results 9 Median Time to Promotion From I-D TO I-F 19 Median Time to Promotion From I-F to II 20 Table 1: Relative Odds of Proiotion by Gender, Race/National 22 Origin and Disability Controlling for Promotion Announcement and Year: Employees under 40 Only Table 2: Relative Odds of Promotion by Gender, Race/National 22 Origin and Disability Controlling for Promotion Announcement and Year: Employees 40 and Older only Page 2 . Contents : I’ ./- Table 3: Relative Odds of Promotioh by Age Group, Gender, 22 Race/National Origin and Disability, Controlling for Promotion Announcement and Year: Re-gional Offices Only Table 4: Relative Odds of Promotion ,by Age Group, Gender, 23 Race/National Origin and Disability Controlling for Promotion Announcement and Year: Headquarters Only Table 5: Relative Odds of Promotion by Gecder, Race/National .., 23 Origin and Disability Status Controlling for Promotion Announcement and Year: Regional Offices and Employees under ‘,‘I’ 40 only Table 6: Relative Odds of Promotion by Gender, Race/National 23 Origin And Disability Controlling for Promotion Announcement and Year: Regional Offices and Employees 40 and Older only Table 7: Relative Odds of Promotion by Gender, Race/National 24 Oi-igiri And Disability Status Contrdlling for Promotion Announcement and Year: Headquarters and Employees under 40 OdY Table 8: Relative Odds, of Prdmotion by Gender, Race/National 24 Origin and Disability Status ‘Controlling for Promotion Announcement and Year: Headquarters and Employees 40 and Older Only Table 9: Relative Odds of Promotion by Age Group, Gender, 24 Race/National Origin and Disability, Controlling for Promotion Announcement and Year: All Data Combined Page 3 b3 Chapter I Background : The Board’s study and report on promotions at GAO focuses on the Introductioti majority of Banded,.employees: those holding evaluator, evaluator-related, or specialist positions. Although attorneys at GAO are also Banded employees, they are grouped differently than the evaluators and were not .‘, included in the Board’s study.’ In 1987, the Personnel Appeals. Board (PAB or the Board) of the U.S. I. History of the ’1 General Accounting Office .(GAO or the Agency) published a report entitled Promotiofi Study EEO Oversight: Functional Study of GAO'S Career Ladder Promotional Process, That study. reviewed and andlyzed career ladder promotions2 at GAO from the beginning of fiscal year 1980 (October 1,198O) through the end of fiscal year ,1985 (September 30,1985) by race, sex, and national origin 3to determine whether there were significant differences (1) between the rates at which members of protected groups were promoted and (2) in the time members of protected groups spent in grade prior to promotion. At the unit level, the 1987 report focused primarily on evaluators; 4 agency-wide analyses were possible for evaluators, :evaluator-related employeeq5 attorneys, writer-editors, and employees grouped generally in an administrative category. Based on analysis of the data, the Board reached two general conclusions: (1) there were no significant differences inthe rates at which individuals in the protected classes were being promoted6 and (2) black evaluators were spending ‘For a description of the differences in the Bands, see the discussion in Section II of Chapter I, Methodology. “A career ladder refers to a job series that has one or more grade levels between the entry level and the full performance level. Appointment to a career ladder position is competitive; subsequently, the employee may proceed through the grades to full performance without further competition. 3Age and disability status were not included in the analysis. “Only evaluators met the study’s criteria of a minimum of 15 promotion actions per group in the units. GAO’s Oversight: Functional Study of GAO’s Career Ladder Promotional Process, p.4 (hereafter cited as Career Ladder Promotions). 5Employees occupying evaluator-related positions provide technical assistance and support in the audit function. %n analysis of rates of promotion involves looking at whether members of protected groups were promoted (or denied promotions) based on their membership in those groups and consistent with their representation in the applicant pool. It also includes comparisons between and among members of other groups. Page 4 Chapter I Background “significantly~7 more time in grade than were white evaluators. Specifically, the study concluded that during the time period of the study: [TJhere were statistical disparities associated with race in evaluator career ladder promotions. The disparities were particularly evident in cornpeons between Black and White evaluators.s : .( .; _, :_ Based on its analysis of the data and its conclusions, the Board made three specific recommendations designed to correct disparities in the career ladder promotion process. The Board recommended that the Agency: (1) identify any artificial barriers or impediments that may be responsible for disparities; (2) determine whether criteria used by units in making promotions are approPriate; and, (3) consider developing a training course on equal employment opportunity (eeo) for managers and supervisors involved inthe.promotion process.g In response to the draft report containing the Boards conclusions and recommendations, the Agency reported that it was taking immediate action to reduce the time-in-grade disparities in promotions revealed by the Board’s study. The actions included establishing guidelines for assessing individual performake’and potential; setting time-in-grade benchmarks; developing procedures for identifying and addressing the developmental needs of emplo&s whose time-in-grade exceeded the benchmarks; and developing an agency-wide database to monitor career ladder promotions.1o Prior to the issuance of the Board’s report, the Agency also created an qffice of Affirmative Action Plans and implemented training programs on equal employment opportunity and affirmative action responsibilities. With the drafting of a new GAO Order on promotions, selecting officials 7A finding is stat&icaUysignificantwhenit canbe demonstrated that the probability of obtaining that finding purely by chance is relatively low. The generally accepted “probability threshold” is 5 percent, i.e., the result tiould occur no more than 5 out of 100 times in a random sample with chance variations operating. *Ccweer Ladder Promo tiom, p. 5. The level of statistical significance for the time in grade that Hispanic evaluators spent as opposed to white evaluators was .09 (or 91%). In this instance, the Board reported this finding at the .09 level rather than the more commonly used .05 in order to call the agency’s attention to a potential eeo problem. The study revealed no “significant” differences between Asian and white evaluators. 9Zbid., p. 21. loLetter from ha Goldstein, Assistant Comptroller General for Operations to Carl Moore, General Counsel, PAB (August 20,1987)(hereafter cited as Goldstein Letter). Page 5 Chapter I : Background w&e charged with furthering “GAO’S goal that minorities and women be represented at the higher band levels of the work force.“” II; Jurisdiction and Methodology Jurisdiction The GAO .Personnel Act of 1980 charges the Comptroller General with maintainmg a personnel system that ,ensures that all appointments, promotions and assignments are made solely on the bases of merit and fitness. l2 That Act further directs the Board to exercise oversight authority over equal employment opportunity at ~~0.1~ In furtherance of that mandate and pursuant to its regulations, the Board reviews and evaluates GAO’S regulations, procedures, and practices and may require GAO to make changes it deems necessary.‘* Methodology In this study, the Board set out to determine whether members of any particular race, gender, nation~~origin, disability15 or age group received less favorable treatment in the award of promotions at GAO from January 1, 1991 through December 31, 1995.16 lrGA0 Personnei Supplement 2335.8 SUP,‘ch. 151-4. I231 USC. $732(b)(4). e 13M. at 1751. 141d.at $732(f)(2)(A). See, applicable regulations at 4 C.F.R. $128.91 and 28.92. The original study, resulting in the 1987 report EEO Oversight: Functional Study of GAO’s Career Ladder Promotional Process, was conducted by the Board’s Office of General Counsel (pAa/OGC) and submitted to the Board for review. It was shortly after the issuance of that report that the Board created a separate Office of EEO Oversight to carry out its statutory mandate. 15At GAO, disability status depends entirely on self-reporting. When new employees first report for duty, they are asked to complete GAO Form 164 “Self-Identification of Medical Disability.” During the past 10 years, the percentage of the GAO workforce reporting a disability has hovered around rive percent but this figure may not accurately reflect the actual population of persons with disabilities. ln 1996, GAO had 3,458 employees: 44 (1.27%) reported having a severe disability; 122 (3.53%) reported having a non-severe disability. These are relatively small numbers from which to attempt to draw conclusions. ‘@Ihe Board chose to study promotions at GAO for the years 1991-1995 for two reasons: (1) to track the earlier study 10 years later, and (2) to avoid coinciding, as much as possible, with the dates of the recent freeze on promotions at GAO (May, 1995 through March, 1997). Page 6 Chapter I : Background The Board is conducting this study as a follow-up to its 1987,study in which the Board found that black evaluators waited a significantly longer time to receive career ladder promotions than did white evaluators. Soon after that study, the promotional scheme at GAO for evaluators, evaluator-related employees and specialist was fundamentally changed by the “broad banding” of pay rates.” Evaluators, specialists and most attorneys at GAO no longer proceed by career ladder and/or competitive promotions through the General schedule (GS).18 Rather, they are grouped in three broad pay bands: Bsind I, e&ompassing the pay range from GS-7 through GS-12; Band II, encompassing the GS13 and 14 range; and Band III, being equivalent to GS-15.1gWithin a pay band, employees may receive pay increases related to performance without receiving a promotion. The result of broad-banding is that there is now one non-competitive promotion point (Band I-D to I-q20 and two competitive promotion points within illl evaluator’s career (Band 1-F to Band II and Band II to Band III). In this study, the Board examined the median time2’ to promotion at those three points and rates of prom&ion for a five year period (1991-95) to determine whether any statistically significant differences based on race, gender, national origin, age, or disability can be discerned and whether the prior racial disparities for time-in-grade persist under the new system.22 17Throughout the period of this study to the present, Banded employees have constituted approximately 70% of the GAO workforce. *@lie General Schedule is the pay schedule for most positions in the Federal Government. The Schedule is divided into grades of difficulty and responsibility, and it ranges from GS-1 through GS-15. An employee may progress up a career ladder without competition (e.g. GS-9 through GS-13) but after reaching the top of the ladder, the next level involves a competitive promotion (e.g. to a GS-14). Within each grade, there are ten rates of pay (steps). Step increases within grades are also awarded on a non-competitive basis. 5 U.S.C. $5332. ‘%e banding scheme for attorneys in GAO’s Office of General Counsel differs from that of the rest of the agency. Attorneys are generally grouped in two Bands that encompass grades 11-15. Bands I-D and I-F are comparable to GS-11 through 14; Band II attorneys are comparable to GS-15s. ‘OEmployees at the I-D level are “certified” to I-F, after meeting certain minimum requirements, without having to compete for the positions. See, discussion, supra p. 11. “Mean, median, and mode are statistical ways to describe a central tendency or the point where the population under study is centered. The mean is simply an arithmetical average of all of the values (sum of the values divided by the number of the values); the median is the middle value; the mode is the value that occurs most frequently within a set of variables. In other words, the median time to promotion is the center of the range: half of the employees’ time to promotion fell above the center number and half fell below. sThe Board contracted with the Statistics Laboratory at the University of Maryland, Colle’ge Park, to conduct the data analysis for this study and to prepare the tables and charts found in Parts I and II of Chapter III. Page 7 ’ Chapter I a; _’ Background :, Methodology: For this study, the Board looked atall promotions for Banded employees Time-In-Band Analysis during a five year period to determine the median time to promotion. The effects of race, age, gender and disability status were factored in separately at each promotion pointz3 The analysis of time-in-Band accounted for the fact that the actual time in Band was only known for a subset of employees. Because promotion histories for employees who were previously at agencies other than GAO were not available, the analysis of .the data did not consider promotion histories for GAO employees prior to. January 1; 1991..For; those who were already in Band on January 1, 1991, or for those who were not promoted until after December 341995, a minimum period of time in Band can be discerned. For example, an employee hired on July 1,1995, into Band I-D and not yet promoted as of December 341995, was in Band.for at least 6 months. 24 Standard techniques for this type of data where the entire promotion history is not known were developed to analyze an employee’s known history of promotions during a particular time period. The techniques also allow for computation of median time in grade as the time when 50% of the employees have already been promoted. It is also possible to test whether two or more groups have comparable distributions of time to promotion. Methodology: Promotion The Board compared the promotion rates of males and females, by age, Rate Analysis and by race, national origin, and disability status, after adjusting for the composition of the “Best-Qualified” (BQ) lists for each promotion competition. For this part of the aualysis, all employees were pooled and then separated (disaggregated) by age, by regional office versus headquarters, and by an age-region/headquarters combination. Only those employees who applied for promotions and made the BQ lists are part of this analysis.25 %e two Board promotion studies differ in the methodology for calculating time-ln-grade/Band. The 1987 study compared time-in-grade for all promotions within each career ladder within each unit Career ladders were also grouped on an agency-wide basis and the overall time-in-grade for each career ladder was measured by race, by gender, and by race/gender combinations. For that study, time-in-grade was standardized, i.e. the individual time-in-grade minus the mean time-in-grade of the subgroup divided by the standard deviation of the subgroup produced a standardized score. This allowed for the many different promotion criteria that were present due to the number of different grades and career ladders involved. The earlier report contains no discussion of the methodology used to .determine rates of promotion. %uch data are called right-censored. The techniques used in this report for censored data were developed to analyze lifetime data %&, Part II of Chapter II for a discussion of how BQ lists are compiled. Page 8 Chapter I : Background .I Y”, To analyze the’data on promotion rates, it was necessary to account for the fact that the BQ lists for various promotion competitions may have varied in their age, gender, race, national origin, or disability status mixes and promotion rates may have’ varied from one competition to another. ‘-. If one aggregates the simple numbers on all of the BQ lists, without taking into account the different race, gender, age, national origin, and disability status of those who compose each BQ list, the result would be promotion rates that ‘do not reflect the true rates of promotion for the various groups. For this reason, comparisons of promotion rates were adjusted or controlled for the varying compositions of the BQ lists. This “adjustment” was based on a standard statistical technique that allows for sampling error, called the ManteliHaenszel? statistic. For example, a given BQ list could be comprised mostly of males or have twice as many persons 40 and ‘over as compared to,‘Fersonsunder the age of 40. Using the technique in this study, the ‘analysts were able to compare the odds of promotion independent of the variances or different numbers within groups in the composition of each BQ list. :,, The following hypothetical data illustrate the problem for which the technique adjusts: Competition 1 (30 Vacancies) Male Female Promoted 10 20 Not Promoted 90 180 Total 1ocJ 200 Competition 2 (15 Vacancies) Male Female Promoted 10 5 Not Promoted 190 95 Total 200 100 Combined Results (45 Vacancies) Male Female Promoted 20 25 Not Promoted 280 275 Total 300 300 Page 9 Chapter I Background .. The first competition has a 10% overall promotion rate; the second competition is the more difficult with a 5.0% overall promotion rate. In the frrst competition, there were 30 vacancies: 10 men out of 100 were promoted, as.were 20 women out of 200. In the second competition, there were only 15 vacancies: 10 men were promoted out of 200 who applied, but there were only 100 women in the pool, 5 of whom were promoted. The combined numbers, however, show that, overall, 20 men of 300 were promoted (6.667% promotion rate) and 25.women of 300 were promoted (8.333% promotion rate). The combined numbers, standing alone, incorrectly suggest discrimination in favor of females.26 The numbers do not account for the fact that there were more males in the second and harder competition in which the same number of people (300) was competing for half as many promotions (15 versus 30). A correct analysis using the Mantel-Haenszel technique accounts or adjusts for the differences in both the promotion rates and the differing male-female mix in the preceding hypothetical. ‘GThe corresponding relative odds are 0.786: 20 males promoted/280 males not promoted versus 25 females promoted/ 275 females not promoted. Page 10 Chapter Il : The Promotion Process Generally, new evaluators and evaluator-related employees at GAO are I.Bands ” hired into Band I and assigned to the developmental level (Band I-D). At that level, employees are expected to become familiar with the policies and procedures associated with the evaluative work component of GAO. Typically, they are assigned to gather and analyze data, conduct research and interviews, and write segments of audit plans and GAO reports.27 Employees are “certified” to the full performance level of Band I (Band I-F). Unlike promotions to Band II and Band III, certification to full performance (I-D to&F) does not occur at a specified time of the year, but rather may take place whenever the unit head concludes that it is merited by the employee’s performance and the employee meets certain minimum requirements.28 Certification will normally result from the recommendation of the progress review group, which includes the Director for Operations, or Deputy Regional Manager, the Human Resources Manager, the supervisor and/or Assistant Director most knowledgeable about the employee’s recent performance. At the I-F level, employees are expected to perform the full range of evaluator functions. These include developing job plans, taking the lead in data collection efforts, selecting and applying the analytical method appropriate to a given situation, drafting chapters of GAO reports, and leading meetings with GAO officials to communicate the results of the work. Staff at this level are expected to perform all tasks with decreasing levels of supervision.2g Promotions from Band I-F to Band II and from Band II to Band III are competitive. Band II evaluators are expected to develop, evaluate, and review data collection efforts; to review and revise written products and consolidate them into reports; to be involved inthe planning function; and to ensure the completion of report processing. Their work products are presumed to be technically complete and are reviewed only for conformance to GAO policy. Band III evaluators initiate project proposals “‘Performance Appraisal System for Band I, II, and III Employees, Appendix VIII, p. 95 (October 1997) (hereafter cited as Appendix VIII). %These requirements include that the employee receive a six month progress review and that the employee serve at least 12 months in an evaluator or evaluator-related position. Prior service at GAO, at another federal agency or outside the federal government is creditable toward the 12-month requirement under certain circumstances. &, GAO Order 2540.1, Ch. 3, $1(b)(3). &Appendix VIII at 9697. Page 11 Chapter II : The Promotion Process and-direct their implementation; manage and supervise employees; and work under. very general guidance from a superior.3o As discussed in Section II of this chapter, promotions to the Band II or Band III levels normally occur as part of an annual assessment cycle, with all applications, selections and promotions being made at approximately the same time, agency-wide. For informational purposes, the following charts show the profile, by gender, race, and national origin, of evaluators, evaluator-related employees and specialists in Bands at GAO in 1994.31 30Appendix VIII at 97-100. 31At the same time, employees 40 and over constituted 31% of Band I; 72.5% of Band II; and 90.8% of Band III. Employees claiming a disability constituted 6.4% of Band I; 4.5% of Band Q and, 4.1% of Band III. Page 12 Chapter II : The Promotion Process 100 Banded Employees by Gender (percentages) 90 00 70 60 50 40 30 20 10 0 100 Band I (by rac+aiional,origi,n) percentage? , 90 .. 00 70 60 50 Page 13 : :’ Chapter 11 : The-Promotion Process II* competi&e The promotion process begins with an annual “needs determination” in which each office and division submits a proposal to the Assistant Promotions (Banded Comptroller General for Operations (ACG/OPS) stating the number of Employees) at GAO evaluator, evaluator-related, and specialist positions that it would like to fiu at each Band level and includes a brief justification of the need for these positions.32 The Needs Determination Committee, consisting of senior management officials working under the aegis of the ACG/OPS, considers the proposals and may also identify positions that may be filled by reassignment rather than promotion.33 After a decision has been made on the positions to be filled, information about eligibility for promotions, paperwork requirements, application procedures and deadlines is set out in a special supplement to the GAO Management News. A second supplement is then published containing ComIjrehensive job opportunity announcements listing the numbers, levels, locations, and types of positions to be filled. Some vacancies are only open to employees within the division or unit where the vacancy occurs; others are announced GAO-wide at management’s discretion. An employee wishing to be considered for one of the Band II or Band III vacancies must tile an application. All applicants must have at least 52 weeks in Band at their current level by the effective date of the promotion. Applicants for evaluator-related-positions must also meet selective placement factors ;and applicants for specialist positions must meet government-wide requirements for those positions, in addition to selective placement factors. ’ To apply for a promotion, an employee must submit an application for consideration; an employee profile which demonstrates that the employee has the requisite knowledge, skills, and ability (KSA) to perform at the ?his section describes the current promotion process. The process has changed substantively very little since 1991,the first year that the Board is studying. 33Currently,the Committee consists of the Assistant Comptroller General for Operations, the Assistant Comptroller General for Planning and Reporting, and the Deputy Assistant Comptroller General for Human Resources. Page 15 Chapter II The Promotion Process . higher Band34 a-statement of contributions and accomplishments; 35and performance appraisals for the current and preceding,two.years.36 Once the applications ‘are received for a promotion in a particular unit, a promotion pane1,37selected by the unit head, is typically convened to review the applications and,prepare a ranked list of applicants. (A promotion panel is not required if fewer than 10 employees request assessment for promotion in that unit.) The panel must include three unit employees, all of whom are at least ‘one Band higher than the employees who are being assessed. It may not include the selecting official. ” The promotion process at GAO is a relative ranking system. Candidates are compared to others in their group, and not against established benchmarks. Comparisons are based on performance, experience, and to a lesser extent, education, training, awards, and professional development that demonstrate important knowledge, skills and abilities at the next Band level. After the panel has prepared a ranked list of applicants, the chair of the panel decides how many employees to refer as “Best-Qualified” (BQ). The panel chair must follow the rank order established by the panel, but he or she has discretion as to how many. candidates to. refer and where to draw the cut-off ‘line. When drawing a cut-off, panel chairs are cautioned to consider factors such as natural breaks in scoring, as well as the number of opportunities available within the unit. During the time period of this study, the chair was permitted toconsider affirmative action goals when deciding how many names to forward. The agency’s current affirmative ’ action program, which covers hiring, promotions, separations, and training, places much of the responsibility for the success of the program on the unit managers. Specifically, they are charged with: (1) when requestedby ACOOPS, conducting appropriate barrier analysesregardinghiring, promotions, training, and separations,to determinewhy disparitiesexist in the unit and if %valuator KSAs are found in Appendix 2 to GAO Order 2335.8. They are listed for the following areas: planning; data gathering and documentation; data analysis; written communication; oral communication; workiug relationships, teamwork, and equal opportunity; and, supervision, appraisal, and counseling. %prior to 1994, employees submitted a Contribution Statement It is no longer a requirement but still may be submitted. “Band I employees applying for Band II positions submit their Band I-F appraisals only; I-D appraisals are not considered. GAO Order 2335.8 SUP, Appendix 1, I-l[a-51. 3TFormerly known as a management review panel. Page 16 t Chapter II The Promotion Process such dispariti& cannot be ekplainedon the basisof merit factors, developinga plan and taki$ stepsto correct any identified problems. . .% Unit managers are also held responsible for evaluating promotions on an on-going basis “to ensure that all employees are treated in an equitable manner.” The performance,of unit managers is evaluated on the basis of their equal employment opportunity efforts and results.39 All employees designated “Best-Qualified” are automatictiy considered ‘for any vacancies that occur in the same occupational series in their home unit. Any BQ candidate may also apply for any other vacancies where the j’ area of consideration is “GAO-Wide”, i.e. to ah qualified employees of GAO. 7 However; even employees who are not designated BQ in their home unit may apply for GAowide vacancies for specialist positions if they meet the qualifications. r The selecting official is presented with the BQ list containing the names listed in alphabetical order. Banks are not indicated on the BQ list. The ;-. selecting official may select any ‘candidate on the BQ list, or may make no selection at all. The selecting official may interview candidates prior to selection, but must interview all BQ candidates if any are interviewed. If the’selecting official does determine that interviews are necessary but the number of internal candidates on the BQ list is too large to allow for interviewing, he or she may convene a panel to winnow the list.4o Winnowing panels use the same process as is used to develop the original BQ list. Again, if the panel conducts interviews, everyone on the BQ list must be interviewed; Employees may request feedback about the promotion process. They may learn their BQ status, as well as their ranking, total score and distance between their score and the bottom score among the B&s. Merit selection files containing documentation of the qualilication, evaluation, and selection portions of the process must be maintained by the unit for three years.41 38Affiumative Action Program, U.S. General Accounting Office (1998), p.5. For purposes of analyzing promotion data to determine the existence of disparities in rates, the agency’s benchmarks are based on appropriate civilian labor force data as well as data on the current population of employees eligible for promotion. 3%id. ACGQS conducts statistical analysis on an agency-wide basis by race/ethnicity or gender to determine whether there are statistically significant disparities. If statistical disparities exist, ACG/O~S will work with unit management to correct any problems that are not merit based. 4”The Director of Operations and the Director of Planning and Reporting in each unit constitute the panel for winnowing purposes. 4’GAO Order 2335.8, ch. 3(l). Page 17 Chapter III Findings As noted previously;the Board’s 1987 study of promotions at GAO found I. ‘Time-In-Band that black evaluators spent significantly more time in grade than did white evaluators. Hispanic evaluators also spent more ,time in grade than white evaluators but not at a level determined to be statistically significant. The ,study found no significant differences based on gender alone (male versus female) but when comparing race and gender simultaneously, the same time-in-grade patterns prevailed (i.e., black females spent more time in I grade than white females). In this analysis, the distributions of time-in-Band for Bands I-D, I-F, II, III, were examined to determine the median time to promotion. The effects of race/national origin, age, gender and disability status were factored in separately at each of the three promotion points. From Band I-D to Band I-F At this non-competitive point, where promotion rests completely on the unit head’s determination that an employee has moved from the developmental level to full performance, the median time of promotion, overall, for white employees was faster than that of black, Asian, or ~ Hispanic employees. White employees spent a median time of 490 days in Band I-D prior to promotion; black employees spent a median of 546 days; Asian employees spent a median time of 560 days; and Hispanic employees spent a median of 574 days. There was no difference by gender. . Employees without disabilities spent a median time of 518 days in Band I-D; employees with disabilities spent a median time of 504 days. The largest gap was by age, with employees under 40 spending a median of 518 days in Band I-D compared with a median of 420 days for employees 40 and over. Page Chapter III Findings Median Time (in Days)30 Promotion From I-D to I-F By Race/National Origin Black 546 White 490 Asian 560 Hispanic 574 By Gender Female 518 Male 518 +,, Bv he -. -I- Under 40 518 40 and pver 420 BvI Disabilitv . Status Yes : 504 ‘_. No .- 518 .,. From Band I-F to Band II The ‘first part of this analysis shows the median time to promotion for employees who were under the age of 40. The figures for employees 40 and over are not shown because they exceeded five years in all demographic groups-beyond the five year period encompassed by the Board’s study. The second part of the analysis shows the median time to promotion for all Banded employees regardless of age. Of the employees under 40, white, Asian, and Hispanic employees spent a median number of 1,526 days in Band I-F; black employees spent a median of more than five years. Employees under 40 with disabilities spent a median of 1,806 days in Band 1-F; employees under 40 iyithout disabilities spent a median of 1,526 days. Males and females under 40 spent the same median time in Band I-F prior to promotion. However, when all age groups were combined, females fared noticeably better than males; the median time to promotion was approximately four and a half years for females and more than five years for males. The median time for all white, Asian, and Hispanic employees regardless of age was under five years; the median time to promotion for all black employees at this promotion point was more than five years. Page 19 Chapter III Findings Median Time (in Days) to Promotion -* .A k From I-F to II Under 40 .AII ! By Race/National Origin White 1,526 1,806 Black >5 years X5 years A&an 1,526 1,582 ~ Hispanic 1’,526 1,806 By Gender I Female 1,526 1,638 i Male 1,526 >5 years By Disability Status Yes 1,806 >5 years No 1,526 1,806 From Band II to Band III No disparities based on race, national origin, gender, age, or disability in time-in-Band were discerned at this promotion point. Due to the small number of promotion opportunities available, most of the Band II population never received any promotions during the course of,the Board’s study. On the average, the Band II population constitutes about two-thirds of Banded employees (1,746 after three promotion cycles); Band III’s, were 16 percent (436) of the Banded employees at the same point. ,.., The other prong of the 1987 Board study focused on rates of promotion. II. Promotion Rates, The 1987 study found nonsignificant differences based on race, sex or Adjusted for. national origin in the rates at which employees in those groups were Composition of promoted. “Best-Qualified” Lists In this analysis, the promotion rates of males and females, under 40 and 40 and over, by race, national origin, and disability, after adjusting for the varying compositions of the BQ lists for all of the competitive promotions, were compared.‘This analysis was performed from several perspectives: separating by age group (under 40 and 40 and over), by regional office versus headquarters, by an age-region/headquarters combination and by pooling all categories of employees. Tables 1 through 9 provide the “relative odds” for promotion by age, by regional office versus headquarters, by an age-region/headquarters combination and by pooling all categories of employees. Relative odds Page 20 Chapter III Findings c reflect the likelihood of being promoted between two groups under comparison, i.e. the relative success rate. In each odds ratio in each table, there are two groups being compared to each other (e.g., male/female; under 40/40 and over). Table II, for example, shows that the odds of promotion for men 40 and over is only 51 percent as likely as that for females 40 and over. Application of the previously discussed Mantel-Haenszel statistical technique also allows for the production of confidence limits. When two or more concepts are believed to be related, the relationship is confirmed with a “degree of confidence;” In the following tables, the confidence limits are reported at a 95 percent rate, i.e., the analyst is certain that the limits constructed will bracket the finding within 5 percent or, in some instances, a 99 percent rate, i.e. the limits will bracket the finding within 1 percent. In other words, the 95 and 99 percent confidence hmiti expressed in these tables represent the upper and lower boundaries or range of values. The analyst is 95 or 99 percent confident that v&Gn the interval (the range from lower to upper limit) lies the true mean of the population. A single asterisk in the Tables indicates a significant difference at the 0.05 level. This odds ratio. is statistically significant at the level of 95 percent confidence, limits. A doubled asterisk indicates a significant difference at the 0.01 level. This odds ratio is statistically significant at the level of 99 percent confidence limits. For the purposes of this report, any odds ratio without an asterisk is not considered statistically significant. Tables 1 and 2: Employees Tables 1 and 2 are based on separate analyses, disaggregating the Under 40 & 40 and Over promotion candidates on the basis of age. Among the younger employees, there are no differences due to gender, race/nation origin or disability. By contrast, among the older employees, the odds ofpromotion are only half as good for males as for females. There are no statistically significant differences due to race/national origin or disability. Chapter III Findings by Gender, Race/National Origin 95% Confidence (White v. Minoriiy)@‘and Disability, Group Odds Ratio Limits Controlling for Promotion Gender Mate 0.904 0.734 Announcement and Year: Employees v. Female 1.112 Under40 Only ‘. Race/Nat’1 origin White 0.812 0.640 v. Minority 1.029 Disability Status Yes 1.076 0.541 v. No 2.138 : . 4oFor this report, the term “minority” includes black, Hispanic, and Asian employees. Table 2: Relative Odds of Promotion by Gender, Race/National Origin 95% Confidence (White v. Minority) and Disability Grou,p Odds Ratio Limits Controlling for Promotion Gender Male 0.508** 0.378 Announcement and Year: Employees v. Female 0.682 40 and Older Only Race/Nat’1 Origin White 0.982 0.655 v. Minority 1.473 Disability status Yes 1.629 0.797 v. No 3.328 Tables 3 and 4: Regions v. Tables 3 and 4 are based on separate analyses, disaggregating promotions Headquarters in regional offices and in Headquarters. In both sets of promotion competitions, there was a-disparity in favor of yormger employees when compared to older employees and in favor of females when compared to males. Moreover, there is evidence that whites were less likely to be promoted than minority employees in the regional offices. Disability status had no effect on promotions. Table 3: Relative Odds of.Promotion by Age Group, Gender, Race/National .. 95% Confidence Origin (White v. Minority) and Group Odds Ratio Limits Disability, Controlling for Promotion Age 40 and Older 0.647** 0.483 Announcement and Year: Regional v. Under 40 0.866 Offices Only Gender Male 0.682** 0.517 v. Female 0.899 Race/Nat’1 Origin White 0.706* 0.505 v. Minority 0.987 Disability Status Yes 0.681 0.272 v. No 1.700 .I Page 22 Chapter III Finaings Table 4: l?elative,Cdds of Promotion. by Age Group, Gender, Race/National 95% Confidence Origin (White v. Minority) and Group Odds Ration Limits Disability Controlling for Promotion Age 40 and Older 0.552** 0.449 Announcement and Year: v. Under 40 0;678 Headquarters Only Gender Male 0.660** 0.539 v. Female 0.807 Race/Nat’1 Oriain White 0.828 0.651 v. Minority 1.054 Disability status Yes 1.509 0.873 v.No:. I. 2:607 Tables 5 Through 8: Age Tables 5 through 8 present separate analyses, disaggregating both on the Plus Region/Headguarters basis of Age and re&onIheadq~arters. The re+xlts reveal .no statistically $#ficant differences due to rake/national origin or &i&b& status in any of the four subsets of candidates. There were no gender differences among younger candidates nor among candidates in regional offices. Among older candidates in headquarters, males had a smaller statistically signilkant chance of promotion than females. Table 5: Relative Odds of Promotion by Gender, Race/National Origin 95% Confidence (White v. Minority) and Disability Group Odds Ratio Limits Status Controlling for Promotion Gender Male 0.871 0.615 Announcement and Year: Regional v. Female 1.235 Offices and Employees Under 40 Only Race/Nat’1 Origin White 0.708 0.478 v. Minority 1.049 Disability status Yes 0.439 0.060 v. No 3.230 Table 6: Relative Odds of Promotion by Gender, Race/National Origin 95% Confidence (White v. Minority) and Disability Group Odds Ratio Limits Controlling for Promotion Gender Male 0.549 0.292 Announcement and Year: Regional v. Female 1.030 Offices and Employees 40 and Older Race/Nat’1 Oriain White 1.360 0.545 Only v. Minority 3.396 Disability Status Yes 1.152 0.338 v. No 3.930 Page 23 Chapter III Findings by Gender, Race/National Origin 95% Confidence (White v. Minority) and Disability Group Odds Ratio Limit Status Controlling for Promotion Gender Male 0.922, 0.712 Announcement and Year: v. Female 1.194 Headquarters and Employees Under 40 I. Race/Nat’1 Origin White 0.878 ” 0.652 Only v. Minority 1.181 Disability Status Yes 1.289 0.619 v. No 2.685 Table 8: Relative Odds of Promotion by Gender, Race/National Origin 95% Confidence (White v. Minority) and Disability Group Odds Ratio Limits Status Controlling for Promotion Gender Male 0.495** 0.355 Announcement and Year: v. Female 0.691 Headquarters and Employees 40 and Race/Nat’1 Origin White 0.893 0.568 Older Only v. Minority 1.403 Disability Status j Yes 2.040 0.845 v. No 4.923 Table 9 - AU Data Table 9 compares the promotion rates of various groups, after adjusting Corhbined- for variations in the composition of the BQ list and for variations in the overall promotion rates in each of the competitions. The table reflects a disparity in favor of younger employees compared to older employees; a disparity in favor of females over males; and a disparity in favor of minority employees versus white employees. There are no si,@ificant differences in promotion rates between persons with disabilities and persons without disabilities. Table 9: Relative Odds of Promotion by Age Group, Gender, Race/National 95% Confidence Origin (White v. Minority) and Group Odds Ratio Limits Disability, Controlling for Promotion Age 40 and Older O-582*” 0.492 Announcement and Year: All Data v. Under 40 0.688 Combined Gender Male 0.667** 0.567 v. Female 0.785 Race/Nat? Origin White 0.784* 0.644 v. Minority 0.953 Disability Status Yes 1.175 0.736 v. No 1.875 Page 24 Chapter IV Conclusions and Recommendations ;,y., .‘& The Board’s 1987 study analyzed noncompetitive promotions that occurred at GAO during the 1980-1985 fiscal years. The current study analyzes promotions that took place between January 1,199l and ‘. December 31,1995. During the time between the two studies, GAO revamped its pay and grade. structure for evaluators, for those holding evaluator-related positions, and for most attorneys. Qne of the Board’s y goals for the current study was to determine whether the disljarity in time-in-grade between white and black evaluators, revealed in the previous analysis of non-competitive promotions, persisted in the new system. .’ j Between 1991 and 1995, white employees were promoted from Band I-D to Band I-F (a non-competitive promotion) at a median time of 56 days faster than black employees. White employees were also promoted a median of 76 days faster than Asian employees and a median of 84 days faster than Hispanic employees. From Band I-F to Band II (a competitive promotion), white, Asian and Hispanic employees under 40 spent a median of 1,526 days in Band prior to promotion compared to a median of more than five years for black employees under 40. The median time to promotion for all black employees at the Band I-F promotion point was also more than five years. When all age groups were combined, females spent a median of 1,638 days in Band; males spent more than five years. The median time to promotion for all employees 40 and over was more than five years. No differences in time-in-Band were discerned at the second competitive promotion point (Band II to Band Ill). It appears that the time-in-grade disparity revealed between black and white evaluators persisted into the Band system through the 1995 promotion cycle. This disparity was more pronounced at the non-competitive promotion point (Band I-D to Band I-F) and was also evident at the first competitive promotion point (Band I-F to Band II). Males were also promoted more slowly than females at the second promotion point. The second prong of the Board’s study concerned rates of promotions. In the earlier study, the Board found no disparities based on race, sex, or national origin in rates of promotion at GAO from 1980 through 1985. The most recent analysis, however, reveals disparities in rates of promotion at GAO during the five years studied. At headquarters, employees under 40, in Page 25 ! ‘: Appendix I Agency Comments ..u. .*:.. Assistant Comptroller General of the United States Washington, D.C. 20648 June l&l999 Ms.Gall Gerebenics Director,EEOOversight PersonnelAppealsBoard U.S.GeneralAccountingOffice 8201’ Street,N.E.,Suite560 Washington,DC. 20548 DearMs.Gerebenics: Thisis in responseto your March12,1999letter submittinga draft report from the PersonnelAppealsBoard(PAB) on Promotionsof BandedEmployeesfrom 1991- 1995.Our observationsandgeneralcommentsare asfollows. Attachment 1contains detailedcommentsandsuggestions to improvethe accuracyand clarity of the report ThePAB draft fast summsrlzes the findings of a 1987PAB report on careerladder promotionsfrom 1980through1985.It then examinesall promotions for banded employeesfor the timeperiod1991-1995 to determinethe mediantime for promotion andthe promotionrates of males andfemalesby age,race, national origin, and disabilitystatus.Webelievethat the methodof analysesand conclusionsin the report with respectto time-in-bandandpromotionratesare fiawed in several respects. With regardto mediantimefor promotion,the PAB iirst looks at the time for certificationfrom BandsID to Il?. It reportsthat white employeeshada shorter mediantime than did African Americans,Asian,or Hispanicemployees;employees with disabilitieshad‘ashortermediantime than non-disabledemployees;and employeesover age40hada shortermediantimethan employeesunderage40. PromBandlF’to BandII, the PAB finds that African American employeeshad a longermedianthee for promotionthan did white,Asian or Hispanicemployees.It alsoreachesa similarconclusionwhenageis factored into the race/nationalorigin analysis,with respectto African Americanemployeesunder40. According to the analysisthere wereno disparitieswith respectto the mediantime for promotionfrom BandsII to lII. In performingthe analysisof time-in-bandfrom BandID to IF, the PAB appearsto havegroupedall Bar&ID employeestogether,potentially biasingthe results. Band ID employeesarehired at different pay ratesbasedon their qualifications,which includeexperienceandeducation.Within the ID level there arethree qualification Page 28 Appendix I : Agenby Comments . pay levels,roughly equivalentto GS-7,GS9,and GS11paylevelsin the Executive Branch. For example,a candidatewith a master’sdegreeor equivalentwithout work experiencecanbehired at the GS-9paylevelwhile a candidatewith a Ph.D.andno work experiencecanbe luredat the GSll paylevel, Alternatively, a personwith a bachelor’sdegreeandno work experiencecanbe hir&at the GS-7level and a person with z&bachelor’s degreeandqualifyingwork experiencecanqualifyfor appointment at the GSll. Clearlythosestaff who start at the.higherpaylevelswithin BandID have a greaterlikelihoodof promotionto BandIF soonerthanstaff who start at a lower level. Failureto considerthis differencein hiring levelsoverlooksa slgnliicant factor that shouldbe includedin this analysis. .’ Furthermore, althoughthe summaryof time-in-band analysesnotesdifferences amongracial groupsfor BandsID to IF certiilcationsandBandsIF to II promotions, the report doesnot statewhetherthesedifferencesriseto the level of statistical significance.’Significancelevelsshouldbepresentedin orderto allowthe readerto judge the severity of anyreporteddifference. Indeed,it-isgenerallyrecognizedthat without statisticallysignificantdisparitieswhat actually happened in a decision making processcould reasonablybe attributedto randomvariationor chancewith respectto a protectedgroup. Of courseevenif statisticallysignificantdisparities exist, this doesnot necesarily leadto a findingof discriminationasthere may be merit basedexplanationsfor the results. The PAB notesthat its study isa follow-upto its 1987studyin whichit found that African Amermanevaluatorswaiteda significantlylongertimeto receivecareer ladder promotionsthan did white evaluators.However,the report fallsto note the signiBcantimprovementAfrican Americanshavemadein the lengthof time it takes to move from Bands ID to IF. Basedon the 1987PAB report,on average,anAfrican American wouldtake 155dayslongerthanwhitesto movefrom GS7to GS-12and 195dayslongerto move from GS-9to GS-12.The currentstudy statesthat it takes Afi-lcan Americans56 dayslonger,on average;to movefromID to IF -the equivalent of promotion from GS7 to 6512. Thisrepresentsan improvementof 99days(or 64%)in the GS7 to GS12 categoryand49days(or 4736)in the GS9to GS12 category. Moreover, there appearsto belittle demonstrable differenceamongAfrican Americans,Hispanicsandwhites,with respectto median thnefor promotionsfrom Bands IF to II. This fact, how.ever,is maskedby the combinationof daysandyearsin the related tables. The report statesthat themediantimefor promotionfrom Bands IF to II for African Americansis “morethan5 years,”while the timefor whitesand Hispanicsis “1,806days”. However,1,806daysis over 4.9years. % problem cased by the lack of data on signiiicance is compounded by the wording ued in ch. 3 and again in the concksion, where ?he rep-art states that no stadstically SigniGtit disparities were found al the Rands II to 111level. This tends to imply that results in the preceding discussions regarding tin&n-grade from Bands ID to IF, and iF to II, were significant. Page Appendix I : Agency Comments - With regard to the analyses of promotion rates, the PAB finds a disparity in promotion rstes in favor of younger employees compared to older employees, in favor of females over males, and in favor of mlnorltiesover white employees. In reaching this conclusion, the PAB used the Mantel-Haenssel test and aggregated the .promotion data for all 5 years in question, .1991-1995,‘andfor sll band levels. We believe this aggregation is inappropriate, and results in a misleading representation of the condition. In order to accurately reflect the selectiorrprocess when applying the Mantel- Haenszel test, each of the years and band levels should have been treated as independent decision processes and not have been aggregated. One of the assumptions underlying the use of thls test is that each observation, in this case selection from BQ list; is independent. While the existence of multiple applications by the same employee is not a significant problem when the data is examined year-by- year (as most employees do not apply multiple times in the same year), when cycles are combined for several years,,there clearly are multiple applications by the Same employee. Indeed, for the five promotion cycles in the years 1991-1995,some employees were on over 20 BQ lists over the course of the 5 years. One employee was on 28 such.lists. Multiple applications across several promotion cycles from many staff can distort the statistical analysis. This distortion can largely be overcome by analyzing each cycle separately. When we performed the Mantel-Haenszel analyses of promotion rates by individual year and by individual band level, we found no statistically sign&ant dlsparltks in any cycle from BQ to selection from, 1991-1995,with two exceptions. There was a statistically significant disparity in‘favor of women over men in 1991 at the Band ILI level and in 1993 at the Band II level. We note, however, that these disparities occurred,in only 2 of the’10 sets of data analyzed. (Each analysis consisted of a 1 year period for the Band II or Band LKLlevelsfrom 1991-1995.) Moreover, our data for the most recent yesrs of 19961998 shows no statistically slgniflcant disparities in favor of women. Thus, there is nothing to indicate a pattern of significant statistical disparities ln favor of women in promotions. In the report’s conclusion, the PAB compares the data from the 1987 report and concludes that the time-in-band disparity noted ln 1987for African Americans persisted into the banded system through the 1995promotion cycle, and that, as to other protected groups, certain dlsparlties exist&din promotion rates for the 1991-1995time perlod thatwere not evident in the 1987 study. We believe these comparisons are misleading. The 1987 report was based only on career ladder promotions, which were not competitive. In contrast, the PAP analysis of promotion data for 1991-1995merged non-competitive data (for Bands ID to IF’) wltb competitive data (for Bands IF to II). Therefore, because the comparisons involve different universes, the reporteris in concluding that the length of time for promotion for rage 3 Page 30 ‘. Appendix II PAB Response to Agency Commehts In commenting on the PAB'S study of promotions at GAO over a five year period (Appendix I), the Assistant Comptroller General for Operations made the following observations: (1) employees in Band I-D should have been grouped by three separate pay levels for purposes of the time-in-Band analysis; (2) levels of sigiziscance should be presented in every discussion of differences; (3) there is little difference in time-in-Band with respect to median times to promotion from Band I-F to Band u[; (4) the Board failed to note improvements that AlEcan American evaluators have made in time-in-Band (5) in analyzing rates of promotion, the Board should have reviewed the data by cycle/year rather than aggregating it; and, (6) the Board should not have merged non-competitive promotion data (Band I-D to I-F) with competitive promotion data (Band I-F to II and Band II to III). The Boards responses to those points follow: (1) The criteria for certification from the developmental level of Band I (I-D) to the Full-Performance (I-F) level are found in GAO Order 2540.1 The pertinent part provides that certification “may occur at any time after the first 6month progress review and the employee has completed 12 months in an evaluator or evaluator-related position.” Ch. 3, §&73(a). Although employees may be hired at different pay levels within Band I-D, all are equally eligible for certification once the time criteria are satisfied. The data provided to the Board by the Agency in the initial stages of its study were not separated by pay levels for any Band. The Agency did not provide any data to the Board to support the claim made in its comment letter that staff hired at a higher level of pay have a greater likelihood of earlier promotion to Band I-F than those hired at a lower pay level. (2) The Boards study sets out the time-in-Band numbers for I-D to I-F and from I-F to II. The numbers show that some members of some protected groups spent longer in Band at those points than members of other protected groups. The report does not attempt to measure the statistical significance of the numbers. At the Band II to Band IlI promotion point, the Board notes that it found no disparities. Again, the Board does not attach any statistical significance to this finding. To eliminate confusion, the report no longer describes the lack of disparities at the Band II to Band III level as statistically significant. (3) The analysis of time-in-Band took into account that the actual time in Band wti only known for some employees. For those who were already in Band at the beginning of the study (January 1,199l) or for those who were not promoted until after the end of the study (December 31,1995) only a minimum period of time in Band can be discerned. The techniques used in Page 32 Appendix11 : PAR Response to Agenizy Comments the,Board’s report were developed to analyze an employee’s known history of promotions during a particular time period. They allow for computation of median time in Band as the time when 50% of the employees have already been promoted. The Agency’s point that there is “little‘demonstrable difference” ivith: respect to median time-in-Band at the I-F to II point is an oversimplification of the data. The numbers for time-in-Band at the I-F to IIpromotion point show that 50% of the white employees under 40 were promoted within 1,526 days (4.2 years); fewer than 50% of black employees under 40 were promoted from Band I to II during the entire five year study period. At that same promotion point, for all employees regardless of age, 50% of white Banded employees had been promoted by the time that 1,806 days had elapsed (4.9 years); again, fewer than 50% of black employees were promoted within the entire five year -period encompassed by the study. Because of the five year period of the ‘study, the,Board is unable to determine the precise median for black employees in each of these categories; we can only say that it exceeded five years. : (4) This study was conceived as a follow-up to the 1987 report only in the sense that the Board was revisiting timing and rates of promotions at GAO over a subsequent five year period.> Shortly after the Board published its first report on career ladder promotions for evaluators and evaluator-related employees, GAO completely revamped its pay system for those employees, grouping them into three broad pay bands. Due to the fundamental changes that banding caused in the promotional scheme, the Board could not track or compare data on a category-by-category basis. Rather, the Board examined promotions within the confines of the new system to ascertain whether patterns that had been discerned previously persisted under the new system. In addition, the current study added age as well as disability analysis. (5) GAO contends that aggregating the promotion data for the five year period was inappropriate because multiple applications by the same employee over several promotion cycles can distort the statistical analysis. The Agency noted that some employees were on more than 20 BQ lists over the five-year period. Analysis of the data shows, however, that nearly 97 percent of the employees on BQ lists appeared on five or fewer; nearly 70 percent appeared on just one or two lists. In addition, further disaggregating the data beyond what was done in the analysis in this report decreases the availability of comparative data which, in turn, reduces the chance of identifying differences which may be present. In response to the Agency’s comments, the Boards contractor, the Statistics Page 33 Appendix II PAR Response to Agency Comments Lab at the University of Maryland, analyzed the data by year. The Lab noted that it found no significant differences in the odds ratios when the analysis was split by year. (6) With respect to its Time-in-Band analysis, the Board did not merge ;’ non-competitive and competitive promotion data, but presented it separated by promotion points both in its Findings section and its Conclusions section. The analysis of rates-of promotion is based solely on selections from BQ lists. As there are no BQ lists involved in the non-competitive promotions (I-D to I-F’), they were not included in the analysis of rates. Finally, in a separate communication, the Agency expressed dissatisfaction with the charts found at pages 13-14 in the report, questioning why data was presented for only one promotion cycle. The Agency indicated that, had the Board presented similar charts for each of the five years in the study, they would show increased representation of women and minorities in Bands II and III over the course of the study. The Board’s use of the charts was purely informational and not conclusory. The Board’s intention was to present readers with benchmarks that provide a general description of the composition of the GAO workforce as it existed midway in the Boards study. ‘. Page 34 Appendix III Statistics Lab, University of Maryhnd, Response to Agency Comments The agency’s comments on page three, paragraph one oversimplify the results of the analysis. The findings were in fact the following: 1. When all-employees are considered together, there is a disparity in promotion rates favoring younger employees over older, females over males and nonwhites over whites. 1 2. When promotion candidates are. disaggregated on the basis of age, we find no disparkies among the under-40 employees. The disparities are concentrated in the 40-plus group. The comments reveal a misunderstanding of the Mantel-Haenszel test. This test does indeed combine data from promotion competitions. However, the inferences areconditional on knowing the numbers of promotions granted in each competition and the demographic characteristics of the members of the BQ group for that competition.’ II-I other words, the numbers of promotions and characteristics of the BQ lists are treated as “fixed” in this analysis. Under >thk condition, the only characteristic which varies from competition to competition is the proportion of protected individuals who are promoted at each competition. Moreover, this proportion varies independently Tom competition to competition once the promotion rates and BQ list demographics are held fixed. The Mantel-Haenszel test does not require independence, but only conditional independence given promotion rate ,and BQ demographics of each competition2 The Mantel-Haenszel procedure tests the hypothesis that, holding all other factors fixed, the chance of a protected individual’s promotion in a given competition is identical to that of an unprotected individual’s promotion in the same competition. In other words, even though the composition of the BQ lists and the overall promotion rates may differ from competition to competition, the hypothesis says that the common odds ratio is 1. The GAO comments suggest that the methodology is invalid because some persons compete for multiple promotions. In fact, by looking only at the conditional odds ratio of promotion given the total numbers of promotions and the demographics of the BQ list, the test properly adjusts for employees who compete for several promotions. ‘Agesti, Alan “An Introduction to Categorical Data Analysis” (1996) New York: J. Wiley, p. 231. ‘Ibid. Page 35 Appendix III Statistics Lab, University of lkryland, Response to Agency Comments The Mantel-Haenszel- test is frequently used to compare life data from two separate groups (experimental and control) of patients with chronic disease. In such studies, the patients at risk are compared whenever a death occurs in order to compare’the odds of death in the experimental and control groups. This means that the same subjects contribute to many computations of odds ,ratios. There are certainly many more multiple examinations of patients in this clinical setting than in the GAO application, where very few employees were involved.in more than five competitions. The authors of the comments chose to disaggregate the data by year and band. They do not provide any substantive reasons for disaggregation. As outlined above, there is also no statistical justification for disaggregation, since the Mantel-Haenszel test accounts for differences in promotion rates and BQ demographics. We also tested whether the odds ratios were equal across competitions using the standard Breslow Day test.3 The results overwhelmingly supported the hypothesis that odds ratios did not vary significantly from competition to competition, thereby justifying our combined Mantel-Haenszel analysis. Nevertheless, we also performed separate analyses by band, obtaining the same findings as in the combined analysis. We did not attempt to reproduce the erroneous analyses based on d&aggregating by both.band and year. Unnecessary disaggregation reduces the power of statistical tests, thereby .making it difficult to distinguish genuine effects fiorn sampling error. In plain words, breaking up the data into little subsets reduces the chance of finding disparities if they do exist. 31bid., p. 238. Page 36 Appendix IV Personnel Appeals Board Personnel Appeals 1 M. Gail Gerebenics, Director, EEO Oversight Board Staff *Term expired Page 37
Personnel Appeals Board: Promotions of Banded Employees, 1991-1995
Published by the Government Accountability Office on 1999-09-30.
Below is a raw (and likely hideous) rendition of the original report. (PDF)