oversight

Performance Management: Appraisal and Promotion Results at the U.S. Customs Service

Published by the Government Accountability Office on 1990-05-18.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

              Iteprcssentatives

I
1   -
1 May 1!~90
              PERFORMANCE
              MANAGEMENT
              Appraisal and
              Promotion Results at
              the U.S. Customs
              Service
                                              d
                                              d




                                  IllIllMII
                                    141591
             United States
GAO          General Accounting Office
             Washington, D.C. 20648

             General Government   Division

             B-236938

             May 181990

             The Honorable William D. Ford
             Chairman, Committee on
               Post Office and Civil Service
             House of Representatives

             Dear Mr. Chairman:

             This report responds to your request, made on behalf of the Subcommit-
             tee on Civil Service, that we test the feasibility of analyzing relation-
             ships among employee demographics, performance ratings, and
             promotions. We subsequently agreed with the Subcommittee that due to
             the exploratory nature of this study we would limit our work to one
             agency with automated personnel information. The agency we
             selected-the U.S. Customs Service-had computerized personnel data
             available for the fiscal year 1987 performance rating period.

             The Civil Service Reform Act of 1978 (5 U.S.C. 4302) requires federal
             agencies to develop performance appraisal systems that provide feed-
             back to employees on the extent to which their job performance meets
             management expectations. Among other things, the system is intended
             to help managers determine how well employees are performing and to
             provide information for such management decisions as promotions and
             awards.


             As the principal border enforcement agency, the US. Customs Service is
Background   responsible for safeguarding U.S. agriculture, health, and security and
             for curbing the smuggling of narcotics and contraband into the country.
             In addition to working at its headquarters location in Washington, D.C.,
             employees of the Customs Service are spread across 7 regional offices,
             44 district offices, and approximately 300 points of entry. The majority
             of the agency’s field staff work in white-collar law enforcement posi-
             tions, such as Customs Inspector and Investigator.

             The focus of this report is on Customs employees in two types of pay
             plans-General Schedule (GS) and General Management (GM). GS, which
             consists of employees in nonsupervisory positions in grades 1 through
             14, comprised 13,197 permanent, full-time employees at U.S. Customs
             during fiscal year 1987,66 percent of whom were male and 73 percent
             of whom were white. GM, which consists of employees in managerial and




             Page 1                          GAO/GGPQ&#l   Appraisals   and Promotions   at Customs
                        B-236938




                        supervisory positions in grades 13 through 15, comprised 1,216 employ-
                        ees at Customs, 89 percent of whom were male and 90 percent of whom
                        were white.

                        The Customs Service uses a five-tiered rating scale to assess the per-
                        formance of its GS and GM employees. Emuloyees overall job perform-
                        ance is rated annually at one of the following levels: outstanding, highly
                        successful, satisfactory, minimally successful, and unacceptable.

                        During fiscal year 1987 less than 1 percent of Customs employees were
                        rated below the satisfactory level. Other studies have shown that infre-
                        quent use of low performance ratings is a consistent practice among fed-
                        eral agencies.’


                        The objectives of this assignment were to test whether and to what
Objectives, Scope,and   degree certain federal employee characteristics were related to perform-
Methodology             ance ratings and competitive promotions. The U.S. Customs Service was
                        selected for study because it possessed computerized personnel data
                        that were relatively current and complete. Also, performance ratings
                        were fairly well distributed across three ratings levels (satisfactory,
                        highly successful, and outstanding).

                        The US. Customs Service provided us with a computer tape containing
                        personnel data for fiscal year 1987, which was the most recent perform-
                        ance rating period for which data were available at the time we began
                        this study. The computer tape included demographic, ratings, and per-
                        sonnel action data on permanent, full-time GS and GM employees at Cus-
                        toms as of July 6, 1987. It also covered personnel actions pertaining to
                        these employees through August 25, 1988. We verified the accuracy of
                        the Customs automated data for a random sample of 100 cases and con-
                        sidered the error rate to be reasonable-less than 2 percent. We were
                        unable to verify ethnicity data, as this information was absent from
                        individual personnel files.


                        ‘A 1988 governmentwide study by the Merit Systems Protection Board, based on 1984 performance
                        ratings data for 846,630 GS and GM employees, found that less than 1 percent of employees received
                        either minimally successful or unacceptable ratings-Toward Effective Performance Management in
                        the Federal Government, Merit Systems Protection Board (Washington, DC.: July 1988). Also, a study
                        of Performance Management and Recognition System (PMRS) employees in the General Services
                        Administration found that performance ratings below fully successful were given to 1 percent of 487
                        employees in 1986, .2 percent of 398 employees in 1986, and .9 percent of 352 employees in 1987-
                        d. I,. Perry, B. A. Petrakis, and T. K. Miller, “Federal Merit Pay, Round II: An Analysis of the Perform
                        ante Management and Recognition System,” Public Administration Review, Vol. 49, No. 1, .Jan/Feb
                        1989.



                        Page 2                                     GAO/GGD-99-40     Appraisals   and Promotions    at Customs
                   B-230938




                   We used loglinear statistical techniques to analyze the odds of receiving
                   higher versus lower ratings and the odds of being competitively pro-
                   moted depending on various employee characteristics. We analyzed rat-
                   ings and promotions in relation to employees’ age, sex, ethnicity, work
                   location, grade level, and time in grade for both GS and GM employees. As
                   a measure of time in grade, we used “step in grade” in the GS analyses
                   and “years in grade” in the GM analyses. We took this approach because
                   step data were not available for both groups.

                   Using loglinear techniques allowed us to determine which factors were
                   statistically significant at the .05 level in predicting rating levels and
                   promotions and how these factors interacted with one another. The
                   strength of this particular statistical approach is that multiple variables
                   can be analyzed simultaneously, thereby enabling us to examine com-
                   plex relationships in the data. The technical appendix provides more
                   detailed information on our methodology, the loglinear models tested,
                   and the results obtained.

                   Results from our work cannot be generalized to other federal agencies,
                   nor can they be used alone to draw conclusions about Custom’s person-
                   nel management practices. Our work was done from January to Septem-
                   ber 1989 and in accordance with generally accepted government
                   auditing standards.


                   We found that age was directly related to GS employee ratings, whereas
Results in Brief   other variables (sex and ethnicity, sex and grade, and step in grade and
                   grade) had indirect relationships2 With respect to GM performance rat-
                   ings, we found that age, grade, and years in grade each had direct rela-
                   tionships with ratings, whereas sex and ethnicity had indirect
                   relationships.

                   For promotions, we found mostly indirect relationships in GS employee
                   data. Indirect relationships included sex and age, sex and location, rat-
                   ings and location, and ratings and grade. Only step had a direct relation-
                   ship with GS promotions. In contrast, we found only direct relationships
                   with GM promotions. In the GM group, ethnicity, sex, rating, age, grade,


                   “A direct relationship exists when, after we control for the influences of other factors, the outcome
                   variable (e.g., performance rating) depends on the category of the independent variable (e.g., field or
                   headquarters). An indirect relationship exists when, after we control for the influences of other fac-
                   tors, two or more variables jointly influence the outcome (e.g., being promoted depends on whether
                   one is a black male or female or a white male or female).



                   Page 3                                     GAO/GGD-90-40     Appraisals   and Promotions    at Customs
                        B-236938




                        years in grade, and location each had a direct relationship with the odds
                        of being promoted.

                        Although the findings should not be used to draw cause-effect conclu-
                        sions, they are useful in revealing what combinations of factors were
                        significantly related to ratings and permanent competitive promotions
                        at Customs. Thus, the findings can provide useful insights into the
                        potential dynamics of ratings and promotion decisions and indicate
                        directions for more in-depth work.


                        Our analyses of GS data sought to determine which of our test factors
Statistically           differentiated between those 5,654 employees (58 percent) rated out-
Significant             standing or highly successful and those 4,109 (42 percent) rated satis-
Relationships Existed   factory. The GM analyses sought to determine which factors
                        differentiated among the 175 employees (16 percent) rated outstanding,
Between Certain         the 636 (58 percent) rated highly successful, and the 287 (26 percent)
Employee                rated satisfactory.:’ The ratings analyses covered 74 percent of the
Characteristics and     13,197 GS employees and 90 percent of 1216 GM employees. These are
                        the employees for whom we had complete data on age, ethnicity, sex,
Performance Ratings     grade, step (or years in grade), and work location. The factors having
                        statistically significant relationships with performance ratings, after we
                        controlled for the influences of other factors, are as follows.


GS Employees            Age. Employees under age 40 had 1.3 times greater odds of being rated
                        highly than those 40 and over.

                        Ethnicity, sex. White females had 1.3 times greater odds of being rated
                        highly than nonwhite females. White males, on the other hand, had the
                        same odds of being rated highly as nonwhite males.

                        Grade, ethnicity, sex. In grades 1 through 10 and 13 through 14, white
                        females had 2.1 times greater odds of being rated highly than white
                        males. Similarly, nonwhite females in those grades had 1.7 times greater
                        odds of being rated highly than nonwhite males. Differences in grades
                        11 through 12 were much smaller.




                        “In some analyses we combined the outstanding and highly successful rating groups into a “high”
                        rating category. This was done because preliminary analysis revealed that it would simplify the anal-
                        ysis without significantly affecting its results.



                        Page 4                                    GAO/GGD-90-40     Appraisals   and Promotions   at Customs
                        B-236938




GM Employees            Age. Employees under age 50 had 1.8 times greater odds of being rated
                        outstanding rather than highly successful and 1.8 times greater odds of
                        being rated highly successful rather than satisfactory compared to those
                        over age 50.

                        Ethnicity, sex. Among females, whites had 2.1 times greater odds than
                        nonwhites of being rated outstanding rather than highly successful.
                        Among males, nonwhites had 1.6 times greater odds than whites of
                        being rated satisfactory rather than highly successful.

                        Grade. Compared to employees in grade 13, those in grades 14 through
                        15 had 1.6 times greater odds of being rated outstanding rather than
                        highly successful and 1.6 times greater odds of being rated highly suc-
                        cessful rather than satisfactory.

                        Years in grade. Compared to employees who had been in grade for 1
                        year, employees in grade for 2 to 5 years had 1.9 times greater odds of
                        receiving outstanding rather than highly successful ratings. Compared
                        to employees who had been in grade for 6 or more years, employees in
                        grade for 2 to 5 years had 1.6 times greater odds of receiving highly
                        successful rather than satisfactory ratings.


                        Our analyses of GS promotion data sought to determine which factors
Statistically           differentiated between those 1415 employees (17 percent) who were
Significant             permanently competitively promoted between July 1987 and August
Relationships Existed   1988 and those 6,892 (83 percent) who were not promotedS The GM
                        analyses sought to determine which factors differentiated between the
Between Certain         138 employees (13 percent) who were permanently competitively pro-
Employee                moted and the 954 (87 percent) who were not promoted. The analyses
Characteristics and     covered 63 percent of 13,197 GS employees and 90 percent of 1,216 GM
                        employees-the employees for whom we had complete data on perform-
Competitive             ance rating, age, ethnicity, sex, grade, step (or years in grade), and work
Promotions              location. The factors having statistically significant relationships with
                        competitive promotions after we controlled for the influences of other
                        factors are as follows.


                        ‘The promotion analyses excluded employees who were promoted either temporarily or noncompeti-
                        tively. This was done because one of our main interests was to examine relationships between per-
                        formance ratings and promotion decisions. Since many employees who are promoted
                        noncompetitively advance automatically if they receive a performance rating level of fully successful
                        or higher, we felt that ratings were less of a factor for them than for those who compete for promo-
                        tions that are not automatic.



                        Page 6                                   GAO/GGD-90-40     Appraisals   and Promotions   at Customs
               B-236938




GS Employees   Step. Employees in lower steps had 1.4 times greater odds of being pro-
               moted than those in higher steps.

               Ethnicity, location. In headquarters locations, whites had 1.4 times
               greater odds of being promoted than nonwhites. In field locations,
               whites had .7 times lower odds of being promoted than nonwhites.

               Sex, rating, location. Males rated outstanding had 2.3 times greater odds
               of being promoted if they were in field rather than headquarters loca-
               tions. In contrast, females rated either highly successful or satisfactory
               had 2 times greater odds of being promoted if they were in headquarters
               rather than field locations. Being in field rather than headquarters loca-
               tions did not affect the promotion odds of females rated outstanding and
               males rated highly successful or satisfactory.

               Grade, rating. Among employees rated outstanding, those in grades 1
               through 6 had 1.2 times greater odds of being promoted than those in
               grades 7 through 14. However, among employees rated highly successful
               and satisfactory, employees in grades 1 through 6 had 2.0 and 3.0 times
               greater odds of being promoted, respectively, than employees in grades
               7 through 14.

               Age, sex, location. Among headquarters employees, females under age
               40 had 2.2 times greater odds of being promoted than males in that age
               range, while females 40 and over had 1.6 times greater odds of being
               promoted than males in that age range. In contrast, among those under
               40 in field locations, there was almost no difference in promotions
               between males and females. Finally, among those 40 and over in field
               locations, males had 1.4 times greater odds of being promoted than
               females.

               Rating, grade, location. Employees who were rated highly successful
               had 2.1 times greater odds of being promoted than those who were rated
               satisfactory if they were in grades 1 through 6 and 3.1 times greater
               odds of being promoted if they were in grades 7 through 14. However,
               when we compared those who received outstanding rather than highly
               successful ratings, the influence of the higher rating was not as evident.
               Receiving outstanding rather than highly successful ratings in head-
               quarters decreased the odds of promotion by half for those in grades 1
               through 6 and by a factor of 0.8 for those in grades 7 through 14. For
               lower grades in field locations, being rated outstanding rather than
               highly successful had no influence on promotions. Only for higher
               graded employees in the field did outstanding ratings have a positive


               Page 6                        GAO/GGD-9040   Appraisals   and Promotions   at Customs
               B-236938




               influence. For them, the odds of promotion were 1.7 times greater if
               they received outstanding rather than highly successful ratings.


GM Employees   Age. Employees under age 50 had nearly 3 times greater odds of being
               promoted than those 50 and over.

               Ethnicity. Whites had 2 times greater odds of being promoted than
               nonwhites.

               Sex. Females had nearly 2 times greater odds of being promoted than
               males.

               Grade. Employees in grade 13 had 2.5 times greater odds of being pro-
               moted than those in grades 14 through 15.

               Years in grade. Those in grade for two or more years had 3.3 times
               greater odds of being promoted than those in grade for less than 2 years.

               Rating. Highly rated employees (i.e., those with either outstanding or
               highly successful ratings) had 3.9 times greater odds of being promoted
               than those with satisfactory ratings.

               Location, Headquarters employees had 1.5 times greater odds of being
               promoted than field employees.


               Our analysis indicates that it is feasible to use existing data to examine
Observations   complex relationships between employee characteristics and such per-
               sonnel management actions as performance ratings and promotions. Our
               results indicate that statistically significant relationships existed among
               the test factors. For example, age, sex, and ethnicity were all found to be
               related to ratings and permanent competitive promotions.

               While loglinear analysis techniques enabled us to determine the extent
               to which certain factors were related to ratings and promotions, inter-
               pretation of what these relationships mean is very difficult. Statistical
               findings by themselves are not sufficient to draw conclusions about
               agency practices. Rather, such results either confirm or do not confirm
               that certain relationships exist between employee characteristics and
               personnel actions, and they can be used to point to directions for further
               inquiry. In this respect, the U.S. Customs Service could use our results



               Page 7                         GAO/GGD+NMO   Appraisals   and Promotions   at Customs
               B-236938




               as a basis for making more in-depth analyses of these matters to deter-
               mine why these relationships exist.


               We obtained informal comments on the information contained in this
Agency Views   report from responsible officials of the U.S. Customs Service. They
               lauded our methodology as being an excellent exploratory tool for iden-
               tifying areas for further analysis. However, they expressed concern
               about three issues that they feared could cast their management prac-
               tices in a negative light. We have considered Customs’ views, but feel
               that our report appropriately alerts the reader to the limitations of our
               study and adequately addresses the concerns that Customs raised.

               Customs expressed concern about the validity of the ethnicity data used
               and the possibility that our findings may be misleading because Customs
               recently discovered coding errors in its data. These errors involved the
               erroneous coding of an unknown number of nonwhites as whites. We
               agree that, as in any analysis, the most accurate available data should
               be used. As discussed in the report, we verified data elements where
               feasible and found them to be within a S-percent error tolerance. To
               check ethnicity data, which could not be directly verified against source
               documents, we compared the data we obtained from Customs with
               governmentwide ethnicity data for the same occupations. This compari-
               son showed that the percentage of whites and nonwhites in Customs’ GS
               and GM pay plans were very similar to those of the rest of the govern-
               ment. We therefore accepted Customs data as a reasonable foundation
               for our analyses. Using the data, we found that loglinear analysis was a
               feasible approach for identifying patterns in personnel management
               that may warrant further evaluation.

               With respect to Customs’ concern over the validity of our findings
               because of the misclassification of nonwhit,es as whites, we believe that
               in all likelihood this misclassification would lead to an underestimation
               of differences between the two groups. This would occur because mixing
               the characteristics of one group with those of another would tend to
               make the two groups more similar, not more distinct. Thus, rather than
               invalidating our findings, Customs’ misclassification of nonwhites as
               whites would tend instead to produce a lower limit estimate of actual
               ethnicity differences at Customs.

               A second Customs concern pertained to our exclusion of noncompetitive
               promotions from the analysis. Specifically, Customs officials felt that
               our findings may have differed had we considered employees who were


               Page 8                        GAO/GGD-9040   Appraisals   and Promotions   at Customs
B-236938




promoted noncompetitively through the career ladder. It is correct that
our analyses excluded both noncompetitive promotions and temporary
competitive promotions. However, one of our main interests was to
examine relationships between performance ratings and promotion deci-
sions. Since employees in the career ladder are promoted if they receive
a performance rating of fully successful or higher, advancement for
them is automatic given a minimum level of performance. In contrast,
there is nothing automatic about permanent, competitive promotions,
and it was our view that rating levels would factor more prominently in
decisions concerning these types of promotions. We revised the report to
more clearly explain why we excluded noncompetitive promotions from
our promotion analyses.

Customs’ third concern was with the fact that because our study did not
examine the proportional representation of whites and nonwhites in
promotions, it should not be used to draw conclusions about Customs’
equal employment opportunity (EEO) practices. We agree. Our promotion
analyses focused on the odds of whites and nonwhites being promoted
after we controlled for the influences of other factors. We made no
attempt to study Customs’ compliance with EEO guidelines, and it would
be improper to conclude that discriminatory practices were occurring at
Customs without further analysis. As discussed in the report, our statis-
tical findings point to areas requiring more in-depth inquiry and by
themselves are insufficient to draw conclusions about agency personnel
management practices.


As requested, we plan no further distribution of this report until 30
days after its issuance unless you publicly announce its contents earlier.
At that time, we will send copies to the Commissioner of the U.S. Cus-
toms Service and other interested parties.




Page 9                         GAO/GGD-90-40   Appraisals   and Promotions   at Customs
B-236938   .




The major contributors to this report are listed in appendix II. If you
have any questions concerning the content of the report, please call me
on 275-5074.

Sincerely yours,




Bernard L. Ungar
Director, Federal Human Resource
  Management Issues




Page 10                      GAO/GGD-90-40   Appraisals   and Promotions   at Customs
Page 11   GAO/GGD-So40   Appraisals   and Promotions   at Custon~
Contents


Letter                                                                                                          1

Appendix I                                                                                                 14
Technical Appendix:     Data Analysis Approach                                                             14
                                                                                                           15
                        Variables Analyzed
Loglinear Methodology   Logit Models Tested and Results Obtained                                           16
and Analysis Results
Appendix II                                                                                                40
Major Contributors to
This Report
                                                                                                          --
Tables                  Table I. 1: Variables Analyzed and the Categories Into                             15
                            Which They Were Grouped
                        Table 1.2: GS Performance Ratings: Logit Models Tested to                          17
                            Examine Relationships With Sex, Ethnicity, Age,
                            Grade, and Step
                        Table 1.3: GS Performance Ratings: Observed and                                    18
                            Expected Frequencies, Odds, and Odds Ratios From
                            the Preferred Model
                        Table 1.4: GM Performance Ratings: Logit Models Tested                             23
                            to Examine Relationships With Sex and Ethnicity
                        Table 1.5: GM Performance Ratings: Logit Models Tested                             24
                            to Examine Relationships With Age, Grade, and
                            Years in Grade
                        Table 1.6: GM Performance Ratings and Their                                        26
                            Relationship With Ethnicity and Sex: Observed and
                            Expected Frequencies, Odds, and Odds Ratios From
                            the Preferred Model
                        Table 1.7: GM Performance Ratings and Their                                        28
                            Relationship With Age, Grade, and Years in Grade:
                            Observed and Expected Frequencies, Odds, and Odds
                            Ratios From the Preferred Model
                        Table 1.8: GS Promotions: Logit Models Tested to Examine                           30
                            Relationships With Age, Grade, Step, Sex, Ethnicity,
                            Location, and Performance Rating
                        Table 1.9: GS Promotions and Their Relationship With                               31
                            Ethnicity and Location: Observed Frequencies, Odds,
                            and Odds Ratios



                        Page 12                      GAO/GGD-90-40   Appraisals   and Promotions   at Customs
Table I. 10: GS Promotions and Their Relationship With                             32
     Rating, Sex, Age, Grade, and Location: Observed and
     Expected Frequencies, Odds, and Odds Ratios From
    the Preferred Model
Table I. 11: GM Promotions: Logit Models Tested to                                 36
    Examine Relationships With Performance Ratings,
    Age, Grade, Years in Grade, and Location
Table I. 12: GM Promotions and Their Relationship With                             37
    Sex and Ethnicity: Observed and Expected
    Frequencies, Odds, and Odds Ratios From the
    Preferred Model
Table 1.13: GM Promotions and Their Relationship With                              38
    Rating, Age, Grade, Years in Grade, and Location:
    Observed and Expected Frequencies, Odds, and Odds
    Ratios From the Preferred Model




Abbreviations

EEO         equal employment opportunity
GM          General Management
GS          General Schedule
I'MRS       Performance Management Recognition System


Page 13                       GAO/GGD-9040   Appraisals   and Promotions   at Customs
Appendix I

TechnicalAppendix: Loglinear Methodology
md Analysis Results

                This appendix provides additional technical detail on our analytical
                approach to the Customs data. It contains a general description of log-
                linear methodology, describes the variables analyzed and how they were
                categorized, and presents the loglinear models tested and results
                obtained in each analysis.


                We used logit analysis, a form of loglinear modeling, to test associations
Data Analysis   between various independent variables and two outcome variables: per-
Approach        formance ratings and competitive promotions. For each outcome varia-
                ble, we fit a series of hierarchical logit models that allowed for
                associations among the factors in the model and varied in terms of the
                direct and indirect effects they had on the outcome. Using maximum
                likelihood statistical tests to compare the fit of various models with one
                another, we were able to make inferences about which factors signifi-
                cantly predicted the outcomes and how the factors interacted with one
                another.

                For each outcome, we selected a preferred statistical model. The pre-
                ferred model was the simplest model that fit the data and could not be
                significantly improved by more complex models. The preferred model
                included those factors that had statistically significant direct and indi-
                rect relationships with outcome after we controlled for the influences of
                other factors. Hence, the estimates we obtained were net effects deter-
                mined after the association of each variable with all other variables had
                been taken into account.

                On the basis of the preferred model, we estimated both the direction and
                size of the relationships using odds and odds ratios. The odds indicated
                the tendency for an outcome to occur given a particular combination of
                factors (e.g., the odds of being promoted given that an individual was
                under 50, in grades 14 through 15, and had a satisfactory rating). The
                odds ratio indicated the size of the effect. For example, if the odds of
                being promoted were . 1 among men and .l among women, the odds ratio
                between them would be 1.O,indicating the absence of a relationship
                between sex and promotion. The more the odds ratio diverges from 1.O,
                the greater the association.

                Our analyses were based on available Customs data on employee charac-
                teristics rather than on experimental data collected expressly to test
                theories of the determinants of job outcomes.




                Page 14                        GAO/GGD9040   Appraisals   and Promotions   at Customs
                                                     Appendix I
                                                     Technical Appendix: Logllnear      Methodology
                                                     and Analysis Results




                                                     Table I. 1 displays the variables included in each analysis and the catego-
Variables Analyzed                                   ries into which the variables were grouped. The purpose of the grouping
                                                     was to reduce the number of categories into which the outcome variable
                                                     was divided, thereby increasing the number of individuals in the various
                                                     categories.


Table 1.1:Variables Analyzed and the Categories Into Which They Were Grouped
                                     Categories used in    Categories used in     Categories used in                              Categories used in
                                    ;;;;ysis of GS ratings analysis of GM ratings analysis of OS                                  analysis of GM
Variable
Ratins .._........_...
                 -.-.--.-.-..-l-High(combi”es              data                   promotions data                                 promotions data
                                                                 Outstanding                     Outstanding                      High (combines
                                    outstanding and highly       Highly Successful               Highly Successful                outstanding and highly
                                    successful)                  Satisfactory                    Satisfactory                     successful)
                                    Satisfactory                                                                                  Satisfactory
Promotion                           Not applicable               Not applicable                  Competitively promoted           Competitively promoted
                                                       ---                                       Not promoted                     Not promoted
Sex                                 Male                         Male                            Male                             Male
                                    Female                       Female
                                                                -.---_-                          Female                           Female
Ethnicity                           White                        White                           White                            White
     ^.._....                       Nonwhite
              l-...--..“l . -_.~_-..-______-                     Nonwhite                        Nonwhite                         Nonwhite
Age                                 Under 40                     Under 50                        Under 40                         Under 50
                                   40 and over                   50 and over                     40 and over                      50 and over
Grade                               l-10                                                         1-6                              13
                                    11-12                        1L5                             7-14                             14-15
                                    13-14
Step/ years in gradea               1-4                          Under 2                                                          Under 2
                                   5-10                          2-5                             l-5                              2 and over
                                                                 Over 5
                                                   ___- ...--._______.---                        6-10
Location                            Headquarters                 Headquarters                    Headquarters                     Headquarters
                                    Field                        Field                           Field                            Field
                                                     aTime in grade is indicated by “step”   in the GS analyses and “years in grade” in the GM analyses


                                                     We used loglinear analysis to arrive at statistically sound ways of group-
                                                     ing the four ordered variables - step (or years in grade), age, grade,
                                                     and performance rating. Because associations between independent and
                                                     outcome variables differed in the various analyses, different categoriza-
                                                     tions were appropriate. For example, when analyzing the GS data, we
                                                     compared employees under age 40 with employees 40 and over. When
                                                     analyzing the GM data, we compared those under 50 with those 50 and
                                                     over. It made substantive sense to do so, since supervisory and manage-
                                                     ment personnel under the GM system were likely to be older than the
                                                     nonsupervisory personnel under the GS system. By separately analyzing
                                                     each ordered variable, we were able to group the categories in such a
                                                     way as to simplify our analyses of outcomes while maintaining the sta-
                                                     tistical relationships in the data.



                                                     Page 16                                     GAO/GGD-9040        Appraisals   and Promotions   at Customs
                         Appendix I
                         Technical Appendix   Loglinear   Methodology
                         and Analysis Results




                         We did four sets of logit analyses on the Customs data. The analyses
Logit Models Tested      sought to determine which factors had statistically significant relation-
and Results Obtained     ships with performance ratings and promotion outcomes of Customs’ GS
                         and GM employees. For each analysis, the models we tested to arrive at a
                         preferred model and the odds and odds ratios resulting from the pre-
                         ferred model are as follows.


GS Performance Ratings   These analyses were based on 9,763 GS employees at US. Customs, of
                         whom 58 percent received outstanding ratings and 42 percent received
                         highly successful or satisfactory ratings. We combined the highly suc-
                         cessful and satisfactory rating groups because preliminary analyses
                         showed no significant effects of the factors of interest to be rated in one
                         versus the other category. Collapsing highly successful and satisfactory
                         ratings simplified the analysis while retaining nearly the same amount
                         of explanatory power as did the original categories.

                         Table I.2 shows the loglinear models tested to arrive at the preferred
                         model of performance ratings. The preferred model, Model 29, indicated
                         that age had a direct relationship with ratings, that step and grade had
                         an indirect relationship with ratings, and that grade and ethnicity inter-
                         acted with sex in affecting ratings. Table I.3 shows the odds and odds
                         ratios resulting from the preferred model.




                         Page 16                                  GAO/GGD9040   Appraisals   and Promotions   at Customs
                                            Appendix I
                                            Techlcal Appendix        Loglinear Methodology
                                            and Analysis    RemIts




Table 1.2: GS Performance Ratings: Logit
Models Tested to Examine Relationships                                        Models tested8
With Sex, Ethniclty, Age, Grade, and Step                                                                                      Degr;es      Likelihood
                                            MOdd                                                                                             ratio chi-
                                            Number                                                                             freedom        square
                                            (1)     WAGS1 -___WI                                                                      47            457.20
                                            (2)   [XEAGS]       [XR]     WI           [AR1      Ml         PV                         41              88.30
                                            (3)   [XEAGS] -.-~--[XR]     PI           [AR1 WI                                         42              92.37
                                            (4)   [XEAGS]       [XR]     [ERI         [AR1 WI                                         43            357.66
                                            (5)   [XEAGS)R]              WI           WI   WI                                         42            112.30
                                            (6)   [XEAGSl       [XRl     lAR1         FGRl      K3Rl                                  42              96.49
                                            ;7j   ~XEAGS~ ~---~ERI       ~ARJ         ~&RI      ~SRI                                  42            230.16
                                            (8)   [XEAGS]       [XER]    [AR]         WI        [SRI                                  40              83.33
                                            (9)
                                            ~-_   [XEAGS]
                                                      --      - [XAR]    [ER]         WI        WI                                    40              84.22
                                            (IO)  [XEAGSJ       [XGRl    [ERl         lAR1      WV                                    39              73.59
                                            (1 I) ~xEAGS~ [xs~;          i~Rj         iA~j      [GR]                                  40              82.40
                                            (12)
                                            ~_I   [XEAGS]       [EAR]    [XR]         WI   WI                                         40        .--   87.06
                                            (13)  [XEAGS]       [EGR] [XR]            [AR1 [SRI                                       39              85.76
                                            (14)  [XEAGS]       [ESR]    [XR]         [AR1      Ml                                    40              88.00
                                            (15)
                                            -___  [XEAGS]       [AGR] [XR]            FRI       [SRI                                  39              81.60
                                                                                                                                                        -
                                            (16)
                                            -     [XEAGS]       [ASR]    [XR]         WI        [W                                    40              86.35
                                            (17)  [XEAGS]       [GSR] [XR]            Hl        [AR1                                  39           -- 76.43
                                            (18)  [XEAGS] --    [iER]Ai]
                                                                    -.__              EXGiI]    EXSh]      [AGR] [GSR]                32              43.77
                                            (19)  [XEAGS] -___  [XER]    [XAR]        [XGR]     [XSR]      [AGR]                      34 --__         55.58
                                            (20)  [XEAGS]       [XER]    [XAR]        [XGR]     [XSR]      [GSR]                      34              48.07
                                            (21)
                                            __-- [XEAGS]  _____ [XER][XAR]            [XGR]     [AGR]      [GSR]                      33              43.81
                                            (22) [XEAGS]        [XER]    [XAR]        [XSR]     [AGR]      [GSR]       -              34              63%
                                             (23) [XEAGS]____ [XER]      [XGR]        [XSR]     [AGR]      [GSR]                      33
                                                                                                                                   .________         46.41
                                            (24) [XEAGSJ        [XARl - -[XGRl        [XSR]     [AGRl      [GSRl                      33           --49.42
                                             (25) [XEAGS]
                                            /AAL
                                            --                  [XER]
                                                                  -.     [XGR~        [GSR]     [AR] -           ----                 36              51.51
                                             (26) rX=rXERl               rXGR1        rGSR1     lXAR1                                 35              48.11
                                            (27) [XEAGS]        [XER]    [XGR]        [GSR]     [XSR]      [AR]            ___-       35              50.74
                                            (28) [XEAGS]        [XER]    [XGR]        [GSR]     [AGR]                                 34             47.19
                                            (29)b [XEAGS] -[XER]         [XG,R]       [G,SR]    [AR]                                  38.------       51.93
                                            aThe notations used in this table are explained below:
                                            R = Performance Rating (high or satisfactory)
                                            X = Sex (male or female)
                                            E = Ethnicity (white or nonwhite)
                                            A = Age (under 40 or 40 and over)
                                            G = Grade (I-10 or 11-12 or 13-14)
                                            S = Step (1-4 or 5-10)
                                            bModel 29 was the preferred model at the .05 significance level. The model states that sex and ethnrcity
                                            interacted in affecting ratings, that sex and grade (in which the second category of grade was con-
                                            trasted against the first and third) interacted in affecting ratings, that step and grade (in whrch the first
                                            category of grade was contrasted against the second and third) interacted in affecting ratings, and that
                                            age had a direct relationship with ratings.




                                            Page 17                                      GAO/GGD-9040        Appraisals   and Promotions     at Customs
                                     Appendix I
                                     Technical Appendix: Loglinear   Methodology
                                     and Analysis Results




Table 1.3: QS Performance Ratings:
Observed and Expected Frequencies,
Odds, and Odds Ratios From the                                                                             Obsenred frequencies
Preferred Model                                                                                              Hi hC     Satisfactory
                                     Sex               Ethnicity              Age    Grade      Step        ratBng            rating
                                     Female            White                  <40      I-10       1-4           355                214
                                                                                                5-10            220                114
                                                                                      11-12       l-4           132                  43
                                                                                                5-10             38                  16
                                                                                      13-14       1-4            32                   5
                                                                                                5-10              12                  1
                                                                              40+-     l-10       l-4           110                  75
                                                                                                5-10            309                170
                                                                                      11-12       l-4            65                  29
                                                                                                5-10             86                  36
                                                                                      13-14       1-4             12        ___-      0
                                     -                                                          5-10              12                  3
                                                       Nonwhite               <40      l-10       l-4           220                166
                                                                                                5-10            132                  83
                                                                                      11-12       1-4            66                  36
                                                                                                5-10              12                  6
                                                                                      13-14       1-4              6                  2
                                                                                                5-10               1                  1
                                                                              40+      l-10       l-4            50                  50
                                                                                                5-10            187                121
                                                                                      11-12       l-4            31                  22
                                                                                                5-10             27                  16
                                                                                      13-14      1-4               3                  2
                                                                                                5-10               0                  2
                                     Male              White                  <40      l-10      l-4            268                358
                                                                                                5-10            296                222
                                                                                      11-12      l-4            368              --169
                                                                                                5-10           i69                   87
                                                                                      13-14      l-4             92                  23
                                                                                                5-10             27                   9
                                                                              40+      I-10      l-4             63                  95
                                                                                                5-10            445                592
                                                                                      11-12      l-4            196                106
                                                                                                5-10            642                403
                                                                                     13-14       l-4             55                - 31
                                                                                                5-10            121                  56
                   Y




                                     Page 18                                 GAO/GGD9040      Appraisals   and Promotions   at Customs
                                                           Appendix I
                                                           Technical Appendix   Loglinear   Methodology
                                                           and Analysie Results




                                                                                                                 Odds ratios’
             Expected frequenclesb                                             Sex            Ethnicity           Age               Grade                  Step
                HIa he     Satisfactory                             aHd!tg    Fe::l?z              White:           < 40:     11-12:      13-14:            5-10:
             ..rat ng             rating                   Sati*f                              nonwhite              40+       l-10       11-12               l-4
             361.01
        .- _-_...
                -..-----~                     207.19 --               1.75         2.1                    1.3         1.3
             229.12                           104.88                  2.19         2.1                    1.3         1.3                                      1.3
             129.91                            45.09                  2.88         1.3                    1.3         1.3          1.6
              38.74                            15.26                  2.54         1.3                    1.3         1.3          1.2                         0.9
              31.92                             5.08                  6.28         2.1                    1.3         1.3                       2.2
              11.01
           .._...._ .._.___-.----                   1.99            5.53           2.1                    1.3         1.3                       2.2            0.9
            107.48                                77.52             1.39           2.1                    1.3
            303.83                               175.17             1.73           2.1                    1.3                                                  1.3
              65.41                               28.59             2.29           1.3                    1.3                      1.6
              81.55                               40.45             2.02           1.3                    1.3                      1.2                         0.9
              10.00                       ----___I_-i.oo            5.00           2.1                    1.3                                   2.2
____--        12.22
                  I,._--__--.___                    2.78            4.40           2.1                    1.3                                   2.2            0.9
            219.61                               166.19             1.32           1.7                                1.3
            134.00                                61.00             1.65           1.7                                1.3                                      1.3
              69.95
               ..-..- .._.--                      32.05             2.18           1.0                                1.3          1.6
              11.84                                 6.16            1.92           1.0                                1.3          1.2                         0.9
                6.61                                1.39            4.76           1.7                                1.3                       2.2
                 1.61
         ..- -.---...---..          --.             0.39            4.13           1.7                                1.3                       2.2            0.9
              51.22
     _.~.--. ..-.._.              .               40.78             1.05           1.7
            174.87                               133.13             1.31           1.7                                                                         1.3
       __     33.61
                 .,._- _.._--..--..~.          -- 19.39             1.73           1.0                                             1.6
            . 25.98
              .._..-.-~_---~.-__--.               17.02             1.53           1 .o                                            1.2                         0.9
                3.95                                1.05            3.76           1.7                                                          2.2
                 1.54                               0.46            3.35           1.7                                                          2.2            0.9
            283.11                               342.89             0.83                                  1 .o        1.3
            263.19 .-~ --_-                      254.81             1.03                                  1 .o        1.3                                      1.3
          .'a~i .sl                     .-~-.i~g                    2.25                                  1.0         1.3          2.7
            170.10                                85.90             1.98                                  1.0         1.3          1.9                         0.9
              86.03                               28.97             2.97                                  1.0         1.3                       1.3
              26.05
      -. -_-.--.~~_--.----                          9.95            2.62                                  1.0         1.3                       1.3            0.9
              62.56                               95.44             0.66                                  1.0
            467.24                               569.76             0.82                                  1.0                                                  1.3
            193.52                               108.48             1.78                                  1.0                      2.7
            638.72                               406.28             1.57                                  1.0                      1.9                         0.9
              60.39
     __ .-I.-.._-.                   ~___
                                                  25.61        _____2.36                                  1.0                                   1.3
            119.49                                57.51
                                             I_____-.               2.08                                  1.0                                   1.3            0.9
                                   1                                                                                                                  (continued)




                                                           Page 19                                  GAO/GGD-9040      Appraisals   and Promotions     at Customs
Appendix I
Technical Appendix: Loglinear   Methodology
and Analysis Results




                                                                     Observed frequencies
                                                                       HighC     Satisfactory
Sex
-__               Ethnicity              Age   Grade      Step        rating            rating
                  Nonwhite               <40     I-10      l-4            126                129
                                                          5-10            111                138
                                                11-12      1-4             90                 45
                                                          5-10             14                   9
                                                13-14      l-4             14                   1
                                                          5-10              2                   1
                                     __- 40+     l-10      1-4             45                 55
                                                          5-10            166                221
                                                11-12      1-4             76                 36
                                                          5-10            136                 96
                                                13-14      1-4              7                   5
                                                          5-10              7                  -5




Page 20                                 GAO/GGD-90-40   Appraisals   and Promotions   at Customs
                                  Appendix I
                                  Technical Appendk    Loglinear      Methodology
                                  and Analysis Results




                                                                                          Odds ratios’
    Expected frequenciesb         %tgY                    Sex           Ethnicity          Age
                                                                                             <40:      11-12:
                                                                                                              Grade
                                                                                                                    13-14:
                                                                                                                                      Step
                                                                                                                                       5-10:
      HiPhC       Satisfactory           High?       Female:                 White:
    -.-
     rat ng              rating   Satisfactory         male              nonwhite             40+        l-10       11-12                1-4
     112.48              142.52              0.79                                                1.3
     123.70              125.30              0.99                                                1.3                                     1.3
      92.11               42.89              2.15                                                1.3          2.7
      i5.05   .-           7.95              1.89                                                1.3          1.9                        0.9
     11.09   -~-           3.91              2.84 -.___                                          1.3                       1.3
     ~2.14~                0.86              2.49                                                1.3                       1.3           0.9
     38.52~ -~            61.48              0.63
     170.05              216.95              0.78                                                                                        1.3
      70160               41.40              1.71                                                             2.7
     139.30               92.70              1.50                                                             1.9                        0.9
       8.31                3.69              2.25                                                                          I.3
      10.64                5.36              1.98                                                                          1.3           0.9

                                  aDue to rounding, the odds ratios may not match precisely those that would be obtained from direct
                                  calculation from the expected frequencies.

                                  bThese values were generated by the preferred model.
                                  CHigh rating combines outstanding    and highly successful ratings




GM Performance Ratings            These analyses were based on 1,098 GM employees on whom we had
                                  complete age, grade, sex, ethnicity, years in grade, and performance rat-
                                  ing data. Overall, 16 percent were rated outstanding, 58 percent were
                                  rated highly satisfactory, and 26 percent were rated satisfactory.

                                  To test relationships between performance ratings and GM employee
                                  characteristics, we carried out a series of analyses similar to those for GS
                                  employees. However, the way we grouped categories of variables in our
                                  GM ratings analyses was not necessarily the same as for GS employees. In
                                  Table I.1 for example, there are differences between the GS and GM anal-
                                  yses in how we grouped the age, grade, and ratings variables. Our
                                  rationale for such groupings was that they retained much of the original
                                  variation in performance ratings while simplifying our work.

                                  Because of the analytical difficulties that arise from trying to look at too
                                  many variables simultaneously, we carried out a two-phase analysis
                                  procedure. In the first phase, we determined that the sex and ethnicity
                                  variables interacted with one another in affecting performance ratings
                                  and that their relationships were unrelated to and unaffected by any
                                  other variable. (Table I.4 shows the loglinear models pertinent to this


                                  Page 2 1                                    GAO/GGD90-40       Appraisals   and Promotions     at Customs
Appendix 1
Technical Appendk    Loglinear   Methodology
and Analysis Results




determination and the preferred model identified.) Because sex and
ethnicity were related to ratings independently of age, grade, and years
in grade, the latter three variables were analyzed separately in the sec-
ond phase of this work. (Table I.5 shows the loglinear models that test
the relationships between age, grade, and years in grade with ratings
and the preferred model identified.) The odds and odds ratios from the
preferred models are shown in tables I.6 and 1.7.




Page 22                                  GAO/GGIHO-40   Appraisals   and Promotions   at Cust~me
                                           Appendix I
                                           Technical Appendix; Loglinear      Methodology
                                           and Aualysls Results




Table 1.4: C?MPerformance Ratings: Logit
Models Terted to Examine Relationahipr                                                                            De rre\~o       Likelihood ratio
With Sex and Ethniclty                     Model number                     Models tested*                          8                   chl-square
                                           (1)                              [ESAGY]         [RI                             78              161.62
                                           (2)                              [ESAGY]         WI                              76              153.00
                                           (3)                              [ESAGY]         [SRI       WI                   74              151 .oo
                                           (4)                              [ESAGY]         WA                              72              145.02
                                           (5)                              [ESAGY]         WI         [AR1                 74              124.53
                                           (6)                              [ESAGY]         [SARI                           72              120.84
                                           (7)                              [ESAGY]         WI    [GRI                      74              132.50
                                           (8)                              [ESAGY]         [SGRI                           72              132.44
                                           (9)                              [ESAGY]         WI    WI                        72              114.91
                                           (10)                             [ESAGY]         [SW                             68              114.15
                                           $)                               [ESAGY]         FW                              76              160.01
                                           (12)                             [ESAGY]         [W         [AR1                 74              128.06
                                           (13)                             [ESAGY]         W=‘l                            72              127.95
                                           (14)                             [ESAGY]         [ERI       WV                   74              140.55
                                           (15)                             [ESAGY]         WV                              72              139.84
                                           (16)                             [ESAGY]         WI         [W                   72              120.93
                                           (17)                             [ESAGY]         P’RI                            68              116.73

                                           (14                              PI        [RI                                     6               16.60
                                           (24                              WI        [W                                      4               14.99
                                           W
                                           -.-                              WI        [SRI                                    4                7.98
                                           (44                              WI        WI           [SRI                       2                5.98
                                           (54                              WI        FSRI                                    0                0.00
                                           Wb                               El        [V,R,l       [VA1                       4                4.15
                                           aThe notations used in this table are explained below:
                                           R = Performance Rating (outstanding, highly successful, or satisfactory)
                                           E = Ethnicity (white or nonwhite)
                                           S = Sex (male or female)
                                           A = Age (under 50 or 50 and over)
                                           G = Grade (13 or 14-15)
                                           Y = Years in Grade (1, 2-5, or 6 and over)

                                           bModel 6a was the preferred model at the .05 significance level. The model states that ethnicity and sex
                                           interacted in affecting performance rating. Further, they had the same relationship when comparing
                                           outstanding with highly successful as when comparing satisfactory with highly successful. Additional
                                           information on linear constraints can be found in Stephen E. Fienberg, The Analysis of Cross-Classified
                                           Categorical Data (Cambridge: MIT Press, 1988).




                                           Page 23                                     GAO/GGD90-40       Appraisals   and Promotions   at Customs
                                          Appendix I
                                          Technical Appendix: Loglinear      Methodology
                                          and AnalysiR Results




Table 1.5:GM Performance Ratings: Loglt
Models Tested to Examine Relationships                                                                          De ree8 of       Likelihood ratio
With Age, Grade, and Years in Grade       Model number                  Models tested*                            Breedom              chi-square
                                          (1)                           LAW        PI                                      22                101.35
                                          (2)                           [AGYI      [AR1      WV       WV                   14                 14.78
                                          (3)                           LAW        [AR1      [W                            18                 47.96
                                                                                                                                            ___-
                                          (4)                           LAG’4      [AR1      [YRI                          16                 39.50
                                          (5)
                                          ---                           LAG’4      WI        WI                            16                 38.36
                                          (6,                           [p;z;Yl [AGW WI                                    12                 14.68
                                          (7)
                                          --.-...-.--   -.-. -          [AGY]      [AYR]     [GR]                          i0                  8.94
                                          (8)                           [AGY]      [GYR]     [AR]                          10                 10.15
                                          (9)"                          LAG’4 [AR,1 E&l               [YR,l                18                 20.31
                                          aThe notations used in this table are explained below:
                                          R = Performance Rating (outstanding, highly successful, or satisfactory)
                                          A = Age (under 50 or 50 and over)
                                          G = Grade(l3or    14-15)
                                          Y = Years in Grade (1, 2-5, or 6 and over)

                                          “Model 9 was the preferred model at the .05 significance level. The model states that age, grade, and
                                          years in grade each had a direct relationship with performancebafing. In this model, rating was linearly
                                          constrained so that the relationship when we compared outstanding and highly successful was the
                                          same as when we compared highly successful and satisfactory.




                                          Page 24                                     GAO/GGD-90-40     Appraisals   and Promotions     at Customs
Page 25   GAO/GGD-90-40   Appraisals   and Promotions   at Customs
                                        Appendix       I
                                        Technical      Appenti      Loglinear   Methodology
                                        and Analysis      Results




Table 1.6:GM Performance Ratings and
Their Relationship With Ethnicity and
Sex: Observed and Expected
Freauencies, Odds, and Odds Ratios                                                            Observed frequencies
From the Pr&erred.Model                                                                                          Highly
                                        Ethnicity                     Sex                     Outstanding    successful              Satisfactory
                                        White                         Female                             26               51                    20

                                                        __-           Male                              132              527                  233
                                        &white                        Female                              3               15                    2
                                                                      male                               14               43                   32




                                        Page 26                                         GAO/GGD-9040   Appraisals   and Promotions    at Customs
                                           Appendix I
                                           Technical Appendix   Log&ear    Methodology
                                           and Analysis Results




                                                                                                      Odds on
                                                                        Odds ratio@                   ratings                Odds ratios’
           Expected frequenciesb                 Outstandin :      Ethnicity       Sex               Satisfactory:        Ethnicity     Sex
                      Highly                            high PY        White:     Female:                   highly        Nonwhite:     Male:
Outstanding     successful
       26,00 .-~---.-~~~~~---    Satisfactory     successflli       nonwhite         male              successful              white female
                                         21.41             0.52             2.1             2.1                    0.43

      134.44            529.10          228.47             0.25             1.0                                    0.43                       1.0
        3.01             11.86            5.12             0.25                             1 .o                   0.43              1.0
       11.55             45.45           32.00             0.25                                                    0.70              1.6      1.6

                                          %e to rouncilng, the odds ratios may not precisely match those that would be obtained from direct
                                          calculation from the expected frequencies.

                                          bThese values were generated by the preferred model




                    Y




                                          Page 27                                   GAO/GGD9040       Appraisals    and Promotions    at Customs
                                          Appendix I
                                          Technical Appendixz Loglinear    Methodology
                                          and Analysis Results




Table 1.7: GM Performance Ratings and
Their Relationship With Age, Grade, and
Years in Grade: Observed and Expected
Frequencies, Odds, and Odds Ratios
From the Preferred Model                                                                             Observed frequencies
                                                                          Years in
                                          Age            Grade            grade            Outstanding       succ%%!              Satisfactory
                                           <50           13                <2                          8               43                    35
                                                                          2-5                         41              137                    43
                                                                          6+                          11               70                    35
                                                         14-15             <2                         20               54                    29
                                          ---.-                           2-5                         51              119                    19
                                                                          6+                          21               56                    22
                                          50+            13                <2                          0                4                     3
                                                                          2-5                          3               24                    18
                                                                          6+                           6               47                    40
                                                         14-15            <2                           2                6                     2
                                          ~.
                                                                          2-5                          1               24                     4
                                                                          6+                          11               52                    37




                                          Page 28                                    GAO/GGD90-40   Appraisals   and Promotions     at Customs
                                                  Appendix I
                                                  Technical Appendix: Loglinear     Methodology
                                                  and Analysis Results




                                                                   Odds ratio@                       Odds ratiosa
                                               Odds on                                   Odds on
                                               ratings                                    rating5                                               Years   in
       Expected frequenciesb                 Outstandin . Age Grade Years in grade
                 Highly                              bight  < 50: 14-15:      2-5: 2-5: succ:~k#      -+%       %%                              2-:“*2-6:
Outstanding successful Satisfactory           successful
                                                 ----______  so+       13        1  6+  satisfactory     50+       13                          ~~--i ----~6+
       7.48    _    47.48              31.04                   .I6    1.8                                     1.53          1.8
                                                                                                                              ______-___-~                     -
      40.35        134.50 - ~~ .~~   .46.16            ..-.._. .30    1.8              1.9        1.6        2.91           1.8 -__-          1.9            1.6
      12.99         67 ,21            35;8i------.             T19    1.8                                    1.88           1.8
      15.94         62.13             24.93                    .26    1.8 -   1.6                            2.49           1.8       1.6
                                                                                                                                     ___
      54.35        111.22             23.43                    .49    1.8     1.6      1.9        1.6        4.75           1.8       1.6    __---
                                                                                                                                              1.9            1.6
      18.98         60.30              19.72       -_- --..~.31    -~-1.8     1.6                            3.06           1.8       1.6_____~
       0.27          3.10                3.63                  .09                                           0.85                     ---.__   ..- --.-~.~
       4.23         25.25              15.52                   .I7
                                                      ~~ ___-..---                     1.9 -      1.6        1.63                                1.9       1.6
       4.87   .-    45.10             43.03..~_ -.        ~..  .11                                           1.05            -___
       0.77          5.37               3.86                   .14 _---..     1.6                            1.39         ___..-__.     1.6
       4.80         17.57               -6.63                  .27            1.6      1.9        1.6        2.65                       1.6      1.9         1.6
       9.98         56.77             33.25                    .18            1.6                            1.71                       1.6

                                                 Que to rounding, the odds ratios may not precisely match those that would be obtained from direct
                                                 calculation from the expected frequencies.

                                                 “These values were generated by the preferred model.




GS Promotions                                    These analyses were based on 8,307 GS employees for whom we had
                                                 complete age, grade, step, sex, ethnicity, ratings, location, and promo-
                                                 tion data. Overall, 17 percent of the GS employees were permanently
                                                 competitively promoted between July 1987 and August 1988, and 83
                                                 percent were not promoted.

                                                 To avoid too many cross-classifications of the data and to facilitate pres-
                                                 entation of findings, we again analyzed the data in two stages. In the
                                                 first stage, we determined that step was directly related to promotion
                                                 and that ethnicity interacted only with location in affecting promotions.
                                                 Therefore, age, grade, sex, location, and rating could be analyzed sepa-
                                                 rately in the second stage of the analysis. (Table I.8 shows the loglinear
                                                 models pertinent to this determination.) The odds and odds ratios from
                                                 the preferred model are shown in tables I.9 and 1.10.



                      Y




                                                 Page 29                                     GAO/GGD-90-40   Appraisals     and Promotions      at Customs
                                                  Appendix I
                                                  Technical Appendix; Loglinear     Methodology
                                                  and Analysis Results




Table 1.8:GS Promotions: Logit Models Tested to Examine Relationships With Age, Grade, Step, Sex, Ethnicity, Location, and
Performance Rating
                                                                                                                     Likelihood
                                                                                                     De rees of        ratio chi-
Model number             Models tested”         ---                                                     8reedom          square
(1)                      [RSEAGTL]     --- [P]                                                                360        1081.35
(2)                      [RSEAGTL]
                           -..-- --.--     [RP]   [SP] [EP] [API  [GP]  [TP]    WI                            350          451.43
(3)                         [RSEAGTL]           [RP]    [SP]    [EP]    [API    [GP]    [TP]                                       351          453.31
(4)                         [RSEAGTL]           [RP]    [SP]    [EP]    [API    [GP]    [LPI                                       352          480.83
C5)            ~~           [RSEAGTL]           [RP]    [SP]    [EP]    [API    [TP]    [LPI                                       352           545.22
(6)                         [RSEAGTL]
                             ~-.-..~-           [RP]
                                                ___-    [SP]    [EP]    [GP]    [TP]    [LPI                                       351           517.25
                                                                                                                                                ---
(7)                         [RSEAGTL]           [RP]    [SP] ~- [API    [GP]    [TP]    [LPI                                       351           457.50
(8)                         [RSEAGTL]           [RP]    [EP]    [API    [GP]    [TP]    [LPI                                       351          451.43
(9)            .-.          [RSEAGTL]           [SP]
                                       __---_____-.-    [EP]    [API    [GP]    [TP]    [LPI                                       352           733.03
(10)                        [RSEAGTL]
                              -... .~--         [EP]    [API
                                                 _____.___      [GP]    [TP]    [LPI    [RSP]                                      348           440.61
(11)                        [RSEAGTL]           [SP]    [API    [GP]    [TP]    [LPI    [REP]                                      348     ~- 450.77
(12)                        [RSEAGTL]           [SP]    [EP]    [GP]    [TP]    [LPI    [RAP]                                      348           444.32
(13)                        [RSEAGTL]           [SP]    [EP]    [API    [TP]    [LPI    [RGP]                                      346          432.89
(14)                        [RSEAGTL]           [SP]    [EP]    [API    [GP]    [LPI    [RTP]                                      346          446.62
(1%                     _.._[RSEAGTL]
                             ..-..---.---       [SP]    [EP]    [API    [GP]    [TP]    [RLP]                                      348           440.62
(16)                        [RSEAGTL]           [RP]    [API    [GP]    [TP]    [LPI    [SEP]                                      349          449.47
(17)                        [RSEAGTL]           [RP]    [EP]    [GP]    [TP]    [LPI    [SAP]                                      349           444.95
(18)                        [RSEAGTL]           [RP]    [EP].-  [API    [TP]    [LPI    [SGP]                                      348           446.56
(19)                        [RSEAGTL]
                      -~__--.____.              [RP]    [EP] ____
                                                                [API    [GP]    [TP]    [STP]                                      348          450.11
(20)                        [RSEAGTL]           [RP]    [EP]    [API    [GP]    [TP]    [SLP]                                      349           431 .oo
(21)                        [RSEAGTL]           [RP]    [SP]    [GP]    [TP]    [LPI    [EAP]                                      349 --       449.19
(22)                        [RSEAGTL]           [RP]    [SP]    [API    [TP]    [LPI    [EGP]                                      348          445.93
(23)                        [RSEAGTL]           [RP]    [SP]    [API    [GP]    [LPI    [ETP]                                      348           450.82
(24)                        [RSEAGTL]-__-.-l__---.--~
                                                [RP]    [SP]- [API      [GP]    [TP]    [ELP]                                      349          437.09
(25)                        [RSEAGTL]           [RP]    [SP]    [EP]    [TP]    [LPI    [AGP]                                      348           447 82
(26)                        [RSEAGTL]           [RP]    [SP]    [EP]    [GP]    [LPI    [ATP]                                      348           448.39
(27)                        [RSEAGTL]           [RP]    [SP]
                                                           --   [EP]    [GP]    [TP]    [ALP]                                      349          450.12
(28)                        [RSEAGTL]           [RP]    [SP]    [EP]    [API    [LPI    [GTP]                                      346           442.70
(2%                         [RSEAGTL]           [RP]    [SP]:pl         [API    [TP]    [GLP]                                      348          437.96
                                                                                                                                            --.-___
(30)                        [RSEAGTL]           [RP]    [SP]    [EP]    [API    [GP]    [TLP]                                      348          451.09
(31)                        [RSEAGTL]
                         ~ _-----.          --. [TP]    [RSP]   [RAP]   [RLP]   [SAP]   [SLP]     [ELP]   [GLP]    [RGP]           335           379.29
(32)                        [RSEAGTL]           [TP]    [RAP] [RLP]     [SAP]   [SLP]   [ELP]     [GLP]   [RGP]                    337          382.25
(33)                        [RSEAGTL]           [TP]
                                                 __-.- [RLP] [SAP]      [SLP]   [ELP]   [GLP]     [RGP]                            339           387.37
(34)                        [RSEAGTL]           [TP]    [RLP] [SAP]     [SLP]   [ELP]   [RGP]                                      341           389.93
(35)                        [RSEAGTL]           [TP]    [SAP] [SLP]     [ELP]   [RGP]                                              343          398.01
                      3~~
(36)                        [RSEAGTL]           [API    [TP]    [SLP]   [ELP]   [RGP]   [RLP]                                      342           395.39-
                                                                                                                                          (continued)




                                                  Page 30                                   GAO/GGD-90-40     Appraisals   and Promotions   at Customs
-
                                               Appendix I
                                               Technical Appendix: Loglinear      Methodology
                                               and Analysis Results




                                                                                                                                           Likelihood
                                                                                                                          De rees of         ratio chi-
Model number              Models tested’                                                                                    Preedom            square
(37)                      [RSEAGTL]      [TP]   [ELP]      [RGP]   [RLP]    [SAP]                                                 342           402.64
(38)                      [RSEAGTL]
                     .~ ..__
                           -~ -.. -.--   [EP]   [TP]       [RGP]   [RLP]    [SAP] [SLP]                                           342           398.54
(39)                      [RSEAGTL]      [GP]   [TP]       [RLP]   [SAP]    [SLP] [ELP]                                           345           407.54
(40)”                     [RSAGLl        IR,LPl [SAP1      rSLP1   TR,GPl   fR,GPl                                                 34            41.85
                                               aThe notations used in this table are explained below:
                                               P = Promoted (competitively promoted or not promoted)
                                               R = Performance Rating (outstanding, highly successful, or satisfactory)
                                               S = Sex (Male or female)
                                               E = Ethnicity (white or nonwhite)
                                               A = Age (under 50 or 50 and over)
                                               G = Grade (l-6 or 7-14)
                                               T = Step (1, 2-5, 6-10)
                                               L = Location (headquarters or field)

                                               ‘Model 40 was the preferred model at the .05 significance level. It is a reduced form of Model 34,which
                                               is the preferred model when all eight variables were analyzed simultaneously. On the basis of Model 34,
                                               we determined that ethnicity and location could be analyzed separately. Model 40 states that sex inter-
                                               acted with both age and location in affecting promotions and that ratings interacted with both location
                                               and grade in affecting promotions.




Table 1.9: OS Promotions and Their Relationship With Ethniclty and Location: Observed Frequencies, Odds, and Odds Ratios
                                                                                       Odds on              Odds ratios’
                                                                                      promotion       Ethnicity      Location
                                                       Observed frequencies               Promoted:        White:
Ethnicity                 Location
                           ~~--- .~.-.                Promoted     Not promoted        not promoted     nonwhite            fI%
White                     Headquarters
                                               -___---             142                  519                       .27              1.4              1.5
                          Field
                                         ___                       806                 4448                       .I8               .7
Nonwhite                  Headquarters                              42                  209                       .20                                   .8
                          Field                                    425                 1716                       .25
                                               aDue to rounding, the odds ratios may not precisely match those that would be obtained from direct
                                               calculation from the expected frequencies.




                                               Page 31                                    GAO/GGD-9040       Appraisals   and Promotions   at Customs
                                       Appendix I
                                       Technical Appendk    Loglinear     Methodology
                                       and Analysis Results




Table 1.10: GS Promotions and Their
Relationship With Rating, Sex, Age,
Grade, and Location: Observed and
Expected Frequencies, Odds, and Odds                                                                             Observed frequencies
Ratios From the Preferred Model                                                                                         Not
                                       Rating
                                       --___               Sex                  Age Grade          Location       promoted    Promoted
                                       Outstanding         -Male               <40           l-6         HQ                 8              3
                                                                                                       Field               42             23
                                                                                            7-14         HQ                27              8
                                                               -                                       Field               83             35
                                                                                4ot          l-6         HQ                 3              0
                                                                                                       Field               46             20
                                                                                            7-14         HQ                25              5
                                                           ---                                         Field               65             17
                                                           Female              < 40          l-6         HQ                 1              0
                                                                                                       Field                7              6
                                                                                            7-14         HQ                14              6
                                                                                                       Field              177             93
                                                           -___                 4ot          l-6         HQ                 0             %
                                                                                                       Field                4              2
                                                                                            7-14         HQ                28              5
                                                     .___.--                                           Field              223             66
                                       Highly               Male
                                       successful                              < 40          l-6         HQ                36             14
                                                           -                                           Field              179            121
                                                                                            7-14         HQ     -          66             18
                                                                                                       Field              275             84
                                                                                4ot          l-6         HQ                16              3
                                                                                                       Field              158          -82
                                                                                            7-14         HQ                58             14
                                                                                                       Field              274             48
                                                           Female              <40           1-6         HQ                 2              0
                                                           -____                                       Field               56             29
                                                                                            7-14         HQ -..-___        59             33
                                                                   .--.                                Field              635            198
                                                           ___-.                4ot          l-6         HQ                 0              0
                                                                                        -              Field               64              9
                                                                                            7-14    7x-                    96             36
                                                                                                       Field             1183            165




                                       Page 32                                    GAO/GGD-90-40    Appraisals   and Promotions   at Customs
                                                  Appendix I
                                                  Technical Appends    Logllnear        Methodology
                                                  and Analysis Results




                                         Odds on                                                            Odds ratios’
                                        promotion                                                                                         Rating
Expected frequenciesb                    Promoted:          Location        e_e          Age                Sex              Outstandin :                      Highly
       Not                                      not            Field:         l-6                          Female:                  high By             successful:
 promoted      Promoted                   promoted                HO           7t         ‘50:               male             successful                satisfactory
      8.59           2.41                                .28                    1.2           1.6                                           .5
       39.46               25.54                         .65         2.3        1.2           1.6                                         1 .o
     -~28.26..~.       -     6.74                        .24                                  1.6                                          .8
       76.15               41.85                         .55
                                             .-_.-__________         2.3                      1.6                                         1.7
        2.55                  .45                        .I8                    1.2                                                        .5
       47.06 .._.           18.94
                          .-_  ~~ -. .- -..              .40         2.3        1.2                                                       1 .o
       26,13                 3.87           ----___-.    .15    -                -.-                                                       .8
      61.11 --.~~~ ~~--_   20.89                         .34         2.3                                                                  1.7
         .,62                 I~~. -~-~---_sl                                  1.2            2.2c             2.2                         .5
        8.02                 4.98                        .62
                                               _______-_________     1.0    ___-1.2           2.2              1.0                        1.0
       13.14                 6.86                        52
                                                   --.-.-..                                   2.2 -___         2.2                          .8
   .-176.86                93.12                         .53         1.0                      2.2              1.0                        1.7
        0.00.                0.00                            d                  1.2c                           l.6c                         .5"
        4.69                 1.31                        .28         1.0"
                                                                  _____         1.2
                                                                                 -.-.                            .7                       1.0
       26.71 -               6.29                        .24
                                                      - -.-----                                                1.6                          .8
     233.62 _._ ..--.---. .^~
                           55.38
                             _...--.--.-.--__-           .24         1.0                                         .7                       1.7

      31.81              18.19                   .57                             2.0          1.6                                                                 2.1
     142.84~~'          117.16                   .64 ____-----_____ 1.1         2.0           1.6                     -                                           2.1
      65.40.             18.60 ..-___            .28                                          1.6                                                                 3.1
     272.27              86.73                   .32                1.1                       1.6                                                                 3.1
      14.01               4.99                   .36                            2.0                                                                               2.1
     171.60              68.40                   .40                1.1
                                                                    ---.--~-    2.0                                                                               2.1
      61.18              10.82                   .I8          ___-__                                 .-.                                                          3.1
     268.75              53.25                   .20                1.1                                                                                           3.i
         .89              1.11                  1.25                            2.0
                                                                             ___--_____       2.2"             2.2                                                2.1
      52.67              32.33 ~~~~~.-.....      .61  _....~-..-.-__  .5         2.0
                                                                                -.-.          2.2              1.0                                                2.1
      56.67              35.33              .62                        ~________              2.2              2.2                                                3.1
    .638.22-~           194.78                   .30           --..- .5_____-.                2.2              1.0                                                3.1
       0.00               0.00                       d
                                                                                2.0" -...--                    1.6"                                               2.1"
      57.19              15.81             -~~728          ~~----_5--.-~~                                       .7                                                2.1
      98.38              27.62                   .28                                 ___--                     1.6                                                3.1
   1185.12.             162.88                   .14                  .5                                        .7                                                3.1
                                                                                                                                                         (continued)




                                                  Page 33                                            GAO/GGD-9040         Appraisals   and Promotions    at Customs
    Appendix I
    Technical Appendix: LogWear   Methodology
    and Analysis Results




                                                                         Observed frequencies
                                                                                Not
    Rating          Sex                 Age Grade         Location        promoted    Promoted
    Satisfactory    Male               < 40      1-6            HQ                 37             10
                                                              Field               115             45
                                                7-14            HQ                 34              3
                                                              Field               200             25
                                        40+      l-6            HQ                 18              3
                                                              Field               123           - 13
                                                7-14            HQ                 45              4
                                                              Field               233      --___ 15
                    Female             < 40      l-6            HQ                  2              3
                                                              Field                66             20
                                                7-14            HQ                 43             15
                                                              Field               546      ___.   50
                                        40+      l-6            HQ                  2              0
                                                              Field                90             13
                                                7-14            HQ                108              7
                                                              Field
                                                                 -720                             52




Y




    Page 34                               GAO/GGD-90-40   Appraisals    and Promotions   at Customs
           .


                                                   Appends I
                                                   Technical Appendix: Loglinear      Methodology
                                                   and Analysis Resulta




                                             Odds on                                                   Odds ratios”
                                            promotion                                                                               Rating
         Expected frequenclesb               Promoted:       Location       Grade       Age           Sex              Outstandin :                     Highly
                Not                                 not         Field:                   <50         Female:                  high BY            successful:
         promoted       Promoted              promoted             HQ          2         50+           male             successful               satisfactory -
                36.84
               122:23..   - ..-.   10.16
                                   37.77            .28
                                                    .31             1.1        3.0
                                                                               3.0          1.6
                                                                                            1.6                                                         ---.

               33.92                3.08            .09                                     1.6
             204.22                20.78            .I0             1.1                     1.6
               17.92                3.08            .I7                        3.0
             114.07                21.93            .19             1.1        3.0
               46.38 ------?.62                     .06
             233.23                14.77            .06.            1.1
                 3.12               1.88            .60                        3.0         2.2             2.2
               66.36               19.64            .30              .5        3.0         2.2             1.0
  ..-.         48.37                9.63            .20                                    2.2             2.2
             543.05                52.95            .I0              .5                    2.2             1.0                                                 -
              ..~1.57 -- __.. -..-    .43           .27                        3.0                         1.6
               90.88               12.12            .13              .5        3.0                          .7
.- ._     .~ 105.54                 9.46            .09                                                    1.6
            1314.29                57.71            .04              .5                                     .7

                                                   ‘Due to rounding, the odds ratios may not precisely match those that would be obtained from direct
                                                   calculation from the expected frequencies
                                                   bThese values were generated by the preferred model.

                                                   ‘The odds ratio was not directly calculable, but its value could be extrapolated   from the model.
                                                   dThe odds could not be calculated for this group




GM Promotions                                      We carried out a series of statistical analyses similar to our work on the
                                                   GS promotion data to test relationships   between promotions and GM
                                                   employee characteristics. These promotion analyses were based on
                                                   1,092 GM employees for whom we had complete age, grade, years in
                                                   grade, sex, ethnicity, ratings, location, and promotion data. Overall, 13
                                                   percent of the GM employees were promoted between July 1987 and
                                                   August 1988 and 87 percent were not.

                                                   To simplify our calculations and avoid an expanded cross-classification
                                                   of the data that would result in many sparse or empty categories, we
                                                   again did the analyses in two stages. In the first stage, we examined the
                                                   relationship of ethnicity and sex with promotions. These two variables
                                                   were examined separately for two reasons. First, there were only 116
                                                   females and 107 nonwhites in the Customs GM population. Because of


                                                   Page 38                                        GAO/GGD-90-40    Appraisals   and Promotions    at Customs
                                            Appendix I
                                            Technical Appendix: Loglinear     Methodology
                                            and Analysis Results




                                            the small numbers of females and nonwhites, it would have been diffi-
                                            cult to work with them in a multidimensional context. Second, prelimi-
                                            nary analysis showed that sex and ethnicity were independent of age,
                                            grade, years in grade, location, and rating in affecting promotion.
                                            Because sex and ethnicity did not interact with these other variables,
                                            they could be analyzed separately. In the second stage, we analyzed
                                            relationships between promotion and age, grade, years in grade, loca-
                                            tion, and rating. (Table I.1 1 shows the loglinear models testing this rela-
                                            tionship and the preferred model identified.) The odds and odds ratios
                                            from the preferred models are shown in tables I.12 and 1.13.


Table 1.11: GM Promotions: Logit Models Tested to Examine Relationships With Performance Ratings, Age, Grade, Years in
Grade, and Location
                                                                                               D”tg,‘e”e”d”o”m’ Likelihood ratio
Model number                               Models tested@                                                             chi-square
(1)                                        [RAGYL]    [P]                                                31                107.84
                                                                                                                         ____-
(2)”                                        [RAGYL]   [RP]   [API           [GP]    [VP]     [LPI                        26         --.         -- 24.77
(3)                                         [RAGYL]   [RP]   [API           [GP]    [VP]                                 27                         28.68
(4)                                         [RAGYL]   [RP]   [API           [GP]    [LPI                            -__  27                     __.-40.67
(5)                                         [RAGYL]   [RP]   [API           [YP]    [LPI                                 27                         44.89
(6)                                         [RAGYL]   [RP]
                                                       ___   [GP]           [VP]    [LPI                                 27
                                                                                                                     .__.__                         43.14
(7)                                         [RAGYL]   [API   [GP]           [VP]    [LPI                                 27                         48.82
(8)                                         [RAGYL] _____.__
                                                      [RAP] [VP]            [LPI    [GP]                                 25                  ___.- 24.54
(9)                                         [RAGYL]   [RGP] [API            [VP]    [LPI                       _________ 25             I_-         24.66
                                                                                                                                                I__.-
(10)                                        [RAGYL]   [RYP] [API            [GP]    [LPI                                 25                         21.84
(11)                                        [RAGYL]   [RLP] [API            [GP]    [VP]                                 25
                                                                                                                  I___.__-                    ------22.97
(12)                                        [RAGYL]   [AGP] [RP]            [VP]    [LPI                                   25
                                                                                                                        .___--...__                 24.73
(13)                                        [RAGYL]   [AYP] [RP]            [GP]    [LPI                               -__  25                      24.06
(14)                                        [RAGYL]   [ALP] [RP]            [GP]    [VP]                                    25                      24.7;
(15)                                        [RAGYL]   [GYP] [RP]            [API    [LPI                                   25           -24.17
                                                                                                                            2j----.-24.77
(16)                                        [RAGYL]   [GLP] [RP]            [API    [VP]                      __-.__
(17)                                        [RAGYL]   [YLP] [RP]            [API    [GP]                                   25                     21.02
                                            aThe notations used in this table are explained below:
                                            P = Promoted (competitively promoted or not promoted)
                                            R = Performance Rating (high or satisfactory)
                                            A = Age (under 50 or 50 and over)
                                            G = Grade (13 or 14-15)
                                            Y = Years in grade (under 2 or 2 and over)
                                            L = Location (headquarters or field)

                                            hModel 2 was the preferred model at the .05 significance level. The model states age, grade, years in
                                            grade, and locatron each had direct relationships with promotion.




                                            Page 36                                    GAO/GGD-90-40     Appraisals     and Promotions     at Customs
                                             Appendix    I
                                             Technical   Appendix:   Loglinear     Methodology
                                             and Analysis Results




Table 1.12: GM Promotions and Their Relationship With Sex and Ethnicity: Observed and Expected Frequencies, Odds, and Odds
Ratios From the Preferred Model
                                                                                           Odds on         Odds ratios*
                                            Observed frequencies   Expected frequenciesb  promotion      Sex      Ethnicity
                                                               Not                    Not     Promoted: Female:         White:
Sex                   Ethnicity
             _. --- .-.-..~.------__-_____- Promoted     promoted  Promoted     promoted  not promoted     male     nonwhite
Female            White                       20                77         20.74                 76.26             .27            1.9             2.0
                  Nonwhite              --     3                16          2.26                 16.74             .14            1.9
Male              White                      110              778        109.26              778.74                .I4                            2.0
                  Nonwhite                     5               83           5.74                 82.26             .07

                                             aDue to rounding, the odds ratios may not precisely match those that would be obtained from direct
                                             calculation from the expected frequencies.
                                             “These values were generated by the preferred model




                                             Page 37                                       GAO/GGD90-40   Appraisals     and Promotions   at Customs
                                         Appendix I
                                         Technical Appendix: Logliuear    Methodology
                                         and Analysis Results




Table 1.13: GM Promotions and Their
Relationship With Rating, Age, Grade,
Years in Grade, and Location: Observed
and Expected Frequencies, Odds, and                                                                             Observed frequencies
Odds Ratios From the Preferred Model                                               Years in                            Not
                                         Rating         Age              Grade       grade        Location       promoted    Promoted
                                         Highb          -< 50                13           1             I-IQ                11               2
                                                                                                      Field                34                5
                                                                                        2+              HQ                 36              19
                                                                                                      Field              154               49
                                                                         14-15            1             HQ                 31 I_---          5
                                                                                                      Field                36               0
                                                                                        2+              HQ               110               21
                                                                                                      Field              102               14
                                                        50+                  13           1             HQ                   2              0
                                                                                                      Field-                 2               0
                                                                                        2+              HQ                  13               3
                                                                                                      Field              27                 7
                                                                         14-15            1             HQ                   5               0
                                                                                                      Field                  3              0
                                                                                        2+              HQ                 44                1
                                                                                                      Field                41                2
                                         Satisfactory   < 50                 13           1             HQ                   3              d
                                                                                                      Field                31               0
                                                                                        2+              HQ                  17      ------  0
                                                                                                      Field               --53   __---      8
                                                                         14-15            1             HQ                  IO              0
                                                                                                      Field                18              -ti
                                                                                        2+              HQ                 18               0
                                                                                                      Field                22._I     __.~~ 1
                                                        50+                  13           1             HQ                 2                 0
                                                                                                      Field                1                 0
                                                                                        2+              HQ            ___-11                 1
                                                                                                      Field .-__         46 ___--            0
                                                                         14-15            1             HQ                 1                 0
                                                                                                      Field
                                                                                                      ____I_               1                 0
                                                                                        2+              HQ                15 ..-.---.-       1
                                                                                                      Field              24                  1




                                         Page 38                                  GAO/GGD-90-40   Appraisals   and Promotions   at Customs
                                             Appendix I
                                             Technical Appendix: Loglinear Methodology
                                             and Analysie Reeulta




                                                                                                            Odds ratio’
                                           Odds on                                                                             Years in
Expected frequenciesb                     promotion                           Rating              Age           brade           grade              Location
      Not                                    Promoted:                              High:c        < 50:                             2+:
promoted       Promoted                  not womoted                         satisfactorv          50+           14%                <2                    fL%
    11.31              1.69                               .I5                             3.9       3.0            2.5                                      1.5
    33.66  -. .-      3.34                                   .I0       -                  3.9        3.0           2.5
    36,70'         -iaT~o--.-.-                              .50                          3.9        3.0           2.5               3.3                      1.5
    52.63            50.37                                   .33                          3.9        3.0           2.5               3.3
    33.97             2.03                                   .06                          3.9        3.0                                                      1.5
    34.63 ~'           1.37                                  .04                    --    3.9        3.0
   109.24            21.76                                   .20                          3.9        3.0                             3.3                      1.5
   102.49            13.51                        ____-- .13                              3.9        3.0                             3.3
     1.90                 .I0                                .05-                         3.9                      2.5                                        1.5
     1.94                 .06                                .03                          3.9                      2.5
    13.71              2.29                                  .I7                          3.9                      2.5               3.3                      1.5
    57.62             6.36                                   .ll     ---                  3.9                      2.5               3.3
     4.90-                .lO                                .02                -         3.9                                                                 1.5
     2.96                 .04             ______             .Ol                          3.9d
    42.18 --          2.82                                   .07                          3.9                                        3.3                      1.5
    41.18              1.82                                  .04                          3.9                                        3.3
     2.89.                .ll- .---~__--                     .04                                     3.0           2.5                                  __-
                                                                                                                                                         -~- 1.5
    30.22.         -      .78  --~~~~~ -_.. -.. ..--         .03                                     3.0           2.5                              ___--
    15.oi              , :gj       -.-_-- .-_             .13                                        3.0           2.5               3.3                     1.5
    56.19             b.sl                    ------~_09                                             3.0           2.5               3.3
     9.85                 .I5 .- .-.-~~~~~~....              .02                                     3.0                                                      1.5
    17.82               ..118.                 ~--   ..-_ol                  -.                      3.0d
    17.12~               .88                                 .05                                     3.0                             3.3                      1.5
    22.24.-               .76                                .03                                     3.0                             3.3
     1.97                 .03._^.".._-.-__.__,-__            .02. -.-..-.-_-
                                                 I-... ...__~~~.                         ---                       2.5                                        1.5
       .99                .Ol                                .Ol                                                   2.5O
    11.50.               .50                                  04                                                   2.5               3.3             ---    1.5
    44.72              1.28                                  .03                                                   2.5               3.3             -.____
       .99                .Ol                                .Ol                                                                      ~-______              1.5d
     1.00                .oo                                     e
                                                        ~~ ..___~ ___
    15.73.               .27                                 .02                                                                     3.3.______.              1.5
    24.72                .28                                 .Ol                                                                     3.3d

                                             aDue to roundrng, the odds ratios may not precisely match those that would be obtained from direct
                                             calculation from the expected frequencies.
                                             ‘These values were generated by the preferred model.
                                             ‘High rating combines outstanding and highly successful.
                                             dThe odds ratio was not directly calculable, but its value could be extrapolated from the model.
                                             eThe odds could not be calculated for this group.




                                             Page 39                                             GAO/GGD-9040     Appraisals    and Promotions     at Customs
                                                                                                       “,
Appendix II                                                                                                 ,I

Major Contributors to This Report


                        Norman A. Stubenhofer, Assistant Director
General Government      Eva L. Rezmovic, Senior Social Science Analyst
Division, Washington,   Barry Reed, Senior Social Science Analyst
D.C.                    Annette Hartenstein, Staff Evaluator
                        Douglas Sloane, Statistical Consultant




(HA6320)                Page 40                       GAO/GGD-9040   Appraisals   and Promotions   at Customs
‘I’t~lt~pllont~ 202-27wi24   1




‘I‘lwre is a 25% discouti~. tm ordthrs for 100 or more copies mailtd   to a
singltb adtlrtwi.