oversight

DOD Personnel: Documentation of the Army's Civilian Workforce-Planning Model Needed to Enhance Credibility

Published by the Government Accountability Office on 2003-08-22.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

              United States General Accounting Office

GAO           Report to the Ranking Minority Member,
              Subcommittee on Readiness, Committee
              on Armed Services, House of
              Representatives

August 2003
              DOD PERSONNEL
              Documentation of the
              Army’s Civilian
              Workforce-Planning
              Model Needed to
              Enhance Credibility




GAO-03-1046
              a
                                                August 2003


                                                DOD PERSONNEL

                                                Documentation of the Army's Civilian
Highlights of GAO-03-1046, a report to the      Workforce-Planning Model Needed to
Ranking Minority Member, Subcommittee
on Readiness, Committee on Armed                Enhance Credibility
Services, House of Representatives




Between fiscal years 1989 and 2002,             The Army has taken adequate steps to ensure that the historical
the Department of Defense (DOD)                 personnel data used in the model are sufficiently reliable and that the
reduced its civilian workforce by               information technology structure adequately and appropriately supports
about 38 percent, with little
attention to shaping or specifically
                                                the model. For example, the Army has established adequate control
sizing this workforce for the future.           measures (e.g., edit checks, expert review, etc.) to ensure that the
As a result, the civilian workforce             historical data that goes into the model are sufficiently reliable.
is imbalanced in terms of the                   Moreover, it has taken adequate steps to ensure that the information
shape, skills, and experience                   technology support structure (i.e., the software and hardware used to
needed by the department. DOD is                interface with and house the model) would enable continuity of
taking steps to transform its                   operations, functionality, and system modification and operations.
civilian workforce. To assist with
this transformation, the department
is considering adopting an Army                 However, the Army has not demonstrated that it has taken adequate
workforce-planning model, known                 steps to ensure that the model’s forecasting capability provides the basis
as the Civilian Forecasting System              for making accurate forecasts of the Army’s civilian workforce. The
(CIVFORS), which the Army uses                  Army’s original certification of CIVFORS in 1987 was based on a formal
to forecast its civilian workforce              documented verification and validation of the model structure that has
needs. Other federal agencies are               not been formally updated since that time even though the Army has
also considering adopting this                  undertaken several model improvements. According to the Army’s
model. GAO was asked to review
the adequacy of the steps the Army              CIVFORS program manager, the Army has taken several steps, to include
has taken to ensure the credibility             an independent review, peer reviews, and a comparison of forecasted
of the model.                                   data to actual data. However, documentation of these steps is
                                                incomplete and, therefore, does not provide adequate evidence to
                                                demonstrate the credibility of the forecast results. Without adequate
To assure the reliability of Army
                                                documentation, the Army cannot show that it has taken sufficient steps
civilian workforce projections and              to ensure the model’s credibility in terms of its forecasting capability;
the appropriateness of the model                consequently, there exists a risk that the forecasts it produces may be
for use DOD-wide and by other                   inaccurate or misleading. Furthermore, without documentation of
federal agencies, we recommend                  CIVFORS’s forecasting capability, it may be difficult for DOD and other
that the Secretary of Defense direct            federal organizations to accurately determine its suitability for their use.
the Secretary of the Army to
appropriately document the
forecasting capability of the model.

Although DOD stated, in written
comments on a draft of this report,
it did not concur with GAO’s
recommendation, the Army is
taking actions that, in effect,
implement it.



www.gao.gov/cgi-bin/getrpt?GAO-03-1046.

To view the full product, including the scope
and methodology, click on the link above.
For more information, contact Derek Stewart,
202-512-5559, stewartd@gao.gov.
Contents


Letter                                                                                                   1
              Results in Brief                                                                          2
              Background                                                                                3
              Civilian Workforce-Planning Model’s Data Reliability and
                Information Technology Structure Are Adequate, but
                Forecasting Ability Not Fully Established                                               4
              Conclusions                                                                               8
              Recommendation for Executive Action                                                       8
              Agency Comments and Our Evaluation                                                        8
              Scope and Methodology                                                                     9

Appendix I    Comments from the Department of Defense                                                   11



Appendix II   GAO Contact and Staff Acknowledgments                                                     13




              Abbreviations

              CIVFORS           Civilian Forecasting System
              DOD               Department of Defense
              WASS              Workforce Analysis Support System


              This is a work of the U.S. government and is not subject to copyright protection in the
              United States. It may be reproduced and distributed in its entirety without further
              permission from GAO. However, because this work may contain copyrighted images or
              other material, permission from the copyright holder may be necessary if you wish to
              reproduce this material separately.




              Page i                                                      GAO-03-1046 DOD Personnel
United States General Accounting Office
Washington, DC 20548




                                   August 22, 2003

                                   The Honorable Solomon P. Ortiz
                                   Ranking Minority Member
                                   Subcommittee on Readiness
                                   Committee on Armed Services
                                   House of Representatives

                                   Dear Mr. Ortiz:

                                   Between fiscal years 1989 and 2002, the Department of Defense (DOD)
                                   reduced its civilian workforce from 1,075,437 to 670,166—about a 38
                                   percent reduction—with little attention to shaping or specifically sizing
                                   this workforce for the future. As a result, the civilian workforce is
                                   imbalanced in terms of the shape, skills, and experience needed by the
                                   department. DOD plans to downsize its civilian workforce by an additional
                                   55,000 through fiscal year 2007. In addition, in April 2003, DOD submitted a
                                   proposal to Congress that would authorize DOD to establish a National
                                   Security Personnel System to transform its current civilian personnel
                                   system.1 DOD is also exploring the feasibility of placing hundreds of
                                   thousands of civilians into essentially nonmilitary jobs that are currently
                                   performed by uniformed personnel. To assist in determining its future
                                   workforce, DOD will need reliable workforce planning tools, such as
                                   workforce projection models. According to DOD guidance, a model used
                                   to provide data for decision making should be accredited--that is, the
                                   model should be officially certified as acceptable for use for a specific
                                   purpose.

                                   In a February 2003 testimony, the Chief of Staff of the Army stated that the
                                   Army has begun to transform its civilian personnel system. To assist with
                                   this transformation, the Army is using its workforce-planning model,
                                   known as the Civilian Forecasting System (CIVFORS), which forecasts
                                   future civilian workforce needs.2 The Army is working closely with the



                                   1
                                    The system is proposed in the Transformation for the 21st Century Act of 2003. The act
                                   also proposes other wide-ranging changes, affecting civilian personnel pay and
                                   performance management, collective bargaining, rightsizing, and other human capital
                                   areas.
                                   2
                                    The Civilian Forecasting System was adapted from an Army military forecasting model for
                                   civilian use in 1987.



                                   Page 1                                                      GAO-03-1046 DOD Personnel
                   Office of the Secretary of Defense and the Office of Personnel
                   Management to demonstrate the applicability of the model for use DOD-
                   wide and in other federal agencies. According to Army guidance (Army
                   Regulation 5-11 and Department of the Army Pamphlet 5-11), verification
                   is one of the steps needed to ensure a model’s suitability to perform its
                   intended purpose. The verification process evaluates the extent to which a
                   model has been developed using sound and established software
                   engineering techniques, and it establishes whether the computer code
                   correctly performs the intended functions. Army guidance also states that
                   assessment of the correctness and forecasting capability is required.

                   In this report, we reviewed the adequacy of the steps the Army has taken
                   to ensure the credibility of the model. In March 2003, we briefed your staff
                   on our preliminary findings. To determine the Army’s efforts to ensure the
                   credibility of its model, we interviewed and obtained pertinent
                   documentation from the Army’s CIVFORS program manager. We also
                   reviewed DOD and Army guidance relevant to the management of Army
                   models and interviewed DOD officials to discuss their plans to adopt
                   CIVFORS. We conducted our review from September 2002 to June 2003.
                   More detailed information on our scope and methodology appears at the
                   end of this report.


                   The Army’s steps were adequate to ensure that the historical personnel
Results in Brief   data used in the model are sufficiently reliable and that the information
                   technology support structure3 adequately and appropriately supports the
                   model, but the Army has not documented its steps to ensure the credibility
                   of the model’s forecasting capability. The Army has established adequate
                   control measures (e.g., edit checks, expert review, etc.) to ensure that the
                   historical data that goes into the model are sufficiently reliable. Moreover,
                   it has taken adequate steps to ensure that the information technology
                   support structure would enable continuity of operations, functionality, and
                   system modification and operations. However, the Army has not
                   documented that it has taken adequate steps to ensure that the model’s
                   structure (including its forecasting capability and the appropriateness of
                   its assumptions) provides the basis for making accurate forecasts of the
                   Army’s civilian workforce. The Army’s original certification of CIVFORS in
                   1987 was based on a formal documented verification and validation of the
                   model structure that has not been formally updated since that time, even


                   3
                       The software and hardware used to interface with and house the model.




                   Page 2                                                       GAO-03-1046 DOD Personnel
             though the Army has undertaken several model improvements. According
             to the Army’s CIVFORS program manager, the Army has taken several
             steps, to include an independent review, peer reviews, and a comparison
             of forecasted data to actual data. However, documentation of these steps
             is incomplete and, therefore, does not provide adequate evidence to
             demonstrate the credibility of the forecast results. Without adequate
             documentation, the Army cannot show that it has taken sufficient steps to
             ensure the credibility of the model’s forecasting capability; consequently,
             there exists a risk that the forecasts it produces may be inaccurate or
             misleading. Furthermore, without documentation of CIVFORS’s
             forecasting capability, it may be difficult for DOD and other federal
             organizations to accurately determine its suitability for their use.

             We are recommending that the Secretary of Defense direct the Secretary
             of the Army to appropriately document the Army’s forecasting capability
             of the civilian workforce-planning model. Although DOD stated, in written
             comments on a draft of this report, it did not concur with our
             recommendation, the Army is taking actions that, in effect, implement it.


             According to an Army Human Resource official, the Army uses the
Background   workforce-planning model—CIVFORS—for human resources
             management. CIVFORS is a collection of software programs that
             anticipate future impacts on the workforce so that management can plan
             for changes instead of reacting to them. The model is used to evaluate a
             number of critical areas in civilian workforce planning, including
             projected recruitment of personnel, impact of organizational realignments,
             and changes in workforce trends (such as aging, retention, and projected
             personnel shortfalls). It is a life-cycle modeling and projection tool that
             models the most significant events that describe the life-cycle path of
             personnel, which includes accessions, promotions, reassignments,
             retirements, and voluntary and involuntary separations over a 7-year
             period.

             Verification and validation of models are important steps to building
             credible models because they provide the foundation for the accreditation
             process to ensure the suitability of the models for their intended purposes,
             as stated in Army guidance, Management of Army Models and




             Page 3                                            GAO-03-1046 DOD Personnel
                            Simulations.4 The verification process evaluates the extent to which a
                            model has been developed using sound and established software
                            engineering techniques, and it establishes whether the model’s computer
                            code correctly performs the intended functions. Model verification
                            includes data verification, model documentation, and testing of the
                            information technology structure that supports the model; model
                            verification is contained in such documents as the programmer’s manual,
                            installation’s manual, user’s guide, analyst’s manual, and trainer’s manual.
                            According to Army guidance, assessment of the correctness and
                            forecasting capability of the model is also required, and it should be
                            performed by a subject matter expert independent from the model
                            developer; however, the developer is expected to conduct in-house
                            verification and testing to assist in the overall model development process.
                            Validation is the process of determining the extent to which the model
                            adequately represents the real world.


                            The Army has taken steps to ensure the reliability of the historical
Civilian Workforce-         personnel data used by the model and the adequacy of its information
Planning Model’s Data       technology structure used to support the model, but it has not provided
                            documentation that it has sufficiently tested and reviewed the most critical
Reliability and             aspect of the model—its forecasting capability and the appropriateness of
Information                 its assumptions. As a result, the forecasting credibility of the current
                            version of the model is not sufficiently validated or documented. Without
Technology Structure        proper documentation of the abilities of the model, there is a risk that the
Are Adequate, but           forecasts it produces may be inaccurate or misleading and the suitability
Forecasting Ability         for use by other organizations may be difficult to determine.

Not Fully Established

Historical Personnel Data   The Army’s review of the historical personnel data used to provide
Reliability Is Adequate     information for workforce planning was adequate to show that the data
                            are sufficiently reliable for use in the workforce model. Data regarding
                            personnel (such as date hired, education, age, grade level, and
                            occupational series) are taken from the Army’s Workforce Analysis



                            4
                             Headquarters, Department of the Army, Management of Army Models and Simulations,
                            Army Regulation 5-11 (Washington, D.C., July 10, 1997). This regulation prescribes policy
                            and guidance and assigns responsibilities for the management of Army models and
                            simulations, including development and maintenance.




                            Page 4                                                       GAO-03-1046 DOD Personnel
                         Support System (WASS).5 CIVFORS uses the most recent 5 years of
                         historical data to forecast the civilian workforce planning needs during the
                         next 7 years.

                         According to Army guidance, to ensure that data are sufficiently reliable
                         for use in the Army model, support documents should contain information
                         about the overall characteristics of the database. Furthermore, the
                         documents should show the intended range of appropriate uses for the
                         model as well as constraints on its use. They should also include concise
                         statements of the condition of the database for the purpose of indicating
                         its stability. The Army provided most, but not all, of the documents
                         referred to in Army guidance; we believe that the documents provided are
                         key ones and are adequate to show that WASS data are sufficiently reliable
                         for use in CIVFORS. In addition, the Army program manager for the
                         CIVFORS workforce-planning model stated that the workforce data are
                         checked by reviewing the arithmetic in the numerical algorithms to verify
                         that there is no unexplained change in the size of the civilian personnel
                         workforce contained in the database. Further, edit checks include
                         matching social security numbers for personnel from one time period to
                         another to account for actual personnel and personnel transactions
                         processed. In addition, CIVFORS has automated checks for inappropriate
                         numbers or characters. Such steps help to assure that the data contained
                         in WASS accurately and completely reflect critical personnel aspects and
                         transactions.


Information Technology   The Army’s procedures for validating the information technology support
Structure Is Adequate    structure (the software and hardware used to interface with and house the
                         model) were also sufficient. For example, the Army (1) adequately
                         documented the information technology structure to allow for continuity
                         of operations, (2) tested its functionality, and (3) provided expertise for
                         system modification and operation. Procedures used by the Army include
                         documenting the model’s system description and hardware and software
                         requirements, providing system and user manuals, planning for
                         configuration management,6 and conducting functionality tests to help



                         5
                          WASS enables analysis of data on Army civilians from 1974 to the present. It has analysis
                         capabilities that range from frequency distribution to trend analysis.
                         6
                          Configuration management is the control and documentation of changes made to system
                         hardware, software, and documentation throughout the development and operational life
                         of the system.




                         Page 5                                                       GAO-03-1046 DOD Personnel
                              ensure the system’s usability and operability over time and to demonstrate
                              the adequacy of the information technology structure to support use of the
                              workforce model.


Model’s Forecasting Ability   The Army’s documentation cannot show that the forecasting ability of
Is Not Fully Established      CIVFORS has been adequately evaluated and, therefore, we cannot fully
                              assess the credibility of the model. According to Army guidance, validation
                              is the process of determining the extent to which a model adequately
                              represents the real world. According to the Army program manager, over a
                              7-year period, CIVFORS forecasts the anticipated impacts on the
                              workforce based on the most significant events in the life-cycle path of
                              personnel (to include accessions, promotions, reassignments, retirements,
                              voluntary separations, and involuntary separations). Army guidance states
                              that an independent, peer, and subject matter expert review of the model
                              should be conducted. The Army guidance also suggests generally accepted
                              methods, such as conducting a careful line-by-line examination of the
                              model design and computer code and algorithms. The Army’s program
                              manager said this had been done for the original certification of CIVFORS
                              in 1987. However, no formal document of the reviews has been prepared
                              in the years since, even though the Army has undertaken several model
                              improvements, such as (1) an expanded scope to include more dimensions
                              in the modeling process; (2) a more integrated, streamlined process that
                              involves fewer steps; and (3) greater flexibility, achieved by generalizing
                              the formulas and parameters.

                              In addition, there is insufficient documentation regarding tests performed,
                              since 1987, in which CIVFORS’s forecasts for prior years are compared
                              against equivalent historical data (called an “out of sample” test) to
                              measure the model’s forecasting capability. Such testing, which is one
                              method to validate a model’s forecasting capability, would involve using
                              the first 5 of the last 7 years of historical data to forecast the 2 subsequent
                              years. The forecasts for the last 2 years could then be compared to the
                              actual historical data. The Army, however, performed tests comparing
                              patterns of forecasts against historical data (called “in sample” tests),
                              showing that forecasts reflect the same patterns as the historical data used
                              to develop them for a sample of three Army major commands. However,
                              the draft document that was provided to us was inadequate to fully assess
                              the sampling used by the Army and the value of the tests.

                              Finally, the Army could not provide adequate documentation of an
                              independent or peer review of the model. The Army’s CIVFORS program
                              manager stated that the major commands served as peer reviewers by


                              Page 6                                              GAO-03-1046 DOD Personnel
conducting a comparison of their workforce data to WASS and CIVFORS
workforce data. We believe that such assessments by users provide
important information but do not constitute a peer review as defined in
Army guidance. Also, the results of these assessments were not available
for us to review. The program manager also stated that an independent
subject matter expert reviewed the functional design and the code in 1999,
but a formal report of the activities performed and the specific changes or
modifications implemented during the review were not produced.

Documentation has often not been a priority for several reasons.
According to the Army’s CIVFORS program manager, lack of
documentation is primarily due to limited funding, which was spent on
implementing changes to CIVFORS and WASS rather than on the
production of formal documents. Further, a shortage of staff (only one
staff person—the program manager) and loss of documents during the
attack on the Pentagon on September 11, 2001, also affected the amount of
documentation the Army could provide us. The program manager also
stated that some documentation was not needed because CIVFORS’s
design is predicated on proven methods in other Army active-duty, military
manpower forecasting models. In addition, the program manager stated
that the Army and contractors have primarily been adapting technology
(upgrading from mainframe to personal computer to Web-based) to
improve model functionality rather than creating new technology.
However, without proper documentation of the abilities of the model,
there exists a risk that the forecasts it produces may be inaccurate or
misleading. Consequently, decisions about future workforce requirements
may be questionable, and planning for the size, shape, and experience
level of the future workforce may not adequately meet the Army’s needs.

These issues may extend beyond the Army. In April 2002, DOD published a
strategic plan for civilian personnel, which includes a goal to obtain
management systems to support workforce planning. According to a DOD
official responsible for civilian workforce planning tools, components
within DOD have been requesting a modeling tool to assist them with
civilian workforce planning. As a result, DOD has decided to test the
Army’s civilian forecasting model. In October 2002, DOD purchased
hardware, installed modified software, and provided training to a small
number of personnel. Recently, DOD obtained a historical database of
civilian personnel data from the Defense Management Data Center and
provided the database to the contractor to load into the model. Two
agencies have volunteered to test the model: the Defense Logistics Agency
and the Washington Headquarters Service. DOD is working to develop a
test for these organizations using their own civilian personnel data to test


Page 7                                            GAO-03-1046 DOD Personnel
                     the model. At the end of the testing period, DOD will assess the model to
                     obtain a better understanding of its logic and determine whether or not it
                     should be implemented departmentwide.


                     As DOD continues to transform and downsize its civilian workforce, it is
Conclusions          imperative that the department properly shape and size the workforce.
                     One tool that could assist in this effort is CIVFORS—the Army’s workforce
                     planning model. However, proper documentation of the verification and
                     validation of CIVFORS is needed before expanding its use. The Army has
                     taken adequate steps to ensure that the historical personnel data used in
                     the model are sufficiently reliable and the information technology
                     structure appropriately supports the model; however, it has not fully
                     documented that it has taken adequate steps to demonstrate the credibility
                     of the model’s forecasting capability. Further, a model should be fully
                     scrutinized before each new application because a change in purpose,
                     passage of time, or input data may invalidate some aspects of the existing
                     model. Without sufficient documentation to demonstrate that adequate
                     steps have been taken to ensure the credibility of the model’s forecasting
                     capabilities, decisions about the Army’s future civilian workforce may be
                     based on questionable data and other potential users cannot determine
                     with certainty the model’s suitability for their use.


                     To assure the reliability of Army civilian workforce projections, as well as
Recommendation for   the appropriateness of the model for use DOD-wide and by other federal
Executive Action     agencies, we recommend that the Secretary of Defense direct the
                     Secretary of the Army to appropriately document the Army’s forecasting
                     capability of the model.


                     Although DOD stated, in written comments on a draft of this report, that it
Agency Comments      did not concur with our recommendation, the Army is taking actions that,
and Our Evaluation   in effect, implement it. DOD’s written comments are contained in
                     appendix I.

                     Regarding our recommendation that the Secretary of Defense direct the
                     Secretary of the Army to appropriately document the Army’s forecasting
                     capability of the model, DOD stated that the Army recognizes the need to
                     fully document its verification and validation efforts. Further, DOD stated
                     the staff of the Assistant Secretary of the Army, Manpower and Reserve
                     Affairs, has begun developing a verification and validation plan to enable
                     outside parties to assess the suitability and adaptability of the model for


                     Page 8                                             GAO-03-1046 DOD Personnel
              their organizational use. This verification and validation process is
              scheduled for completion in September 2003. However, during our review,
              DOD did not provide information about the full scope of this verification
              and validation effort. We believe that as the Army undertakes its
              verification and validation effort, it should clearly document, as we
              recommended, its assumptions, procedures, and the results so that future
              users can replicate the tests to appropriately establish the model’s validity
              for their purposes.

              DOD also did not concur with our finding that the forecasting ability of the
              model has not been fully established. DOD stated that the ultimate test of a
              system is performance and that CIVFORS has been consistently generating
              Army projections with high standards of accuracy. We did not
              independently evaluate the model’s accuracy. As our report makes clear,
              our basic point is that the model’s forecasting ability has not been
              documented in accordance with Army guidance. We continue to believe
              that without adequate documentation, the Army cannot show that it has
              taken sufficient steps to ensure the model’s credibility in terms of its
              forecasting capability. DOD also provided technical comments, which we
              incorporated where appropriate.


              We did not independently evaluate the model or the application of the
Scope and     steps; rather, we reviewed the adequacy of the steps that the Army
Methodology   program manager stated were taken to ensure the credibility of the model.
              To determine the adequacy of the steps the Army has taken to ensure the
              credibility of its civilian workforce-forecasting model, we discussed
              CIVFORS with the Army’s CIVFORS program manager in the Army G-1
              office, Civilian Personnel Policy Directorate, who has overall
              responsibility for the workforce analysis and the forecasting system. In
              addition, Army contractor officials who are responsible for providing
              technical, analytic, and management support to operate, maintain, and
              enhance the planning tool and model participated in several of our
              discussions with the program manager. We reviewed the following
              CIVFORS’s documents regarding the information technology support
              structure: the Configuration Management Manual, the System’s
              Specifications, the Design/Subsystem Documentation, the Operator’s
              Manual, and the User’s Manual. In addition, we reviewed the 1987 and
              draft 2002 test analysis report on the Civilian Forecasting System and
              other documentation provided by the Army to obtain information on how
              the model operates according to model assumptions. We also reviewed the
              DOD Defense Modeling and Simulation Office guidance on verification and
              validation of models, the Army regulation and pamphlet pertaining to the


              Page 9                                             GAO-03-1046 DOD Personnel
management of Army models and simulations, and other literature
regarding model credibility. We also interviewed DOD officials in the
Civilian Personnel Management Service responsible for developing plans
to adopt the Army’s workforce forecasting model to discuss the status of
their efforts.

We conducted our review from September 2002 to June 2003 in
accordance with generally accepted government auditing standards.


We are sending copies of this report to the appropriate congressional
committees, the Secretary of Defense, the Under Secretary of Defense for
Personnel and Readiness, and the Secretary of the Army. We will also
make copies available to others upon request. In addition, the report will
be available at no charge on the GAO Web site at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me
at (202) 512-5559. Key contributors to this report are listed in appendix II.

Sincerely yours,




Derek B. Stewart
Director, Defense Capabilities
 and Management




Page 10                                            GAO-03-1046 DOD Personnel
             Appendix I: Comments from the Department
Appendix I: Comments from the Department
             of Defense



of Defense




             Page 11                                    GAO-03-1046 DOD Personnel
Appendix I: Comments from the Department
of Defense




Page 12                                    GAO-03-1046 DOD Personnel
                  Appendix II: GAO Contact and Staff
Appendix II: GAO Contact and Staff
                  Acknowledgments



Acknowledgments

                  Christine Fossett (202) 512-2956
GAO Contact
                  In addition to the name above, David Dornisch, Barbara Johnson, Barbara
Acknowledgments   Joyce, John Smale, Dale Wineholt, and Susan Woodward made significant
                  contributions to this report.




(350385)
                  Page 13                                         GAO-03-1046 DOD Personnel
                         The General Accounting Office, the audit, evaluation and investigative arm of
GAO’s Mission            Congress, exists to support Congress in meeting its constitutional responsibilities
                         and to help improve the performance and accountability of the federal
                         government for the American people. GAO examines the use of public funds;
                         evaluates federal programs and policies; and provides analyses,
                         recommendations, and other assistance to help Congress make informed
                         oversight, policy, and funding decisions. GAO’s commitment to good government
                         is reflected in its core values of accountability, integrity, and reliability.


                         The fastest and easiest way to obtain copies of GAO documents at no cost is
Obtaining Copies of      through the Internet. GAO’s Web site (www.gao.gov) contains abstracts and full-
GAO Reports and          text files of current reports and testimony and an expanding archive of older
                         products. The Web site features a search engine to help you locate documents
Testimony                using key words and phrases. You can print these documents in their entirety,
                         including charts and other graphics.
                         Each day, GAO issues a list of newly released reports, testimony, and
                         correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site
                         daily. The list contains links to the full-text document files. To have GAO e-mail
                         this list to you every afternoon, go to www.gao.gov and select “Subscribe to e-mail
                         alerts” under the “Order GAO Products” heading.


Order by Mail or Phone   The first copy of each printed report is free. Additional copies are $2 each. A
                         check or money order should be made out to the Superintendent of Documents.
                         GAO also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a
                         single address are discounted 25 percent. Orders should be sent to:
                         U.S. General Accounting Office
                         441 G Street NW, Room LM
                         Washington, D.C. 20548
                         To order by Phone:     Voice:    (202) 512-6000
                                                TDD:      (202) 512-2537
                                                Fax:      (202) 512-6061


                         Contact:
To Report Fraud,
                         Web site: www.gao.gov/fraudnet/fraudnet.htm
Waste, and Abuse in      E-mail: fraudnet@gao.gov
Federal Programs         Automated answering system: (800) 424-5454 or (202) 512-7470


                         Jeff Nelligan, Managing Director, NelliganJ@gao.gov (202) 512-4800
Public Affairs           U.S. General Accounting Office, 441 G Street NW, Room 7149
                         Washington, D.C. 20548