oversight

Federal Training Investments: Office of Personnel Management and Agencies Can Do More to Ensure Cost-Effective Decisions

Published by the Government Accountability Office on 2012-09-17.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                 United States Government Accountability Office

GAO              A Report to the Chairman, Subcommittee on
                 Oversight of Government Management, the
                 Federal Workforce, and the District of Columbia;
                 Committee on Homeland Security and
                 Government Affairs, United States Senate

September 2012
                 FEDERAL TRAINING
                 INVESTMENTS

                 Office of Personnel
                 Management and
                 Agencies Can Do More
                 to Ensure Cost-
                 Effective Decisions




GAO-12-878
                                               September 2012

                                               FEDERAL TRAINING INVESTMENTS
                                               Office of Personnel Management and Agencies Can
                                               Do More to Ensure Cost-Effective Decisions
Highlights of GAO-12-878, a report to the
Chairman, Subcommittee on Oversight of
Government Management, the Federal
Workforce, and the District of Columbia;
Committee on Homeland Security and
Governmental Affairs, United States Senate

Why GAO Did This Study                         What GAO Found
OPM and agency CHCOs play an                   Many Chief Human Capital Officers (CHCOs) reported that they are
important role in ensuring that federal        implementing several leading practices important to making strategic decisions
training dollars are invested effectively.     about training delivery, such as determining the best mix of decentralized and
GAO was asked to review the extent to          centralized training and considering government-wide reform when planning
which: (1) CHCOs of selected federal           training. However, many CHCOs reported they are not implementing some
agencies have established processes            practices that support making more cost-effective training investment decisions,
to set and prioritize training                 such as prioritizing training so that the most important needs are met first and
investments that are aligned with              evaluating the benefits of training. In addition, many CHCOs do not have
leading practices; and (2) OPM’s
                                               information from component or sub-agency leaders regarding their level of
guidance and assistance for
                                               investments and priorities. Consequently, some agencies are duplicating internal
developing training investment
strategies align with these leading
                                               training investments and missing opportunities to leverage economies of scale
practices. GAO obtained information            across their agencies. Federal agencies also need reliable information on how
from 27 CHCOs on their training                much they spend on training and for what purposes. However, several CHCOs
investment practices through a                 reported they do not completely and reliably track training costs agency-wide.
questionnaire, and selected four               The Office of Personnel Management (OPM) provides guidance and assistance
agencies—the Departments of Energy             to agencies on a number of the leading practices, such as evaluating the benefits
(DOE), Homeland Security (DHS), the            of training in three of its guides and in workshops. In some practice areas that
Interior (DOI) and Veterans Affairs
                                               are challenges to agencies, such as prioritization of investments and determining
(VA)—to provide illustrative examples.
                                               whether to design training and development programs in-house or obtain these
We compared both CHCO and OPM
practices to leading practices,                services from a contractor, guidance is minimal or absent. OPM also requires
identified through past GAO and expert         agencies to submit training investment data and provides guidance on how to do
studies.                                       so, but considers this data to be unreliable because it is incomplete. However,
                                               OPM officials have not internally assessed improvements in the completeness of
                                               the data over the last 3 years or the quality of the data in the six years that
What GAO Recommends                            agencies have been required to submit it, and have only provided agencies with
GAO recommends, among other                    one summary of their data for correction. Agencies and OPM reported there are
things, that OPM improve guidance              also opportunities for OPM to help agencies reduce duplicative investments
and assistance to agencies in                  across agencies. For example, currently, agencies independently purchase or
establishing a process for setting and         develop training for the same mandated or common occupational training.
prioritizing training investments;             Agency leaders and OPM recognize that this has led to redundant and inefficient
improve the reliability of agency              federal training investments. According to OPM officials, HR University—which is
training investment information; and           a website currently administered by OPM to provide training for the HR
identify the best existing courses that        community—has already resulted in a cost savings of $14.5 million as a result of
fulfill governmentwide training                sharing the best HR training government-wide. Several agencies and OPM
requirements, and offer them to all
                                               officials reported that HR University could be expanded to provide mandatory
agencies through the HR University or
                                               training and serve as a model for centralizing training in other occupations or
other appropriate platforms. OPM fully
or partially concurred with four               functional areas, which could save millions more and help standardize training.
recommendations and did not concur
with a portion of another. OPM, DOI
and VA provided technical comments,
which GAO incorporated, as
appropriate, into the report. DOE and
DHS had no comments.

View GAO-12-878. For more information,
contact Yvonne J. Jones at (202) 512-2717 or
Jonesy@gao.gov.

                                                                                      United States Government Accountability Office
Contents


Letter                                                                                       1
               Background                                                                    5
               CHCOs are Implementing Leading Practices Related to Training
                 Delivery, but Not Practices that Support Making More Cost-
                 Effective Training Investments                                            10
               OPM Guidance and Assistance to Agencies on Federal Training
                 Investments Addresses Many Leading Practices that We
                 Identified, But Could be Improved in Areas That Are Challenges
                 to Agencies                                                               29
               Conclusions                                                                 45
               Recommendations for Executive Action                                        46
               Agency Comments and Our Evaluation                                          48

Appendix I     Objectives, Scope, and Methodology                                          52



Appendix II    Table of Statutes, Regulations, and Executive Orders Related to
               Leading Practices                                                           57



Appendix III   Illustrations of Building a Business Case for Training from OPM’s
               Guide to Strategically Planning Training and Measuring Results              59



Appendix IV    Comments from the Office of Personnel Management                            61



Appendix V     GAO Contact and Staff Acknowledgments                                       66



Tables
               Table 1: Leading Training Investment Practices                                3
               Table 2: OPM’s Guidance Documents and Description                             6
               Table 3: The Leading Training Investment Practices CHCOs
                        Reported Implementing                                              11
               Table 4: Leading Training Investment Practices that Many CHCOs
                        Reported Not Implementing Routinely                                21



               Page i                                   GAO-12-878 Federal Training Investments
          Table 5: Comparison of Agency Implementation of Leading
                   Practices and OPM Guidance and Assistance                                        29
          Table 6: Selection of Case Example Agencies                                               55


Figures
          Figure 1: OPM illustration of questions used to develop a business
                   case for training                                                                59
          Figure 2: OPM illustration of developing a business case for
                   training                                                                         60




          Abbreviations
          HCO             chief human capital officer
          CLO             chief learning officer
          DHS             Department of Homeland Security
          DOE             Department of Energy
          EHRI            Enterprise Human Resources Integration
          GPRA            Government Performance and Results Act
          GEAR            Goals-Engagement-Accountability-Results
          GPRAMA          GPRA Modernization Act of 2010
          HR              human resources
          DOI             Department of the Interior
          OPM             Office of Personnel Management
          TED             Training and Executive Development
          VA              Department of Veterans Affairs
          VALU            Veterans Affairs Learning University
          VBA             Veterans Benefits Administration
          VHA             Veterans Health Adminis



          This is a work of the U.S. government and is not subject to copyright protection in the
          United States. The published product may be reproduced and distributed in its entirety
          without further permission from GAO. However, because this work may contain
          copyrighted images or other material, permission from the copyright holder may be
          necessary if you wish to reproduce this material separately.




          Page ii                                           GAO-12-878 Federal Training Investments
United States Government Accountability Office
Washington, DC 20548




                                   September 17, 2012

                                   The Honorable Daniel K. Akaka
                                   Chairman
                                   Subcommittee on Oversight of Government Management,
                                     the Federal Workforce, and District of Columbia
                                   Committee on Homeland Security and Governmental Affairs
                                   United States Senate

                                   Dear Chairman Akaka:

                                   Constrained budgets and the need to address gaps in critical federal skills
                                   and competencies make it is essential that agencies identify the
                                   appropriate level of investment and establish priorities for employee
                                   training and development, so that the most important training needs are
                                   addressed first.1 The Office of Personnel Management (OPM) and
                                   agency chief human capital officers (CHCOs) play important roles in
                                   ensuring that federal training dollars are invested wisely. OPM provides
                                   leadership and guidance on the establishment and operation of efficient
                                   federal training and development programs government-wide and advises
                                   the President on means for furthering and strengthening federal training
                                   programs. CHCOs statutorily have the primary role in setting the
                                   workforce development strategies of their agencies, and advising and
                                   assisting their agency heads who are responsible for establishing
                                   priorities for needed training and development, and providing for the use
                                   of funds and resources in accordance with these priorities. The decisions
                                   that these key players make in determining how to invest federal training
                                   dollars will ultimately have an important impact on how well the federal
                                   government is equipped to address its current and future performance
                                   and fiscal goals and challenges.

                                   In order to better understand how federal training investment decisions
                                   are made and whether improvements are needed, you asked us to review
                                   the methods that agencies are using to establish their training investment
                                   strategies and OPM’s training investment guidance to agencies.


                                   1
                                    In the February 2011 update of our High Risk Series, we reported that the most
                                   significant human capital challenge to the government is to close current and emerging
                                   critical skills gaps. GAO, High-Risk Series: An Update, GAO-11-278 (Washington, D.C.:
                                   February 2011).




                                   Page 1                                          GAO-12-878 Federal Training Investments
Accordingly, this review assesses the extent to which (1) CHCOs of
selected federal agencies have established processes to set and prioritize
training investments that are aligned with leading practices and (2) OPM’s
guidance and assistance for developing training investment strategies
align with these leading practices.

For the purposes of this review, we define the key terms “training”,
“development” and “agency-wide” in the following ways:

•   Training is making available to employees planned and coordinated
    educational programs of instruction in professional, technical, or other
    fields that are or will be related to the employee’s job responsibilities.
    Training can be accomplished through a variety of approaches, such
    as classroom training, e-learning, and professional conferences that
    are educational or instructional in nature. 2

•   Development is generally considered to include training, structured
    on-the-job learning experiences, and education. Developmental
    programs can include experiences such as coaching, mentoring, or
    rotational assignments.

•   Agency-wide includes all components, sub-agencies or offices within
    a cabinet department or independent agency.

For both objectives of the review, we compared OPM and CHCO
practices against eight federal training investment leading practices,
which are based on our prior studies; other expert studies; and statutory,
regulatory and executive order training requirements. 3 (See table 1). OPM
reviewed these criteria and agreed that they are practices that agencies
should be implementing to support effective training investment decisions.
While some of the leading practices we identified are related to statutory,
regulatory and executive order training requirements, the leading practice
questions we examined were not developed to assess whether agencies



2
 This definition of training includes mandatory training that governs how employees
conduct themselves when carrying out their responsibilities, such as mandatory Equal
Employment Opportunity training or Information Technology Security training.
3
 All of these practices are described in GAO, Human Capital: A Guide for Assessing
Strategic Training and Development Efforts for the Federal Government, GAO-04-546G
(Washington, D.C.: March 2004). We also used several studies from the Corporate
Leadership Council.




Page 2                                          GAO-12-878 Federal Training Investments
are compliant with these requirements. 4 OPM officials agreed agency
responses to our questions do not indicate whether or not agencies are in
compliance with statutory, regulatory, and executive order training
requirements. Therefore, we do not make determinations regarding
compliance with these requirements in this review.

Table 1: Leading Training Investment Practices

Agencies should implement:
Practice 1: (a) Identify the appropriate level of investment to provide for training and
            development efforts and (b) prioritize funding so that the most important
            training needs are addressed first.
Practice 2: Identify the most appropriate mix of centralized and decentralized
            approaches for its training and development programs.
Practice 3: Consider government-wide reforms and other targeted initiatives to improve
            management and performance when planning its training and development
            programs.
Practice 4: Have criteria for determining whether to design training and development
            programs in-house or obtain these services from a contractor or other
            external source.
Practice 5: Compare the merits of different delivery mechanisms (such as classroom or
            computer-based training) and determine what mix of mechanisms to use to
            ensure efficient and cost-effective delivery.
Practice 6: Track the cost and delivery of its training and development programs agency-
            wide.
Practice 7: Evaluate the benefits achieved through training and development programs,
            including improvements in individual and agency performance:
            (a) Has a formal process for evaluating employee satisfaction with training.
            (b) Has a formal process for evaluating improvement in employee
            performance after training.
            (c) Has a formal process for evaluating the impact of training on the agency’s
            performance goals and mission.
Practice 8: Compare training investments, methods, or outcomes with those of other
            organizations to identify innovative approaches or lessons learned.

Source: GAO analysis based on prior GAO reports, other related expert studies, and federal training requirements.




4
 Appendix II includes a table that lists the laws, regulations, and Executive Orders that are
related to the leading practices.




Page 3                                                                   GAO-12-878 Federal Training Investments
To obtain government-wide information on agency training investment
practices, we obtained high-level information from a Council made up of
CHCOs representing 27 agencies (known as the CHCO Council) through
a questionnaire on their training investment practices and processes. 5 To
obtain additional perspective and insights on the training investment
practices identified in the questionnaire, we discussed the responses with
the CHCO Council and a Council made up of the Chief Learning Officers
(CLO) from the 27 agencies (known as the CLO Council). We describe
these Councils in greater detail in the background of this report. Based on
the responses to the questionnaire and workforce size, we selected four
agencies—the Department of Homeland Security (DHS), Department of
Veterans Affairs (VA), Department of the Interior (DOI), 6 and Department
of Energy (DOE)—from which to obtain illustrative examples of how they
implemented the training investment practices identified in the
questionnaire. As part of our review of agency practices, we also obtained
information on the steps that agencies are taking to identify and prioritize
investment allocations for training required to implement the GPRA
Modernization Act of 2010 (GPRAMA). 7

To identify and assess OPM’s guidance and assistance to agencies on
training investment strategies, we reviewed OPM training guidance, and
relevant documentation on forums, workshops or other assistance and
activities. In addition, we interviewed officials from the OPM offices with
primary responsibility for providing training policy guidance and technical
assistance to agencies. We compared this information to the leading
practices identified in table 1. We also identified and described the steps



5
 We sent a questionnaire to all 27 members of the CHCO Council and received responses
from all 27 members of the Council. For a full list of Council members, see the full
discussion of scope and methodology in Appendix I.
6
 In a separate review, we are also assessing the Department of the Interior’s human
capital program, including training requirements for key oil and gas oversight positions and
to what extent the department has developed and implemented plans to address ongoing
training challenges.
7
 GPRAMA mandated the Director of OPM to: (1) within a year of enactment, identify, in
consultation with the Performance Improvement Council, the key skills and competencies
needed by federal government personnel for developing goals, evaluating programs, and
analyzing and using performance information for the purpose of improving government
efficiency and effectiveness; (2) within 2 years of enactment, incorporate, as appropriate,
key skills and competencies into relevant position classifications; and (3) within 2 years of
enactment, work with each agency to incorporate the key skills identified into training for
relevant employees at each agency Pub.L. No. 111-352, 124 Stat. 3866 (Jan. 4, 2011)




Page 4                                             GAO-12-878 Federal Training Investments
             that OPM has taken to identify the skills and training needed to implement
             performance management improvements, such as those required by
             GPRAMA, as a foundation for future agency training investments.
             However, we did not assess the effectiveness of OPM’s efforts to identify
             GPRAMA-related skills and actions to develop related training.

             Based on information obtained from agencies and OPM, we assessed
             which leading training investment practices were being implemented by
             agencies and addressed by OPM guidance and assistance. We also
             identified the challenges or limitations reported by agencies to
             implementing the practices, and opportunities for improvement in agency
             processes and related OPM guidance.

             We conducted this performance audit from December 2011 to September
             2012, in accordance with generally accepted government auditing
             standards. Those standards require that we plan and perform the audit to
             obtain sufficient, appropriate evidence to provide a reasonable basis for
             our findings and conclusions based on our audit objectives. We believe
             that the evidence obtained provides a reasonable basis for our findings
             and conclusions based on our audit objectives. Further details of our
             objectives, scope, and methodology are provided in appendix I.


             The federal government has established a policy to develop its
Background   employees through training programs, to improve public service, increase
             efficiency and economy, and build and retain a force of skilled and
             efficient employees, among other things. In 1967, President Johnson
             signed Executive Order No. 11348, to provide agency heads and OPM
             with presidential direction on how training is to be carried out. 8 Under
             Executive Order No. 11348, OPM is responsible for planning and
             promoting the development, improvement, coordination, and evaluation of
             training in accordance with chapter 41 of title 5 of the U.S. Code and the
             established policy. Chapter 41 of title 5 sets forth the statutory framework
             for federal government training and development. The executive order
             further requires OPM to identify functional areas in which new or
             expanded interagency training activity is needed and either conduct such
             training or arrange for agencies having the substantive competence to do


             8
              Exec. Order No. 11348, “Further Training of Government Employees”, 32 Fed.Reg. 6335
             (April 20, 1967), as amended by Exec. Order No. 12107, 44 Fed. Reg.1055
              (Dec 28, 1978).




             Page 5                                        GAO-12-878 Federal Training Investments
so; as well as to coordinate interagency training conducted by and for
agencies. It also requires OPM to assist agencies in developing sound
programs and financial plans for training and provide advice, information,
and assistance to agencies on planning, programming, budgeting,
operating, and evaluating training programs. In addition to these activities,
OPM provides advice and assistance to agencies on training and
development programs. OPM’s Training and Executive Development
(TED) group (a subcomponent within the Office of Executive Resources)
is the primary office that provides policy direction and leadership to
agencies in developing plans and strategies to implement training and
development programs. It also provides agency guidance to ensure the
government’s training and development programs support strategic
human capital investments. The TED group provides assistance through
two main mechanisms: guidance documents and technical assistance.
OPM has developed five guides that agencies can use as references for
different aspects of making or reporting training investment decisions in
the planning, designing, implementation, and evaluation phases of their
training and development programs (see table 2).

Table 2: OPM’s Guidance Documents and Description

Guidance document           Description
Guide to Human              This guide assists agencies with preparing and submitting
Resources Reportinga        human resources, payroll, and training data files to OPM’s
(March 2012)                Enterprise Human Resources Integration (EHRI) data
                            warehouse.b Agencies are required to provide updates of
                            their training information to OPM’s electronic data collection
                                                              c
                            system, including training costs.
Training Evaluation Field   This guide provides information on methodologies for
Guide (January 2011)d       conducting training evaluations with the goal of identifying
                            cost effective training initiatives that maximize mission
                            accomplishments. OPM developed this guide based, in
                            part, on the statutory requirements related to agency
                            evaluation of training programs under 5 U.S.C . § 4103(c)
                            as implemented by OPM regulations 5 CFR § 410.202.
Draft Training Policy       This guidee provides guidance to agencies on implementing
Handbook                    training activities required by laws and regulations on
(December 2011)             training and development in the federal government.
Guide for Collection and    This document provides guidance to assist agencies in
Management of Training      complying with the legal and regulatory instructions for
Information                 collecting and managing federal training information. Under
(August 2008)               5 CFR § 410.601 agencies are required to maintain and
                            report to OPM government-wide Electronic Data Collection
                            System training data on its employees’ training and
                            development activities.




Page 6                                          GAO-12-878 Federal Training Investments
    Guidance document                   Description
    Guide to Strategically              This document provides guidance to agencies in planning
    Planning Training and               the allocation and use of training resources. Among many
    Measuring Resultsf                  other things, it provides ideas and tools to help agencies
    (July 2000)                         identify potential training investments that will assist them in
                                        achieving their goals and accomplishing their mission, and
                                        provides approaches and models to assist agencies in
                                        evaluating training’s contribution to the organization.
Source: GAO analysis of OPM guidance documents.
a
 OPM, Guide to Human Resources Reporting, accessed August 6, 2012,
http://main.opm.gov/feddata/ghrr/index.asp.
b
 The OPM’s EHRI program’s data warehouse is the government’s premier source for integrated
federal workforce information. The system currently collects, integrates, and publishes data for 2
million executive branch employees on a biweekly basis, supporting agency and government-wide
analytics.
c
    5 C.F.R. § 410.601.
d
 This draft is an updated version of OPM’s Training Policy Handbook: Authorities and Guidelines
published in May 11, 2007.
e
 OPM is currently updating the Training Policy Handbook to align it with the updated regulations on
training (5 C.F.R. part 410) and management development (5 C.F.R. part 412). We obtained a draft of
the updated Training Policy Handbook.
f
 In July 2000, OPM developed this guide in response to Exec.Order No.13111, Using Technology to
Improve Training Opportunities for Federal Government Employees, 64 Fed. Reg. 2793
(Jan. 12, 1999).


The TED group also provides technical assistance on agency training
investments through facilitating discussions and forums, and providing
training to agencies’ human resources (HR) staff. For example, the TED
group uses various web-based mechanisms—such as OPM’s website,
OPM LISTSERV, OPM Federal Training and Development web site and
OPM Federal Training and Development Wiki—to facilitate discussions
between agencies on training investments and to share guidance with
agencies. In addition to these facilitated discussions and forums, the TED
group provides training to federal HR professionals in various areas,
including activities that support making training investment decisions. For
example, OPM provides training to HR staff through its partnership with
the CHCO Council to operate HR University, which OPM officials and the
HR University website report is the federal government’s single “one stop”
training resource center for the HR professional throughout the federal
government. HR University is an effort that is intended to achieve
government-wide savings through pooling and sharing training resources
and identifying the best HR training across government. 9



9
    HR University was established in 2010, in partnership with CHCO Council.




Page 7                                                       GAO-12-878 Federal Training Investments
Agencies have the primary responsibility for establishing, operating,
maintaining, and evaluating their training programs in support of
achieving their mission and goals. OPM regulations specify that agency
employee developmental plans and programs should be designed to build
or support an agency workforce capable of achieving agency mission and
performance goals and facilitating continuous improvement of employee
and organizational performance. 10 Furthermore, Executive Order No.
11348 11 states that agency heads must undertake several activities in
support of developing employees, including:

•     Review periodically, but not less often than annually, the agency’s
      program to identify training needed to bring about more effective
      performance at the least possible cost;

•     Conduct periodic reviews of individual employee’s training needs as
      related to program objectives;

•     Conduct research related to training objectives required for program
      improvement and effectiveness;

•     Plan, program, and evaluate training for both short and long-range
      program needs by occupations, organizations, or other appropriate
      groups;

•     Establish priorities for needed training, and provide for the use of
      funds and man-hours in accordance with these priorities;

•     Establish training facilities and services as needed;

•     Extend agency training programs to employees of other agencies and
      assign his employees to interagency training whenever this will result
      in better training, improved service, or savings to the government.

The CHCO Council, established under the Chief Human Capital Officers
Act of 2002, provides assistance to OPM and agencies in accomplishing
federal human capital goals. 12 The 25-member CHCO Council is


10
    5 C.F.R. § 410.201.
11
    Exec. Order No. 11348, §§ 302 and 303
12
  Pub. L. No. 107-296, Title XIII, 116 Stat. 2287, 2288-89, § 1303 (Nov. 25, 2002) at 5
U.S.C. § 1401 note.




Page 8                                            GAO-12-878 Federal Training Investments
composed of the Director of the OPM, who serves as chairman; the
Deputy Director for Management of the Office of Management and
Budget (OMB), who acts as vice chairman; the CHCOs of the 15
executive departments; and the CHCOs of 8 additional agencies
designated by the OPM Director. 13 Additionally, the CHCO Council has an
Executive Director from OPM who coordinates and oversees the activities
of the council. The CHCO Council supports OPM in leading federal
agencies in the strategic management of human capital, providing a
forum for senior management officials to exchange HR best practices,
and informing the dialogue on civil service reform in order to build and
maintain an outstanding Federal workforce for the Nation. According to
the CHCO Council’s charter, among other purposes, the council is to:

•    advocate and assure a culture of continuous learning and high
     performance, developing and implementing effective strategies to
     attract, develop, manage, and retain employees with superior abilities;

•    identify human capital best practices and benchmarks, and apply
     those exemplars to their agencies and the federal government as a
     whole; and

•    provide leadership in identifying and addressing the needs of the
     federal government’s human capital community, including training and
     development.

To help CHCOs implement their training goals, many of the 24 Chief
Financial Officers Act agencies 14 and smaller agencies established Chief
Learning Officers (CLOs). These officers subsequently formed an
informal Chief Learning Officers (CLO) Council, which is a community of



13
   The CHCO Council was established with 25 members; however, the Director of OPM
may designate other members to the council. Such additional members may include, but
are not limited to: (1) the CHCOs of other executive agencies and (2) members who are
designated on an ex officio basis and who may be invited to contribute to projects, as
particular skills and expertise are needed. The Council currently includes a representative
for Small Agency Council (Corporation for National and Community Service) and the
Office of the Director or National Intelligence (ODNI), as designated by the OPM Director.
Although the Office of the Director or National Intelligence is not generally covered under
title 5 of the U.S. Code, according to OPM, they were still designated to join the Council
and follow many of the same training practices employed by other agencies.
14
  These 24 agencies are subject to the Chief Financial Officers Act. See listing of
agencies at 31 U.S.C. § 901.




Page 9                                            GAO-12-878 Federal Training Investments
                         practice composed of federal CLOs or their equivalents who meet
                         periodically to share best practices and create learning opportunities for
                         agencies and organizations. 15 The purpose of the CLO Council is to
                         provide a regular forum for CLOs to discuss and collaborate on high-level
                         agency strategic and operational issues affecting the Federal learning
                         and workforce development community of the federal government. These
                         two Councils, in partnership with OPM are to play a key role in assisting
                         agencies in the implementation of federal training and development
                         efforts.


                         Many CHCOs reported that they are implementing leading practices we
CHCOs are                identified as being important to making strategic training and development
Implementing Leading     investment decisions, especially regarding the delivery of training. These
                         practices include determining the best mix of decentralized and
Practices Related to     centralized training, considering government-wide reforms when
Training Delivery, but   identifying their training needs, and measuring employee satisfaction with
Not Practices that       training, among other things. However, many CHCOs reported that they
                         are not implementing the leading practices that would allow them to make
Support Making More      more cost-effective training decisions, such as having an agency-wide
Cost-Effective           process for prioritizing training investments so that the most important
                         training needs are addressed first and comparing the merits of different
Training Investments     delivery mechanisms (e.g. classroom or computer-based training) to
                         determine what mix of mechanisms will be most efficient and cost-
                         effective. All of these practices are important to ensuring that training
                         investments will be both effective and efficient in equipping federal
                         employees to accomplish their agencies’ goals.




                         15
                            The CLO Council derives its authority from its members, not their respective agencies.
                         Its members, act in a collective capacity and agree to support and further the ends of the
                         CLO Council’s vision and mission. CLO Council members are the most senior CLOs or
                         equivalent of the federal agencies that make up the cabinet of the President and those
                         who hold similar positions in independent agencies, boards, commissions, and the
                         intelligence community.




                         Page 10                                           GAO-12-878 Federal Training Investments
CHCOs Reported              Many CHCOs reported that they are implementing six of the leading
Implementing Leading        practices that we identified as being important to making strategic training
Practices that Support      and development investment decisions, especially regarding the delivery
                            of training, as shown in Table 3. However, regarding the leading practice
Making Strategic Training   related to tracking training investments agency-wide, we found that even
Delivery Decisions          those who reported that they track training agency-wide did not do so
                            completely or reliably.

                            Table 3: The Leading Training Investment Practices CHCOs Reported Implementing

                                                                                                      Number of agencies with
                                                                                                          the response
                             Leading training investment practices                                     Yes     No    Sometimes
                             Practice 2:       Identify the most appropriate mix of                      27      0
                                               centralized and decentralized approaches
                                               for its training and development programs
                             Practice 3:       Consider government-wide reforms and                      23      4
                                               other targeted efforts when planning its
                                               training and development programs
                             Practice 4:       Have criteria for determining whether to                  15     12
                                               design training and development programs
                                               in-house or obtain these services from a
                                               contractor or other external source
                             Practice 6:       Track the cost and delivery of its training               16     11
                                               and development programs agency-wide
                             Practice 7:       Evaluate the benefits achieved through
                                               training and development programs,
                                               including improvements in individual and
                                               agency performance
                                               (a) Has a formal process for evaluating                   25      1              1
                                               employee satisfaction with training
                                               (b) Has a formal process for evaluating                   13     12              2
                                               improvement in employee performance after
                                               training
                             Practice 8:       Compare training investments, methods, or                 24      3
                                               outcomes with those of other organizations
                                               to identify innovative approaches or lessons
                                               learned.
                            Source: GAO summary of CHCO responses to a GAO questionnaire




                            Page 11                                                        GAO-12-878 Federal Training Investments
Identifying the Most             All CHCOs reported that their agencies have implemented this practice.
Appropriate Mix of Centralized   We have previously reported that, while neither approach fits every
and Decentralized Approaches     situation, agencies need to consciously think about the advantages and
for its Training and             disadvantages of using centralized and decentralized approaches,
Development Programs             particularly for the design of training and development programs. 16
                                 Centralizing design can enhance consistency of training content and offer
                                 potential cost savings. A decentralized approach to training design can
                                 enable agencies to tailor training programs to better meet local and
                                 organizational unit needs. Agencies with decentralized approaches often
                                 embed training representatives within their business lines and field
                                 structures to assist in coordination of training efforts, including design and
                                 development. Nineteen of the 27 agencies reported that they have both
                                 centralized and decentralized training processes, while eight reported
                                 having completely decentralized training processes. Most of these
                                 agencies reported that their CHCOs or CHCO staff typically make
                                 centralized training decisions, while the leadership within the
                                 components, subagencies or offices make mission-specific training
                                 decisions. In the questionnaire responses, CHCOs identified a range of
                                 officials who are involved in making training investment decisions at the
                                 corporate and sub-agency level, including CHCOs and their staff, chief
                                 management officers, chief executive officers, budget officers, chief
                                 information officers, and others. A number of agencies also reported that
                                 advisory or oversight boards or training universities within their agency
                                 are involved in making training investment decisions.

                                 In the four agencies that we selected for review to obtain illustrative
                                 examples of how they implemented the training investment practices, the
                                 CHCOs or their representatives reported that their agencies made a
                                 decision to have both centralized and decentralized processes because
                                 they believe that the components or sub-agencies are more
                                 knowledgeable about their mission-specific training needs, while the
                                 central human capital staff can add the most value by managing
                                 investment decisions for more general training across the department.
                                 VA—which was one of the four agencies that we selected—established a
                                 corporate university known as the Veterans Affairs Learning University
                                 (VALU) to provide training to all VA employees. VALU provides training
                                 primarily in general areas such as leadership, management and
                                 supervision, as well some career and technical training. VALU offers



                                 16
                                  GAO-04-546G.




                                 Page 12                                    GAO-12-878 Federal Training Investments
training to the administrations and staff offices through a request process
that is based on the training needs that the administrations and staff
offices identify. Those training needs are required to be aligned to VA
critical training areas. 17 An Enterprise Training Advisory Board,
established in April 2012, also advises the Dean of VALU on the impact of
training, potential training development, and methods of delivery. 18
However, another tier of training is also provided within VA’s three
administrations—the Veterans Health Administration (VHA), Veterans
Benefits Administration (VBA), and National Cemetery Administration.
Each administration independently makes training investment decisions
and provides training to its employees in mission-specific and some
general and mandatory areas. The leadership of each administration
makes decisions about the level and prioritization of these training
investments. For example, at the VHA the Associate Deputy Under
Secretary for Health (or equivalent) and, subsequently, the Deputy Under
Secretary for Health assess the training requested by their offices against
various criteria, including whether training requests are aligned with and
support VA and VHA strategic goals, objectives, strategies, initiatives,
and performance improvement goals. During each review these officials
prioritize the requests through a voting process, and forward selected
training to the next level. Ultimately, the training is sent for approval to the
Under Secretary for Health and the VA Chief of Staff. The other three
agencies that we met with also reported having both centralized and
decentralized processes for making mission-specific training investment
decisions. However, most often, decentralized training decisions were not
required to be vetted with department level leadership for these three
agencies.




17
   On June 5, 2012 an enhanced training requirements process was launched with an
emphasis on closing identified skill gaps based on training needs assessments.
According to VA officials, this process incorporated lessons learned, best practices, and
prioritization criteria to align training with transformational goals and the VA Strategic plan.
18
  According to the Enterprise Training Advisory Board charter, which was signed on April
10, 2012, the group is an enterprise training and development management advisory
board. It serves as a high-level executive review and recommendation group to advise the
Dean of VALU on business requirements and on the impact of planned policies and
procedures for leadership and mission-critical, non-clinical career technical training and
development. The Enterprise Training Advisory Board shall provide advice to the Dean of
VALU the impact of training and support communication regarding requested or
recommended training plans and projects. However, it does not have authority to approve
training investment decisions.




Page 13                                              GAO-12-878 Federal Training Investments
Considering Government-wide     Nearly all CHCOs in our review reported that they have a process for
Reforms and Other Targeted      these considerations. We have previously reported that when planning
Initiatives When Planning its   training and development efforts, agencies should look to the actions of
Training and Development        the administration, Congress, and internal and external auditors by
Programs                        considering administration priorities, legislative reforms, and major
                                management challenges that might shape agency priorities and strategies
                                for training and development. 19 As an administration focuses its efforts on
                                addressing its priorities, agencies can benefit by having mechanisms or
                                processes for considering whether and to what extent these initiatives
                                could be linked to employees’ skills and competencies and the related
                                training and development approaches that might be needed. Twenty-
                                three of the 27 CHCOs who responded to our questionnaire reported
                                having such a process in place. For example, 16 of the CHCOs reported
                                that they are already setting investment allocations or training priorities to
                                implement GPRAMA.

                                At DOE, another agency we selected for review to obtain illustrative
                                examples, officials reported that they have identified the training that the
                                department currently offers and will need to offer to implement GPRAMA.
                                The Secretary issued a memo to DOE employees on GRPAMA’s
                                implementation and is holding town hall meetings on improving
                                organizational performance. Another effort that DOE expects to support
                                the implementation of GPRAMA is the Goals-Engagement-Accountability-
                                Results (GEAR) model that OPM and OMB are helping to pilot in DOE
                                and four other federal agencies, which includes efforts to improve
                                employee performance, among other things. 20 According to DOE officials
                                and related documentation, DOE’s GEAR implementation plan includes
                                aligning employee performance management with organization
                                performance management and developing training to support these
                                goals, which along with initiating knowledge sharing activities, will
                                promote improvement of DOE’s organizational performance.




                                19
                                 GAO-04-546G.
                                20
                                  Beginning in late May 2011, a workgroup of the National Council on Federal Labor-
                                Management Relations (LMR) partnered with members of the CHCO Council to develop a
                                new model of employee performance management, referred to as GEAR. GEAR focuses
                                on articulating a high-performance culture, aligning employee performance engagement
                                with organizational performance management, implementing accountability at all levels,
                                and create a culture of engagement. OPM is piloting GEAR at five agencies—the Housing
                                and Urban Development, the DOE, the Coast Guard, OPM and VA.




                                Page 14                                        GAO-12-878 Federal Training Investments
Have Criteria for Determining    Many of the CHCOs in our review reported having criteria for this
Whether to Design Training and   purpose. Training can be provided by the agency itself, another
Development Programs in-         government agency, a school, a manufacturer, a professional association,
house or to Obtain These         or other competent persons or groups in or outside of government. To aid
Services from a Contractor or    in making these decisions, agencies should try to develop clear criteria for
Other External Source            determining when to contract for training and development services. We
                                 have previously stated that factors that agencies should consider in these
                                 decisions include the capability of in-house staff to develop and
                                 implement the training; the prior experience, capability, and stability of
                                 possible providers in the marketplace; and agency limitations on cost,
                                 time, and resources 21 Of the 27 CHCOs included in our questionnaire, 15
                                 reported that they have criteria for determining whether to design training
                                 and development programs in-house or obtain these services from a
                                 contractor or other external source.

                                 One agency that we selected for review to obtain illustrative examples
                                 was DOI, which reported implementing this practice, however, the extent
                                 to which this decision-making process is implemented agency-wide is
                                 unclear. In its questionnaire response, DOI’s CHCO reported that DOI’s
                                 Office of Strategic Employee and Organizational Development has
                                 responsibility for offering corporate training through DOI’s university. This
                                 office decides whether to “make or buy” departmentwide training. When
                                 we met with DOI officials in the course of our review, they explained that
                                 although almost all courses are delivered by vendors because DOI has
                                 no internal trainers, they do have a small cadre of instructional designers
                                 who can develop some e-Learning courses. Decisions on whether to
                                 develop the courses internally are based on various criteria, including
                                 whether a course can be developed quickly, does not require a significant
                                 amount of content development, and subject matter experts can be
                                 provided to support the course development. Although this department
                                 level process is useful, DOI officials did not know if the bureaus within the
                                 department consistently use a “make or buy” approach. They reported
                                 that the larger bureaus have some capacity for in-house development
                                 while the smaller bureaus do not have this capability.

Track the Cost and Delivery of   Many CHCOs reported that their agencies implement this practice,
its Training and Development     although most of the CHCOs who reported that they do not track training
Programs Agency-wide             investments agency-wide were leaders of the agencies with the largest



                                 21
                                  GAO-04-546G.




                                 Page 15                                   GAO-12-878 Federal Training Investments
workforces. 22 We have previously reported that to obtain a
comprehensive determination of the costs of these initiatives, agencies
need to find ways around barriers that prevent them from fully and
accurately identifying the expenses associated with all components of
their training and development processes. 23 These costs can include
expenses for instructional development; participant and instructor
attendance; facility, material, and equipment costs, and travel and per
diem expenses. To track the cost and delivery of training and
development programs, agencies need credible and reliable data from
learning management systems as well as accounting, financial, and
performance reporting systems. To the extent possible, agencies also
need to ensure data consistency across the organization (such as having
data elements that are pulled from various systems representing the
same type of information). Variation in the methods used to collect data
can greatly affect the analysis of uniform, quality data on the cost and
delivery of training and development programs. In response to our
questionnaire, 16 CHCOs reported that they track training investments
agency-wide.

In our review of agencies to obtain illustrative examples of practices, two
of the selected agencies, DOE and VA , were among the CHCOs who
reported in their questionnaire that they track training investments
agency-wide. However, our review found both these agencies may not do
so completely or reliably. Both agencies identified the same limitations
and challenges as the agencies whose CHCOs reported that they do not
have a process to track training agency-wide. For example, DOE and VA
officials reported experiencing challenges in tracking training costs
agency-wide because of their multiple learning management systems and
inconsistent practices in tracking training data. 24 Officials from DHS and
DOI, who reported in the questionnaire that they could not track training
investments agency-wide, described similar challenges to those
experienced by DOE and VA. Moreover, officials from all four agencies
were unaware of the total amount their agencies invest in federal training


22
  The eleven agencies that reported that they do no track training investments agency-
wide represented 65 percent of the federal workforce. This figure is based on CRS
calculations of the size of each agency’s workforce, as of December 2010.
23
 GAO-04-546G.
24
  A learning management system is a software application that automates the
administration, tracking, and reporting of training events.




Page 16                                          GAO-12-878 Federal Training Investments
and stated that they could not provide reliable training data to OPM,
which requests these data to address its government-wide training
responsibilities. Under OPM regulations, agencies are required to
maintain data on training activities and expenditures and submit these
data to OPM.

As an example of challenges tracking training investments, DHS reported
that it is unable to track or better leverage training investments across the
department because of the nine, major incompatible Learning
Management Systems that it uses to track training throughout the agency.
We highlighted these same challenges in a 2005 report on DHS training,
noting that the lack of common management information systems and the
absence of commonly understood training terminology across
components, among other things, may impede the agency’s ability to
achieve its training goals. 25 According to more recent documentation on
the limitations of DHS’ tracking systems, the components’ disparate
systems currently limit them from sharing useful training information
across the department, effectively aggregating training data agency-wide,
and reporting complete training investment information to OPM. As a
result, DHS is seeking to purchase a single learning management
system. 26

Even when agencies had a single training information system, the
components may not consistently use them to track training investments
because of inconsistent coding schemes for tracking similar training
activities. For example, even though DOI has a single system for tracking
training information, officials reported that their human capital office must
rely on employees or data stewards 27 to input their training data and
some cost data may not be included, such as training travel costs, or



25
  GAO, Department of Homeland Security-Strategic Management of Training Important
for Successful Transformation, GAO-05-888 (Washington, D.C.: Sept. 23, 2005).
26
  In addition to challenges tracking training data across components, a recent GAO report
highlighted that a DHS component, the Federal Emergency Management Agency (FEMA)
does not track how much it spends within its own component on the Disaster Assistance
Employee (DAE) training, which hinders FEMA’s ability to plan for future training. See
GAO, DISASTER ASSISTANCE WORKFORCE-FEMA Could Enhance Human Capital
Management and Training, GAO-12-538 (Washington, D.C.: May 25, 2012).
27
  Data Stewards are those individuals ultimately responsible for the definition,
management, control, integrity or maintenance of a departmental or Enterprise data
resource.




Page 17                                          GAO-12-878 Federal Training Investments
                                 certain types of training may not be entered, such as conferences. Such
                                 costs are sometimes paid directly by an employee’s immediate office
                                 using a government credit card and are not tracked as training. In
                                 addition, learning management system training data are often not
                                 reconciled with DOI’s financial expenditure data because, until recently,
                                 their financial systems have not captured education tuition and training
                                 fees, and are still unable to track training travel costs. Therefore DOI’s
                                 cost data are most likely incomplete. Similarly, officials from DOE, DHS,
                                 and VA reported that they are aware of some inconsistencies in whether
                                 some types of training, such as conferences, are entered into their
                                 learning management systems. They also stated that there are
                                 inconsistencies in how agency components capture and code workforce
                                 training into their system because they lack a common definition for what
                                 types of activities should be considered training or have varying coding
                                 schemes or tools for capturing the cost. For example, some organizations
                                 use a procurement request or obtain a contractor to deliver training and
                                 do not document these costs using the standard government form for
                                 tracking data or in their learning management systems. In other cases,
                                 training investment data are captured using different coding in various
                                 learning management systems and financial systems. Some officials
                                 report that reconciling these data would be difficult. For example, DOE’s
                                 chief financial officer reported that it takes a couple of months to gather
                                 training investment data from DOE’s various systems, partly because the
                                 systems have inconsistent coding for these data. DOE’ officials reported
                                 that changing financial codes to reconcile training data would be time
                                 consuming and expensive because their financial systems are 20 years
                                 old. However, after years of highlighting this challenge, they are seeking
                                 approval to make such changes. Officials from the four agencies
                                 generally reported that, as a result of all of these factors, there is no
                                 overarching awareness or oversight of how much is spent on training
                                 investments and for which activities .

Evaluate the Benefits Achieved   Nearly all CHCOs reported having a formal process to evaluate employee
Through Training and             satisfaction with training, but fewer had processes to evaluate the impact
Development Programs,            of training on employees or agency performance. We have previously
Including Improvements in        reported that it is increasingly important for agencies to be able to
Individual and Agency            evaluate their training and development programs and demonstrate how
Performance                      these efforts help develop employees and improve the agencies’
                                 performance because it can aid decision makers in managing scarce
                                 resources, and provide credible information on how training and




                                 Page 18                                  GAO-12-878 Federal Training Investments
development programs have affected organizational performance. 28 To
do so agencies need to develop evaluation processes that systematically
track the cost and delivery of training and development efforts and assess
the benefits of these efforts. Training and development programs can be
assessed by measuring (1) participant reaction to the training program,
(2) changes in employee behavior or performance; and (3) the impact of
the training on program or organizational results, which may include a
return on investment assessment that compares training costs to derived
benefits. 29 Some of these methods can help provide better value through
identifying areas for continuous improvement in training programs. We
consider the processes for conducting these evaluations to be formal
when they are systematically conducted throughout the agency, have
established guidelines and criteria that govern how they are implemented
and are documented. However, CHCOs may also have other criteria for
determining what is considered a formal process, based on their
agencies’ environment. We asked CHCOs about their formal processes
for conducting the three levels of evaluation listed earlier, which are the
common types of evaluations. Many CHCOs reported routinely
implementing the first two, but not the third (which we discuss later in this
report). Twenty-five of the 27 CHCOs included in our questionnaire
reported that they measure employee satisfaction; and a little more than
half reported that they measure improvement in employee performance.

Officials from the four agencies that we interviewed reported that they all
assess employee’s reaction to training and sometimes assess changes in
employee performance. For example, officials from DOE reported that
they evaluate all the training that they offer by surveying participants’
reactions to the training—which can include their feedback on the
effectiveness of the instructor, the topics, the presentation style, the
schedule, audiovisuals, and other subjects—and use this information to
make revisions to the program courses. Documents that we reviewed on
training evaluations identified updates or revisions made to course
materials and tests to improve their effectiveness, based on training


28
  GAO-04-546G.
29
  The Kirkpatrick model of evaluation is a popular model of evaluation that includes
measuring (1) participant reaction to the training program; (2) changes in employee skills,
knowledge, or abilities; (3) changes in employee behavior or performance; and (4) the
impact of the training on program or organizational results, which may be split into a fifth
level that includes a return on investment assessment that compares training costs to
derived benefits. For purposes of our review, we focused on three of these areas.




Page 19                                            GAO-12-878 Federal Training Investments
                                  feedback and policy updates. As an example of evaluating the impact of
                                  training on employee performance, DOI officials stated that, while they do
                                  not have an agency-wide process, some of their organizations—such as
                                  those within the Bureau of Land Management and National Park Service
                                  use an online evaluation tool to assess the impact of training courses on
                                  employees’ abilities to perform tasks, about 6 weeks after a course has
                                  been completed. According to the official, at this time, the process is not
                                  used department-wide, but the agency is looking into how it may be able
                                  to do so starting in fiscal year 2013. According to DOI’s CLO, establishing
                                  this link between training, employee competencies and mission critical
                                  occupation work is one that DOI is targeting for improvement.

Compare training investments,     Nearly all CHCOs reported that they implement this practice. We have
methods, or outcomes with         previously reported that there are many ways to help improve
those of other organizations to   performance, so it is important for agencies to continually look to others to
identify innovative approaches    identify innovative approaches that may relate to their training and
or lessons learned                development efforts. Within the context of that agency’s unique
                                  environment and situation, an agency can compare its investments,
                                  approaches, and outcomes with those of public and private organizations
                                  that are undertaking notably innovative and effective training and
                                  development efforts. Agencies can uncover weaknesses in their training
                                  and development strategies that need improvement and identify new
                                  ideas, mechanisms, and metrics that they could employ. 30 Twenty-four of
                                  the 27 CHCOs included in our questionnaire reported that they compare
                                  training investments, methods, or outcomes with those of other
                                  organizations to identify innovative approaches or lessons learned.

                                  Officials from the two agencies we asked to provide examples of this
                                  practice described this process as occurring informally through
                                  interactions with other CLO Council members. For example, DHS officials
                                  that we met with reported that they meet with other agencies to share
                                  best practices and recommend vendors during breaks or after the CLO
                                  meetings. The officials said that examples of sharing ideas on new
                                  training programs included recent discussions by OPM and agencies on
                                  the GEAR pilot program lessons learned and new courses for developing
                                  supervisors. While two agencies reported having informal interactions
                                  with other agencies to share and compare training information, none of
                                  the agencies that we met with described efforts to benchmark their



                                  30
                                   GAO-04-546G.




                                  Page 20                                   GAO-12-878 Federal Training Investments
                                 practices with agencies or other relevant entities. We have previously
                                 reported that benchmarking can help agencies determine who is the very
                                 best, who sets the standard, and what that standard is. 31


Many CHCOs are Not               Many CHCOs reported that they are not implementing the leading
Implementing the Leading         practices that would allow them to make more cost-effective training
Practices that Support           decisions, as shown in Table 4.
More Cost-Effective
Training Investment              Table 4: Leading Training Investment Practices that Many CHCOs Reported Not
                                 Implementing Routinely
Decisions
                                                                                                               Number of CHCOs with
                                                                                                                  the response
                                     Leading training investment practices                                      Yes   No    Sometimes
                                     Practice (1a): Identify the appropriate level of investment to              11    16
                                     provide for training and development efforts
                                     Practice (1b): Prioritize funding so that the most important                12    15
                                     training needs are addressed first
                                     Practice 5: Compare the merits of different training delivery               11    16
                                     mechanisms and determine what mix to use to ensure
                                     efficient and cost-effective delivery
                                     Practice 7(c): Have a formal process for evaluating the impact               8     8            11
                                                                                                a
                                     of training on the agency’s performance goals and mission
                                 Source: GAO analysis of CHCO responses to a GAO questionnaire.
                                 a
                                 One agency that reported “I don’t know” was included in the category “No”.


Identify the Appropriate Level   Many CHCOs included in our review reported that they have not
of Investment to Provide for     implemented this practice. We have previously stated that, to determine
Training and Development         the best ways to leverage investments and establish priorities, agencies
Efforts and Prioritize Funding   can develop an annual training plan that targets developmental areas of
So That the Most Important       greatest need and that outlines the most cost-effective training
Training Needs are Addressed     approaches to address those needs. 32 When assessing investment
First                            opportunities for its training plan, the agency ought to consider the
                                 competing demands confronting the agency, the limited resources
                                 available, and how those demands can best be met with available
                                 resources. If training is identified as a solution to improve agency



                                 31
                                      GAO-04-546G.
                                 32
                                      GAO-04-546G.




                                 Page 21                                                          GAO-12-878 Federal Training Investments
performance, agencies can prioritize training using criteria, such as
expected demand for the investment from internal sources, availability of
resources to support the effort, potential for increased revenue, and risk
of unfavorable consequences if investments are not made. Given current
budget constraints, agencies may also want to prioritize training that has
the potential to improve their efficiency. Developing a business case for
training and development that includes this information sets forth the
expected costs and benefits of the investments and provides decision
makers with essential information they need to allocate necessary
resources. Furthermore, under Executive Order No.11348 and OPM
regulations, agencies are to establish training priorities, although
agencies are not specifically instructed to establish an agency-wide
process to do so. 33 Of the 27 CHCOs questionnaire responses, 16
CHCOs reported that they do not set a level of investment agency-wide
and 15 CHCOs reported that they do not prioritize training agency-wide.

In our meetings with officials from the DOE, DOI, DHS, and VA as well as
the CLO council, agency officials cited several reasons for why they do
not establish a level of training investment agency-wide or prioritize
training agency-wide. Some of the reasons were described as purposeful
decisions not to do so and other reasons were described as limitations in
their ability to do so. First, CHCOs elect to establish and prioritize training
investments for centralized training and are often not involved in the
investment decisions made for specific training within the components or
offices, as we previously described. In addition, large components or sub-
agencies often have autonomy over their training budgets because the
budgets are appropriated directly to them from Congress. As a result,
CHCOs and their staff are often unaware of how much these components
spend for training and do not have input into these decisions. Component
and sub-agency heads often act autonomously and are not required to
communicate with the CHCO about these decisions. Further, because of
limitations in internal tracking systems for training (which we discussed
earlier in this report), CHCOs do not have information on all of the training
that is completed in their agency and the related costs.

Officials from various agencies involved in the CLO Council and three of
the four agencies that we individually met with reported having a lack of


33
  Section 303(e) of Exec. Order No.11348 and 5 C.F.R. § 410.201(c) requires agency
heads to establish priorities for training and allocate resources according to those
priorities.




Page 22                                         GAO-12-878 Federal Training Investments
visibility into the prioritization and level of training investments throughout
their agencies, which they reported limits their ability to better leverage
and reduce duplication in training investments their agencies. Officials in
the agencies that we met with reported that, although they believe that
their components or organizational elements are more capable of making
training decisions related to their specific missions, the lack of
coordination and communication on training investments and priorities
has led to some duplicative and ineffective training investments in their
departments. For example, senior human capital officials in DOI reported
that the department’s leadership, including the CHCO are not aware of
the department’s overall training investments agency-wide and have no
formalized mechanism for ensuring accountability for how the funds are
used. They are aware that bureaus are buying duplicative training or
offering similar training classes that are of varying effectiveness—which is
resulting in inefficient training investments. For example, one bureau
recently independently contracted with an external provider for mid-level
manager leadership training that was already offered at DOI’s university
and paid $50,000 more than DOI University charges. According to
officials, this is a common problem. In addition to duplicative training
courses, in some cases, bureaus are duplicating the creation of new
training facilities. For example, a regional director of an DOI bureau built a
training classroom with a computer lab, despite having access to existing
computer labs within the complex where he worked and also at DOI
facilities a few miles way. Further, according to the officials, because it is
common practice for each bureau to independently secure training, there
is no consistency, little quality control, and no maximization of
procurement tools (such as blanket purchase agreements) across DOI. 34
In order to address these challenges, DOI has formed a one-time
departmentwide task force known as the Department Innovation and
Efficiency Team for Training. This task force was expected to identify:
potential duplication in training, funds expended in training delivery, and
the cost of travel and facilities, among other things. In July 2012, the
committee made recommendations to the CLO on opportunities to
generate efficiencies and savings in training operations. DOI’s Office of




34
  A blanket purchase agreement is a simplified acquisition method that government
agencies use to fill anticipated repetitive needs for supplies or services. Essentially,
blanket purchase agreements are like “charge accounts” set up with trusted suppliers.
According to the General Services Administration, BPAs offer an excellent option for
federal agencies, providing convenience, efficiency, and reduced costs.




Page 23                                            GAO-12-878 Federal Training Investments
                               Strategic Employee and Organization Development is developing action
                               plans to address the committee’s recommendations.

                               Officials from DHS also reported experiencing similar challenges with
                               duplicative or ineffective training investments in their agencies. Some of
                               these challenges are long standing. For example, seven years ago we
                               reported that DHS’s two-tier training process (component and
                               departmentwide) and lack of communication throughout the department
                               on the availability of some training programs and resources were
                               challenges that could impede its ability to achieve departmental training
                               goals and efficiencies. 35 DHS is still taking steps to address this on-going
                               challenge. In June 2005, DHS formally chartered a Training Leaders
                               Council (TLC) and recently revised its charter in June 2011. The TLC is
                               made up of senior training leaders from each component, and
                               representatives from headquarters to serves as an advisory and
                               collaborative community of practice to promote effective and efficient
                               training, education, and professional development opportunities to DHS
                               employees. According to DHS’ Human Capital leaders, while this group
                               does not set or prioritize training investments, it provides a forum for
                               exchanging useful information about common challenges and training
                               practices, which helps in making more efficient use of existing agency
                               resources. DHS also established the Human Resource Information
                               Technology Executive Steering Committee, made up of management
                               chiefs and HR and information technology leadership across DHS in
                               2010, and included TLC leadership as members in July 2011. This group
                               makes some funding decisions related to some training investments, such
                               as their recent decision to fund the purchase of a single learning
                               management system for the entire department. However, according to
                               DHS officials, because DHS has multiple congressional committees and
                               subcommittees from which the components receive funding and training
                               direction, coordinating training investments remains challenging.

Compare the Merits of          Many CHCOs that responded to our questionnaire reported that they do
Different Training Delivery    not compare the merits of different training delivery mechanisms. Our
Mechanisms and Determine       past research and that of others has shown that agencies should
What Mix to Use to Ensure      deliberatively consider the options for delivering training and consider
Efficient and Cost-Effective   essential issues, such as the goals and objectives for the training, the
Delivery                       type of audience intended for the training, the nature of the training



                               35
                                See GAO-05-888.




                               Page 24                                   GAO-12-878 Federal Training Investments
content, the availability of technology and resources, and the timing for
delivering the training. 36 Agencies can use a variety of instructional
approaches to achieve learning—in the classroom, through distance
learning, or in the workplace. When warranted, agencies should also
consider blended learning that combines different teaching methods (e.g.
Web-based and instructor-led) within the same training effort and provide
trainees with the flexibilities to choose among different training delivery
methods while leveraging resources in the most efficient way possible.
When assessing delivery options, agencies can try to achieve economies
of scale and avoid duplication of effort by taking advantage of existing
course content or training, such as sharable on-line courses or
multiagency training programs. However, In the responses to our
questionnaire, 16 of the 27 CHCOs reported that they do not compare the
merits of the different training delivery mechanisms in their agency.

In our meetings with DHS and VA to obtain illustrative examples, DHS
officials reported that their current learning management systems do not
allow them to mine information on the different delivery mechanisms used
throughout the department and to assess and compare their
effectiveness. According to the officials, they could obtain this information
manually, but it would be a very labor intensive process. Therefore, it is
not done. In contrast, VA officials informed us that they are assessing
different delivery mechanisms for training and conferences offered by
VALU because they recognize that opportunities exist to offer more
efficient mechanisms (such as e-learning). Moreover, VHA, which has the
largest workforce in the department, builds into its initial investment
decision-making process considerations of which delivery methods will be
most effective and efficient, and subsequently evaluates employee
satisfaction with the various delivery methods to inform future investment
decisions. Without processes such as these, agencies that do not
compare the merits of different training delivery mechanisms have limited
information for determining what mix of methods provides the most
efficient and effective delivery of federal training.




36
  See GAO-04-546G and Corporate Leadership Council Learning and Development: LDR
a Guide to Selecting the Most Suitable Development Method (Arlington, VA: Corporate
Executive Board, 2011).




Page 25                                       GAO-12-878 Federal Training Investments
Have a Formal Process for   Most CHCOs reported that their agencies do not have a routine formal
Evaluating the Impact of    process to implement this practice. As we previously mentioned, it is
Training on the Agency’s    increasingly important for agencies to be able to evaluate their training
Performance Goals and       and development programs and demonstrate how these efforts help to
Mission                     improve the agencies’ performance, and to assist them in making more
                            effective decisions about how to allocate scarce resources. Agencies are
                            required by statute and OPM implementing regulations to evaluate how
                            well training programs contribute to mission accomplishment and meet
                            organizational performance goals. 37 We have identified having a formal
                            process for this evaluation as a leading practice. However, there are
                            some understandable limitations to regularly and formally implementing
                            this practice. For example, some agency officials that we met with
                            reported that the cost and time required can be significant for obtaining
                            results of evaluations of training that measure the impact on agency
                            performance goals. As a result, they can only conduct this level of review
                            for training that they identify as highly important to key areas of their
                            mission. We have previously reported that not all training and
                            development programs require, or are suitable for, higher levels of
                            evaluation. 38For example, it may be ineffective to try to measure the
                            impact of training in an area that is still undergoing other significant
                            changes that could affect relevant performance goals, such as changes in
                            related policy and management structure. We recognize that higher levels
                            of evaluation (such as evaluating the impact on organizational
                            performance or return on investment) can be challenging to conduct
                            because of the difficulty and costs associated with data collection and the
                            complexity in directly linking training and development programs to
                            improved individual and organizational performance. Factors to consider
                            when deciding the appropriate level of evaluation include estimated costs
                            of the training effort, size of the training audience, management interest,
                            program visibility, and the anticipated “life span” of the effort. Each
                            agency will need to consider the feasibility and cost-effectiveness of
                            conducting these in-depth evaluations, along with budgetary and staffing
                            circumstances that may limit the agency’s ability to complete such
                            evaluations. Given the current budget constraints that agencies face,
                            making thoughtful tradeoffs regarding how to target costly evaluation



                            37
                              5 U.S.C. § 4103(c) requires such evaluation on a regular basis whereas OPM’s
                            implementing regulations specify that such evaluations are to be conducted annually. 5
                            C.F.R. 410.202.
                            38
                             GAO-04-546G.




                            Page 26                                          GAO-12-878 Federal Training Investments
reviews is a sensible approach. While it is important to prioritize reviews
of training, 8 of the 27 CHCOs that responded to our questionnaire
reported that they do not have a formal process for evaluating the impact
of their training on their agency’s performance.

For example, the CHCO at DOE reported in our questionnaire that DOE
does not implement this practice. We met with the CLO from DOE who
informed us that DOE does not have a formal process for implementing
this practice because the agency does not have a systematic
documented approach for conducting this level of review. Moreover,
evaluation data are not collected in a way that allows it to be aggregated
into a comprehensive assessment of its impact on the agency’s overall
mission. For example, different organizations within DOE conduct reviews
to assess the impact of training on their goals, but they are not captured
in an automated system and the methodologies that DOE organizations
use to conduct these reviews vary. As an illustration, DOE organizations
that work with nuclear material evaluate the technical training that they
provide to their employees against required certification and mission
goals. However, the organizations conduct these evaluations differently,
and because of these varied methodologies and lack of automated results
data, it is difficult to aggregate the reviews into an assessment of how
training has affected DOE’s overall training and mission goals. Similarly,
the CLO’s office evaluates cross-cutting training for employee satisfaction
and employee performance at DOE, but does not effectively or
consistently evaluate its impact on agency goals. According to the CLO,
to assist them in developing a more systematic formal process, they are
participating in OPM training on developing training evaluations and in the
GEAR pilot program—which is intended to better link employee
performance to organizational goals.

In contrast, VA’s training review processes illustrate that agencies that
have a formal process for assessing the impact of training on their
performance mission and goals can use it to make better training
investment decisions. VA recently assessed the return on investment of
its corporate training and, the department’s Administrations recently
evaluated the impact of mission-specific training on their performance
goals. In January 2012, VA evaluated the monetary and mission-related
benefits of training that was implemented under its Human Capital




Page 27                                   GAO-12-878 Federal Training Investments
Investment Plan. 39 According to the return on investment assessment and
report developed by VALU and VHA’s National Center for Organization
Development, VA’ s two-year $577 million investment in training and
development under VA’s Human Capital Investment Plan has resulted in
$604 million dollars in savings that are tied to reductions in costly VA
turnovers, fewer overdue accounts renewable, and fewer equal
employment opportunity complaints. 40 The report also states that VA has
gained non-financial returns, such as faster benefits processing,
increased veteran hiring programs, and improved patient satisfaction.
According to VALU officials, they have used details in this report along
with other factors to make decisions about future training and
development investments. Similarly, for its mission-specific training, VHA
recently conducted an in-depth review of training provided to Patient
Aligned Care Teams to improve their collaborative delivery of care to
patients. The evaluation assessed the training participants’ satisfaction,
skill acquisition, application on the job, and impact on VHA’s business.
The assessment ultimately determined that the training was successful in
addressing the desired behavior changes in the work place and that key
organizational results were influenced by the training, but it also identified
some improvements that VHA could make. 41




39
  According to documentation on VA’s Human Capital Investment Plan, the Plan is a
human resource management transformational initiative to develop the department’s
human capital into a proactive, forward looking, well trained and professional workforce
focused on delivery of high quality service to Veterans and their families. The plan
includes a focus on improving the recruitment, hiring and retention of VA employees;
investing in people development (e.g., leadership training); supporting and developing the
capabilities of VA’s Senior Executive Service (SES) employees; establishing a Veteran’s
Employment initiative to support veteran’s recruitment, retention and reintegration; and,
striving to build a broad set of human resources capabilities to support the delivery of
quality care and benefit services to Veterans and their families.
40
 Department of Veterans Affairs’ Human Capital Investment Plan (HCIP) Impact
Assessment /Year One: HCIP Monetization Summary, January 2012.
41
  We did not independently verify the reported cost savings or performance improvements
reported in the VALU or VHA studies.




Page 28                                           GAO-12-878 Federal Training Investments
                         OPM guidance and assistance to agencies on federal training
OPM Guidance and         investments are in line with five of the eight leading practices, but OPM
Assistance to            lacks guidance and assistance in some areas that are challenges to
                         agencies, as shown in table 5.
Agencies on Federal
Training Investments     Table 5: Comparison of Agency Implementation of Leading Practices and OPM
Addresses Many           Guidance and Assistance

Leading Practices that                                                                        Addressed Not Addressed
                                                                                                by OPM      by OPM
We Identified, But                                                                            guidance or guidance or
                          Leading Practice                                                    assistance  assistance
Could be Improved in      Practice 1: (a) Identify the appropriate level of                                        X
Areas That Are            investment to provide for training and development
                          efforts and (b) prioritize funding so that the most
Challenges to             important training needs are addressed firsta
                          Practice 2: Identify the most appropriate mix of                                         X
Agencies                  centralized and decentralized approaches for its
                          training and development programs
                          Practice 3: Consider government-wide reforms and                         X
                          other targeted initiatives to improve management and
                          performance when planning its training and
                          development programs
                          Practice 4: Have criteria for determining whether to                                     X
                          design training and development programs in-house or
                          obtain these services from a contractor or other
                          external source
                          Practice 5: Compare the merits of different delivery                     X
                          mechanisms (such as classroom or computer-based
                          training) and determine what mix of mechanisms to use
                          to ensure efficient and cost-effective delivery
                          Practice 6: Track the cost and delivery of its training                  X
                          and development programs
                          Practice 7: Evaluate the benefits achieved through                       X
                          training and development programs, including
                          improvements in individual and agency performance
                             (a) Has a formal process for evaluating employee
                             satisfaction with training
                             (b) Has a formal process for evaluating improvement
                             in employee performance after training
                             (c) Have a formal process for evaluating the impact
                             of training on the agency’s performance goals and
                             mission
                          Practice 8: Compare training investments, methods, or                    X
                          outcomes with those of other organizations to identify
                          innovative approaches or lessons learned
                         Source: GAO analysis of OPM guidance and assistance tools.




                         Page 29                                                      GAO-12-878 Federal Training Investments
OPM Provides Guidance or         OPM guidance or assistance to agencies on federal training investments
Assistance that Addresses        addresses five of the eight leading practices for federal agency training
Some Leading Training            investment decision-making processes. OPM’s five primary guidance
                                 documents that relate to making training investment decisions include the
Investment Practices             Guide to Human Resources Reporting, Training Evaluation Field Guide,
                                 Draft Training Policy Handbook, Guide for Collection and Management of
                                 Training Information, and Guide to Strategically Planning Training and
                                 Measuring Results. (See table 2 for a brief description of these guides). In
                                 addition, OPM provides technical assistance to agencies via facilitated
                                 forums, discussion, and training.

Consider Government-wide         Our review of OPM’s guidance documents and assistance shows that
Reforms and Other Targeted       OPM has provided some technical assistance to agencies on this
Initiatives to Improve           practice, although OPM does not have guidance documents that provide
Management and Performance       specific advice on this topic. For example, because of requirements in
When Planning its Training and   GPRAMA, OPM is providing assistance to agencies in considering this
Development Programs             government-wide reform when planning their training and development
                                 programs. GPRAMA required OPM to identify the competencies needed
                                 to perform the following three functions: developing goals, evaluating
                                 programs, and analyzing and using performance information for the
                                 purpose of improving government efficiency and effectiveness. OPM,
                                 working with subject matter experts developed a competency model for
                                 the three new roles required by GPRAMA—performance improvement
                                 officer, performance improvement staff, and goal leader. Earlier this year
                                 OPM advised agencies that it would provide guidance on how to
                                 incorporate the skills and competencies into these position descriptions,
                                 as specified in the GPRAMA. The Director of OPM stated that the agency
                                 would work with the CLOs to incorporate the key skills and competencies
                                 into agency training programs. 42

                                 OPM has begun providing this assistance to agencies by facilitating
                                 sessions for agencies to develop training requirements for implementing
                                 the new positions and roles required by GPRAMA. For example, OPM
                                 worked with OMB to gather information on existing training, provide
                                 learning opportunities, and consolidate new and existing training courses
                                 and materials to support this effort. Using this information, OPM and OMB
                                 led two working group meetings with agencies to discuss GPRAMA


                                 42
                                   OPM’s Director made these statements in a January 3, 2012 memo to agency CHCOs
                                 entitled “Government Performance and Results Act Modernization Act of 2010 Functional
                                 Competencies”.




                                 Page 30                                        GAO-12-878 Federal Training Investments
training needs and next steps. In a working group meeting in February
2012, OPM and agencies discussed which competencies identified in
GPRAMA could be improved readily through training. OPM provided
participants with a chart developed from an OPM and the Merit System
Protection Board 2011 Trainability Study on which competencies for the
three new roles required by GPRAMA were highly trainable versus those
that were less trainable. 43 After the discussion, OPM and participants
identified the most critical and manageable next steps, including:

•    create a common competency assessment tool to assess competency
     gaps within agencies;

•    create a course on writing results-oriented goals and standards—
     while also gathering existing training;

•    create a working group to assess the needs and create a solution to
     satisfy the training requirement for the Organizational Performance
     Analysis, Planning and Evaluating, and Performance Measurement
     competencies and to collect relevant case studies, as well as to
     identify opportunities to leverage agency resources;

•    identify existing subject matter experts in the agencies and create
     forums, workshops, training sessions, etc. where they can share their
     expertise and possibly engage in peer-to peer coaching;

•    create working groups where necessary; and

•    consider the development of a career path after OPM’s classification
     study.

In addition to this assistance, although not specific to government-wide
reforms, OPM’s Training Policy Handbook advises agencies to conduct a
training needs assessment that includes an evaluation of organization



43
  According to OPM documents, “Highly trainable” competencies consist primarily of
knowledge and can be improved readily through training. “Moderately trainable”
competencies have both a skills and ability component. The skills can be improved thru
training, but improvement in ability may be limited by the degree of the employees’ natural
talent. These competencies are only partially responsive to training. “Less Trainable”
competencies are identified as “trait” competencies that have personal characteristics that
resist change. These competencies make a difference on the job, but are frequently not
responsive to improvement through training.




Page 31                                           GAO-12-878 Federal Training Investments
                                  needs, which should take into consideration changing demographics,
                                  political trends, technology, and the economy.

Compare the merits of different   OPM’s TED group advises agencies in its Training Policy Handbook and
delivery mechanisms (such as      2000 Guide to Strategically Planning Training and Measuring Results to
classroom or computer-based       use multiple delivery methods, or combine them, when providing training
training) and determine what      to employees. For example, the Training Policy Handbook maintains that
mix of mechanisms to use to       agencies should decide which delivery option is best to achieve the
ensure efficient and cost-        instructional goals of the training, highlighting that some methods are
effective delivery                more effective for certain courses. It states that a performance
                                  management course may include role play scenarios which may not be
                                  suited for an e-learning course. Further, the guide states that agencies
                                  need to develop training delivery mechanisms that effectively limit
                                  unnecessary overlap and duplication of efforts. Similarly, we have
                                  previously reported that agencies need to consider essential issues such
                                  as the goals and objectives for the training, the type of audience for which
                                  the training is intended, the nature of the training content, the availability
                                  of technology and resources, and the timing for delivering the training
                                  when identifying the most effective and efficient delivery mechanism. 44
                                  Agency officials who have implemented this practice reported seeing
                                  positive results. For example, VHA officials that we met with and agencies
                                  that have publically discussed their efforts to assess the different delivery
                                  mechanisms at a March 2012 Partnership for Public Service Forum on:
                                  Going Virtual- Maximizing the Return On Investment of Online Training
                                  reported significant savings and increases in the effectiveness of their
                                  training by assessing and changing their training delivery mechanisms.
                                  Specifically, VHA officials reported achieving several non-financial and
                                  financial benefits as a result of moving from in-person meetings and audio
                                  and video conferencing to providing training on-line for one of its
                                  leadership training programs. According to a VHA assessment report, the
                                  benefits included: consistent curriculum across eight medical centers in
                                  three states; easier accessibility to course materials and job aids;
                                  immediate access to feedback on courses from learners; easier
                                  reproduction of courses for instructors; a return on investment of 140
                                  percent since implementation; and $116,000 saved in travel costs,
                                  facilitation, and facilities; among other things.




                                  44
                                   GAO-04-546G.




                                  Page 32                                    GAO-12-878 Federal Training Investments
                                 OPM guidance informs agencies that they should implement this practice.
                                 However, the guidance does not include methodologies for how to
                                 implement it. Officials from DHS—an agency that reported that it does not
                                 implement this practice—stated that tools provided by OPM could be
                                 strengthened to assist them in comparing training delivery mechanisms.
                                 For example, DHS officials reported that they have difficulty implementing
                                 this practice partly because their components do not track comparative
                                 data on the different delivery mechanisms. According to the DHS officials,
                                 the standard government form for tracking training data (Standard Form-
                                 182) does have a category for tracking training delivery type, but filling out
                                 this block is not mandatory and is often not used. 45 The DHS officials
                                 reported that an OPM requirement to capture these data would improve
                                 their ability to gather the information needed from DHS components to
                                 effectively implement this practice. As noted earlier, 15 of the 27 CHCOs
                                 included in our review, reported that they do not implement this practice,
                                 which indicates that they may also benefit from additional guidance and
                                 tools on ways to do so.

Track the cost and delivery of   OPM’s TED group provides guidance and assistance to agencies on
its training and development     tracking and reporting the cost and delivery of training and development
programs                         programs in four of its five guides. For example, OPM’s 2000 Guide to
                                 Strategically Planning Training and Measuring Results advises agencies
                                 to calculate the cost of the expenses associated with designing,
                                 developing, implementing, and evaluating their training programs and
                                 provides a list of the most common types of training costs. OPM’s Guide
                                 for Collection and Management of Training Information also outlines
                                 agency requirements to track various types of data training and provides
                                 a list of several data sources (e.g. Standard Form-182, agency personnel
                                 records, procurement documents, financial and performance records,
                                 training evaluation forms, etc.) that agencies could use to collect this
                                 information. Similarly, the Training Policy Handbook also incorporates
                                 guidance on tracking the cost and delivery of agencies’ training and
                                 development. In the more recent 2012 Guide to Human Resources
                                 Reporting, OPM outlines requirements for agencies to track training data
                                 and describes the requirement to use certain standard tracking forms,



                                 45
                                   The Standard Form-182 is the government-wide training form used to request, approve,
                                 and certify completion of training courses, conferences, seminars, symposia and
                                 academic courses. The primary purpose of this form is to document and track employee
                                 training, especially training paid with agency funds. The Standard Form-182 also requires
                                 that each training record should include all of the mandatory data.




                                 Page 33                                          GAO-12-878 Federal Training Investments
such as the standard “Authorization, Agreement, and Certification of
Training” (Standard Form-182) to track data. The guide also instructs
agencies to provide all training information included in this form for
submission to OPM’s Enterprise Human Resources Integration (EHRI)
database systems.

Although OPM provides several guidance documents and assistance on
tracking the cost and delivery of training, we found that this practice
continues to be a challenge for many agencies to implement. Agency
officials that we met with reported that they could benefit from additional
assistance from OPM in developing a common definition of what should
be tracked as training, developing policies to strengthen the utilization of
Standard Form -182 to document and report all training costs, and
encouraging agencies through guidance and technical assistance, to
routinely report training cost data to agency learning management
systems.

In addition, to providing guidance on tracking data, OPM facilitates the
collection of federal training data government-wide. Executive Order No.
11348 requires OPM to develop, install, and maintain a system to provide
the training data needed to carry out its own functions and to provide staff
assistance to the President. OPM’s EHRI is the government-wide
repository for these training data. 46 As noted above, agencies have been
required since 2006 to report training data to OPM monthly via this
system. However, according to OPM officials, they consider the data to
be unreliable because they are incomplete. Therefore, OPM officials have
not used it to inform their training guidance and assistance to agencies, to
counsel heads of agencies and other agency officials regarding federal
training needs or investments or to assist agencies in developing sound
programs and financial plans for training programs.



46
  OPM was also required by statute (5 U.S.C. § 4118) to prescribe regulations providing
for the maintenance of necessary information concerning the conduct of training activities
of agencies to enable the President and Congress to discharge effectively their respective
duties and responsibilities for supervision, control, and review of these training programs.
OPM implementing regulations require agencies to maintain records of training activities
and expenditures for submission to OPM. 5 C.F.R. § 410.601(a). As previously noted,
OPM’s EHRI Program’s Data Warehouse is the government’s premier source for
integrated Federal workforce information. The system is intended to collect, integrates,
and publishes data for 2.0 million Executive Branch employees on a bi-weekly basis,
supporting agency and government-wide analytics in various areas, including agency
training data.




Page 34                                            GAO-12-878 Federal Training Investments
According to OPM officials and documents, OPM should assess EHRI
training data for technical compliance and data quality validation.
Technical compliance is the testing and approving of agency systems for
data quality (i.e. correct formatting, adherence to edit rules). Once
systems are technically compliant, agencies are required to send monthly
data feeds of completed training events to OPM. 47 Once agencies are
reporting these data for all major components, all employees, all types of
training (e.g. conferences, on-line, classroom), and training cost data,
OPM are to evaluate the data quality to determine if it presents an
accurate picture of all training in the agency. However, OPM officials told
us that they have not assessed the quality of data or developed a report
on its reliability because no agency is sending information on all training
events. According to OPM officials, when agencies request assistance or
when OPM finds that an agency has been grossly delinquent in providing
data, OPM officials will inquire further and offer assistance to the
agencies. However, they typically do not document reliability issues or the
agreed upon action plans to address the problems. The officials agreed
that this is a problem, but stated that they would need more staff
resources to provide this level of assistance and oversight.

We believe that the current reliability of agency training investment data is
unknown because OPM officials have not internally assessed
improvements in the completeness of the data over the last 3 years or the
quality of the data in the six years that agencies have been required to
submit it. The two internal reviews that OPM conducted of training data
were in 2008 and September 2009. In the 2009 review, OPM reported
that there was an increase from fiscal year 2008 in the amount of training
data being reported by agencies, but that the quality of the data was still
less than what was necessary to provide an accurate picture of federal
training investments. According to the 2009 report, over half of all
agencies were reporting data for the entire agency, 86 percent were
reporting on a regular basis, but only 7 percent were reporting cost data.
The report identified several of the same reasons that we previously
described as limitations to agencies reporting training investment data.
Although the report stated that OPM would continue to work with
agencies to assess the quality/validity of training investment data and


47
  OPM has an EHRI technical office that collects the data from agencies and informs TED
officials about which agencies are providing data each month and whether there are
apparent problems with the data (e.g. missing or miscoded data). OPM requires agencies
to provide a single source of data that represents training throughout their organization.




Page 35                                          GAO-12-878 Federal Training Investments
determine whether agencies are reporting all training events, as noted
above, OPM officials informed us that they have not assessed the quality
of the data because the data are not 100 percent complete.

While it is important to have complete data, we do not believe that having
incomplete data necessarily prevents OPM from assessing the overall
reliability of the data, if it meets standards for sufficiency. In our guidance
on assessing the reliability of computer based data, we have stated that
agencies can assess data if it is sufficiently complete. 48 Data are
sufficiently reliable when testing and reviews of the existing information
provide assurance that (1) the likelihood of significant errors or
incompleteness is minimal and (2) the use of the data would not lead to
an incorrect or unintentional message. Further, we consider the data not
to be sufficiently reliable when there are significant errors or
incompleteness in some of or all the key data elements and if using the
data would probably lead to an incorrect or unintentional message.
Because OPM has not conducted an assessment of improvements in
agency training data in three years, it is unknown whether it is currently
complete enough to test other aspects of its quality and reliability.
According to the officials, although they have not conducted a formal
review of the data, they are able to visually look at the EHRI data base
and tell that the data are significantly more complete than in past years.

OPM also previously identified several steps that its officials would take to
assist agencies in improving their data, but have not yet implemented all
of them. According to OPM’s 2009 report assessing EHRI data, OPM
planned to assist agencies in improving training investment data, by: (1)
working with agencies to fully report all training investment data—
including costs; (2) working with agencies to decrease errors in reporting;
and (3) providing individual agencies with summary reports of the data
that they submitted to OPM for their review and verification. 49 We found



48
  Data validity refers to whether the data actually represent what we think is being
measured. Data reliability refers to the accuracy and completeness of computer-
processed data, given the uses they are intended for. Computer-processed data may be
data (1) entered into a computer system or (2) resulting from computer processing. GAO,
Assessing the Reliability of Computer-Processed Data, GAO-09-365G (Washington, D.C.:
February 2009).
49
  According to the 2009 analysis, the agency reports were intended to include data on the
percent of managers receiving training, average cost spent per employee training, and
percent of salary spent on training per agency, among other things.




Page 36                                          GAO-12-878 Federal Training Investments
that OPM has initiated some related efforts, but has not fully addressed
two of these issues. In order to decrease errors in reporting, OPM officials
and EHRI reports show that OPM has worked with agencies to identify
technical errors in their training data submission. However, to improve
reporting on cost data—which is currently a challenge for agencies—
OPM held one focus group with agencies in 2007, which it used to
updated it’s guidance on tracking training data in 2008. 50 OPM also has
not followed through on plans to annually provide agencies with reports of
their training data for verification and correction. According to OPM, the
purpose of the training data report is to (1) inform agencies of the training
data OPM had received (2) offer them the opportunity to work closely with
OPM in correcting any identified deficiencies and (3) to make note of the
progress they have made in addressing OPM’s training reporting
requirement. OPM officials said that they sent one report to agencies (in
fiscal year 2010) summarizing their training data and requesting
verification and this report was provided in response to expectations that
the data would be posted on the government-wide website Data.gov. 51 In
our review of examples of agency responses, we found that agencies
identified important discrepancies in their data, including significant
underestimates of the costs spent on training, and reported that they
would take steps to address incorrect data. However, OPM officials
informed us that they do not have a process for documenting whether
agencies have taken steps to correct their data.

Although OPM has only provided one summary of EHRI data to agencies,
agency officials that we met with stated that they could benefit from using
this type of data summary to improve their training data. Further, OPM
officials stated that using these summaries to improve EHRI data could
help agencies measure the return of investment on their training and
assist agencies’ stakeholders in making more informed decisions on the
best use of training dollars. During our review, OPM officials reported that
they began developing a report to send to agencies with fiscal year 2011
training data.




50
  OPM officials reported that they used this focus group to revise OPM’s 2008 Guide for
Collection and Management of Training Information.
51
     Accessed August 7, 2012.http:www.data.gov/.




Page 37                                            GAO-12-878 Federal Training Investments
Evaluate the benefits achieved   The TED group provides guidance and assistance to agencies in
through training and             evaluating training programs through three of its guides and in
development programs,            workshops. OPM’s Training Evaluation Field Guide is the primary guide
including improvements in        through which OPM advises agencies on how to evaluate training. The
individual and agency            guide instructs agencies to define the results they want to achieve from
performance                      training and to align training with the agency mission and goals. Further,
                                 the Guide discusses useful models for evaluating training and describes
                                 Return on Expectations 52 as the ultimate indicator to demonstrate the
                                 training value to stakeholders. 53 The guide also provides information on
                                 evaluation requirements outlined in laws and regulations, and provides
                                 practical instruction by identifying common challenges and solutions
                                 related to identifying the most cost-effective methods to train and develop
                                 employees. In addition to this guidance, OPM’s Training Policy Handbook
                                 instructs agencies to evaluate all training to determine whether or not it
                                 provides meaningful contributions to agency results. Similarly, in its Guide
                                 for Collection and Management of Training Information, OPM highlights
                                 the importance of collecting accurate, comprehensive training information
                                 and making it available to decision-makers and others who have a vested
                                 interest in the training activities of the federal government. This guide
                                 discusses the two basic types of performance measures for measuring
                                 training and development program effectiveness: process indicators and
                                 outcome indicators. 54

                                 In addition to its guides, OPM has made evaluation tools available to
                                 agencies on its website and held workshops on training evaluation in
                                 order help agencies identify and share best practices on evaluating



                                 52
                                   OPM’s 2011Training Evaluation Field Guide: Demonstrating the Value of Training at
                                 Every Level defines return on expectations as the ultimate indicator of value. It is the
                                 conversion of stakeholders’ expectations into observable, measureable outcomes of
                                 success for the training.
                                 53
                                   The guide discussed the Kirkpatrick Four-Level model (reaction, learning, performance
                                 and impact of training), the updated New World Kirkpatrick Four Levels and the Kirkpatrick
                                 Business Partnership Model as useful models for evaluating training.
                                 54
                                   Process indicators measure performance or key processes. They track such things as
                                 number of employees trained, number of training courses completed, or number of hours
                                 employees spent in training. Outcome indicators contain measures of service or
                                 performance results. These measure such things as an increase in employee and/or
                                 customer satisfaction, increase in employee job performance, and/or increase in
                                 organizational performance. Outcome indicators may require prolonged periods of
                                 measurement and multiple types and levels of measurements to include surveys, training
                                 impact measures, and return on investment analysis.




                                 Page 38                                            GAO-12-878 Federal Training Investments
                                  training. For example, OPM’s website contains a Training Evaluation Tool
                                  that describes the levels of training evaluation and provides agencies with
                                  evaluation questions to be answered in each of the four levels and the
                                  types of information typically collected.

                                  As previously mentioned, some agency officials reported that it is difficult
                                  to conduct these reviews because their cost and time demands can be
                                  significant. As a result, some agencies only conduct them for the most
                                  critical training and others reported that they do not have a formal process
                                  for conducting these reviews at all. While we agree that it is appropriate to
                                  target costly evaluations to the most important training, those who do not
                                  implement this practice at all could benefit from using OPM’s
                                  comprehensive guidance and assistance on training evaluations.

Compare training investments,     OPM does not have a guidance document that advises agencies on how
methods, or outcomes with         to compare training investment methods and outcomes with other
those of other organizations to   agencies, but provides some support to agencies in this area through
identify innovative approaches    technical assistance. For example, the TED group uses various web-
or lessons learned                based mechanisms such as OPM’s website, OPM LISTSERV, OPM
                                  Federal Training and Development web site and OPM Federal Training
                                  and Development Wiki to facilitate discussions between agencies on
                                  training investments. We observed the exchange and sharing of
                                  information among agencies through OPM’s LISTSERV, which is used by
                                  950 employees from various federal agencies to share training practices
                                  and advice. At times, agencies requested and shared information with
                                  each other on the most effective or efficient ways to implement specific
                                  training programs or requested models for which to compare their
                                  activities. Similarly, OPM’s wiki page contains examples and models of
                                  training programs for others to use when developing their training
                                  programs. According to OPM officials, the TED group also provides best
                                  practice forums on topics when they believe agencies need additional
                                  assistance. For example, OPM officials reported that they have held
                                  forums with agencies on the Training Evaluation Field Guide to share
                                  best practices and tools among agencies. OPM is working on a similar
                                  forum for developing supervisory training.


OPM Does Not Provide              OPM does not have guidance and assistance for three leading training
Guidance and Assistance           investment practices, two of which are areas in which agencies reported
on Three Leading Training         experiencing challenges. We examined the five guidance documents that
                                  OPM provides related to making training investment decisions and
Investment Practices              documentation on OPM’s technical assistance and did not find support for



                                  Page 39                                   GAO-12-878 Federal Training Investments
                                    the following three practices. OPM officials confirmed that that the agency
                                    does not provide direct guidance or assistance in some of these areas.

Identify the appropriate level of   In our review of OPM guidance and documentation on its technical
investment to provide for           assistance, we found that OPM provides some guidance to agencies
training and development            regarding steps to identify training and development investment needs
efforts and prioritize funding so   and related training strategies, but we did not find guidance on prioritizing
that the most important             these investments so that the most important training needs are
training needs are addressed        addressed first. According to TED officials, each agency must assess its
first                               own needs as the primary driver for investment determinations. To that
                                    end, OPM officials provide guidance and tools for conducting training
                                    needs assessments in OPM’s training policy guide and on OPM’s website
                                    and also direct agencies to review benchmarks in American Society for
                                    Training & Development’s State of the Industry reports. OPM officials also
                                    reported that they use the Human Capital Assessment Accountability
                                    Framework and related efforts to emphasize the importance of
                                    considering training as a solution to addressing mission critical
                                    competencies and skill gaps, but acknowledged that OPM does not
                                    provide specific guidance on prioritizing training investments through
                                    these processes. 55

                                    OPM officials identified guidance that they believe addresses the leading
                                    practice of prioritizing training investments; however, we found that the
                                    guidance does not address prioritization. As we previously mentioned, it
                                    is a leading practice for agencies to prioritize their training investments
                                    using criteria, such as expected demand for the training from internal
                                    sources, availability of resources to support the effort, potential for
                                    increased revenue, and risk of unfavorable consequences if investments
                                    are not made, or their potential to improve their efficiency. TED officials
                                    identified OPM’s 2000 Guide to Strategically Planning Training and
                                    Measuring Results as the source of guidance to agencies on this




                                    55
                                      Officials from OPM’s Agency Program Review office oversee OPM’s Human Capital
                                    Assessment Accountability Framework program. According to OPM documents, the
                                    Human Capital Assessment Accountability Framework consists of five human capital
                                    systems that together provide a consistent, comprehensive representation of human
                                    capital management for the federal government: strategic alignment, leadership and
                                    knowledge management, results-oriented performance culture, talent management, and
                                    accountability.




                                    Page 40                                       GAO-12-878 Federal Training Investments
                                practice. 56 In our review of this guide, we did not find any explicit
                                guidance to agencies on prioritizing training investments. Instead, the
                                guide advises agencies to build a business case for their training
                                strategies. OPM defines a business case as a method for projecting and
                                documenting the benefits to be gained as a result of investing resources
                                in a training intervention. The guide encourages agencies to consider
                                questions that are important to building a business case and provides an
                                example of how to build a business case using this information (see
                                Appendix III for diagrams from the guide on building a business case for
                                training). These steps are consistent with our identified leading practice.
                                However, the guide does not take the additional step of advising agencies
                                on how to prioritize training investments selected from their business
                                case(s) relative to each other. We have previously reported that, when
                                budgets are constrained, training is often one of first investments that
                                agencies reduce. Therefore, it is increasingly important for agencies to
                                prioritize their selected training activities, so that the most important
                                training is identified. Moreover, they need to communicate those priorities
                                agency-wide, in order to identify common needs and potential areas for
                                consolidated investments. As previously noted in our review, this is a
                                practice that most CHCOs reported that they do not implement, which, as
                                illustrated by our case example agencies, has resulted in costly
                                duplicative and inefficient training investments at some agencies.

Identify the most appropriate   In our review of OPM guidance and documentation on its technical
mix of centralized and          assistance, we did not identify specific guidance or assistance to
decentralized approaches for    agencies on this practice. As we previously noted, while neither approach
its training and development    fits every situation, agencies need to consciously think about the
programs.                       advantages and disadvantages of using centralized and decentralized
                                approaches, particularly for the design of training and development
                                programs. Although OPM officials confirmed that they do not provide
                                guidance and assistance to agencies in this area, OPM officials agreed
                                with this leading practice. We found that most agencies included in our
                                review reported that they already implement the practice, so additional
                                guidance may not be necessary.




                                56
                                  The guide recommends that agencies identify their agency-wide competency gaps,
                                assess whether training will address the gaps, identify and weigh the cost and benefits of
                                training, and then build a business case to support their selected training strategy.




                                Page 41                                           GAO-12-878 Federal Training Investments
Have criteria for determining    TED officials agreed that they do not provide guidance or assistance to
whether to design training and   agencies on this practice. TED officials stated that agencies need to
development programs in-         incorporate this leading practice into their training investment decision-
house or obtain these services   making process. As we previously noted, once an agency has identified
from a contractor or other       its training and development needs, it should make informed decisions
external source.                 about whether to design and develop training and development programs
                                 in-house or buy these services from a contractor or other external source.
                                 Factors that they should consider include the capability of in-house staff
                                 to develop and implement the training; the prior experience, capability,
                                 and stability of possible providers in the marketplace; and agency
                                 limitations on cost, time, and resources. As previously mentioned, 12 of
                                 the 27 CHCOs reported that they do not implement this leading practice,
                                 and our discussion with DOI officials, one of the four agencies that we
                                 interviewed for this review, illustrates that even those agencies that
                                 reported implementing this practice may not be doing so for all training in
                                 the agency.


Agency Officials Identified      Agency officials reported they could use more OPM assistance in
a Need for OPM to Provide        leveraging federal training investments across the government. Part of
More Assistance in               OPM’s role is to identify functional areas in which new or expanded
                                 interagency training activity is needed and either conduct such training or
Improving the                    arrange for agencies having the substantive competence to do so; as well
Effectiveness and                as to coordinate interagency training conducted by and for agencies.
Efficiency of Training and       Members of the CLO council emphasized that they could benefit from
Development Investments          more OPM assistance in achieving greater interagency collaboration on
Government-wide                  training to reduce duplicative training investments. All four agencies that
                                 we interviewed reported concerns similar to the Council’s. For example,
                                 DOI officials noted that OPM’s knowledge and expertise could help
                                 agencies identify one basic approach to competency management (e.g.,
                                 establishing levels of proficiency and competency validation processes)
                                 that can be used across government rather than using multiple
                                 approaches at various agencies. At the present, each agency individually
                                 identifies training needed for these competencies, which results in
                                 duplication and variation in the quality of training provided throughout the
                                 government. DOI officials stated it could be more efficient if agencies
                                 would use a standard set of Knowledge, Skills, and Abilities to hire and
                                 identify training and development investment priorities. The officials also
                                 suggested that OPM’s HR University could be used to provide training for
                                 other mission-critical occupations. Further, VA officials stated that it would




                                 Page 42                                    GAO-12-878 Federal Training Investments
assist agencies if OPM established government-wide courses for
mandatory training and cross-cutting areas. 57 As an example, the officials
stated the federal government has 17 different versions of No Fear Act
training. 58 The officials suggested that OPM could establish one
government wide training for such subjects which would help agencies
save federal time and money. Officials from DHS and DOE expressed
similar views.

In contrast, the shared training efforts that are being implemented by the
Federal Healthcare Training Partnership collaborative, which consists of
14 federal agencies that provide clinical health care or related training to
support their mission, illustrate the potential magnitude of savings that
could be achieved by leveraging training across agencies. The Federal
Healthcare Training Partnership was created by its members to share
training programs and resources across the agencies to speed up the
provision of employee learning and reduce training costs. According to
VA officials—who lead the effort, the agencies formed this group because
they saw the unaddressed need and opportunity to save costs in common
training areas. Documentation provided by VA on the collaborative group
states that in fiscal 2011, Federal Healthcare Training Partnership partner
organizations shared more than 2,300 programs, generating a total cost
avoidance of more than $82 million. 59 They did so by utilizing the partner
organizations’ existing learning systems to share training that was
originally developed for a single agency’s internal use and making it
available to all federal learners, as well as by coordinating the joint
development or purchase of training needed by two or more partner
agencies. VA officials stated that, while this has been a valuable effort to
specifically improve healthcare-related training investments for the
agencies involved, all federal agencies would benefit from an expansion
of leveraging training investments across the government.



57
  We are conducting a review of agency actions to address critical skill gaps, which
include training and also have work underway assessing training in cross-cutting
occupations, such as grant management and acquisition.
58
  Under the Notification and Federal Employee Antidiscrimination and Retaliation Act of
2002 (No Fear Act) and OPM’s implementing regulations, an agency must provide training
to its employees regarding the rights and remedies available under the employment
discrimination and whistleblower protection laws at least every two years. See, section
202(c) of Pub. L. No. 107-174, 116 Stat. 566, 569 (May 15, 2002) and 5 C.F.R. § 724.203.
59
 We did not verify the costs savings identified in the FHTP’s report.




Page 43                                           GAO-12-878 Federal Training Investments
OPM officials agreed that increased coordination of mandatory and
common training across the government could reduce duplication and
improve the efficiency of federal training investments. The officials
reported that OPM has already engaged in some efforts to partner with or
support CLO and CHCO Council efforts to share specific training across
agencies. For example, the officials worked with the Social Security
Administration to share a “Plain-Language” writing course developed by
the Social Security Administration with other agencies, by placing it on
OPM’s Training and Development Wiki page. In addition, OPM officials
stated that in 2010, the CHCO Council and OPM collaborated to establish
HR University, which is aimed at addressing the competency and skill
gaps within the HR community and achieve savings government-wide by
identifying and sharing the best HR training with all agencies. 60 While the
system was initially designed to provide training to the HR community, it
has also been used to provide some mandatory training and HR training
to other supervisors and managers. For example, officials reported that
they recently added a mandated Uniformed Services Employment and
Reemployment Rights Act training to HR University. The CHCO Council
and OPM have also developed a formula to calculate cost savings
resulting from the shared courses, which agencies can use to track their
savings and return on investment. According to the OPM Executive
Director of the CHCO Council and HR University’s website, in its first
year, HR University has saved the government $14.5 million as a result of
the shared training, and OPM officials expect that it could produce
significantly more savings, when other courses are added. 61 According to
OPM officials, while HR University primarily serves the needs of the HR
community, OPM would support using the HR University model to
centralize training in other occupations or functional areas.




60
  HR University was initially funded by the CHCO Council and operated in coordination
with OPM. However, OPM officials reported that they are currently working on a MOU with
the CHCO Council to transfer more ownership and operation of HR University to OPM,
with the CHCO Council as an advisor.
61
  According to the Executive Director of the CHCO Council, they developed this formula in
consultation with the CLO Council. We did not independently assess the formula or verify
the reported savings.




Page 44                                          GAO-12-878 Federal Training Investments
              The federal government’s efforts to build and retain a workforce of skilled
Conclusions   and efficient employees are essential to addressing skill gaps in critical
              fields and effectively and efficiently deliver services to the public. Training
              and development programs play a vital role in fulfilling these goals.
              However, agency leaders need to be as strategic about how they invest
              resources in this area as they are in other key areas of agency
              operations. Training investment decisions should be based on an
              assessment of the appropriate level of training investments and the
              prioritization of those investments, as well as an evaluation of the most
              cost-effective delivery mechanisms, and the known costs and benefits of
              their training investments. CHCOs and OPM each play a vital role in
              ensuring that these investment decisions are effectively made.

              While CHCOs report that they are implementing leading practices that
              support the successful delivery of training, they could do more to ensure
              that these investments are more cost effective. Because many CHCOs do
              not have the information that they need from component or subagency
              leaders regarding the level of training investments and mechanisms for
              setting priorities agency-wide, their agencies are duplicating some
              internal training investments and missing opportunities to leverage
              economies of scale and share the most effective training across their
              agencies. Many CHCOs are also limiting their opportunities to make
              training more cost effective and accessible because they are not
              comparing the merits of different training delivery mechanisms. OPM’s
              guidance and assistance in these three areas are minimal or absent and
              could be strengthened to assist agencies in implementing these leading
              practices. In addition to these limitations, some CHCOs do not have a
              formal process to evaluate the impact of training on their mission. While
              not all training and development programs require, or are suitable for,
              higher levels of evaluation, those who do not implement this practice for
              any training are missing information that could help them make more
              effective investment decisions and could benefit from using OPM’s
              existing guidance and assistance on conducting such evaluations.

              Federal agencies and OPM also need reliable information on how much
              agencies spend on training and for what purposes, in order to make
              effective training investment decisions. However, CHCOs do not
              completely and reliably track training costs agency-wide and, therefore,
              are unable to provide OPM with the reliable information that it needs to
              more effectively guide government-wide training policies. OPM has
              responsibility for providing regulations for maintenance of agency training
              data, assessing the completeness and quality of those data when
              agencies submit it, and using it to target its assistance to agencies. But


              Page 45                                     GAO-12-878 Federal Training Investments
                      OPM does not know the extent of the reliability of federal training
                      investment data because they have not compared improvements in the
                      completeness of the data over last 3 years and determined if it meets the
                      standards of sufficiency for data assessment and have not assessed the
                      quality of the data in the 6 years that agencies have been required to
                      submit it.

                      Given the fiscal challenges facing the nation, the federal government
                      needs to take advantage of every opportunity to better leverage
                      resources and investments across agencies. However, at present many
                      agencies independently purchase or develop training for the same
                      government-wide mandated courses. OPM has an opportunity to reduce
                      duplicative and inefficient training investments by leveraging existing
                      training resources government-wide. Agency leaders and OPM recognize
                      that this has led to redundant and inefficient federal training investments.
                      HR University—the one-stop-shop training platform administered by OPM
                      for many courses mostly related to the HR community—provides a model
                      that can result in cost savings and help standardize some mandatory
                      training courses across government.


                      To improve federal training investment decision-making processes, the
Recommendations for   Director of OPM should take the following five actions:
Executive Action
                      1. Include in existing or new OPM guidance or technical assistance
                         additional information in the following areas:

                          •     Steps agencies should take and factors they should consider
                                when prioritizing federal training investments agency-wide,
                                including developing a process to rank training using criteria, such
                                as expected demand for the investment from internal sources,
                                availability of resources to support the effort, potential for
                                increased revenue, and risk of unfavorable consequences if
                                investments are not made.

                          •     Steps agencies should take and factors they should consider for
                                comparing the merits of different delivery mechanisms and
                                determining the mix of mechanisms to use, in order to ensure
                                efficient and cost-effective delivery of federal training. Such
                                guidance could include requesting that agencies consistently
                                utilize Standard Form-182 to document and report training costs
                                associated with the different delivery mechanisms employed.




                      Page 46                                     GAO-12-878 Federal Training Investments
2. In line with statutory and regulatory provisions on maintenance and
   reporting of training information, work with the CHCO Council to
   improve the reliability of agency training investment information by:

    •     ensuring that agencies are familiar with and follow guidance
          outlined in OPM’s Guide for the Collection and Management of
          Training Information regarding which training events should be
          documented as training and reported to OPM;

    •     developing policies to strengthen the utilization of Standard Form-
          182 to document and report training costs;

    •     encouraging agencies through guidance and technical assistance,
          to develop policies that require consistent reporting of training
          data to their learning management systems; and

    •     encouraging each agency to assess its existing training
          information system(s) and identify whether it is providing complete
          and reliable data and, if not, to develop approaches to improve the
          system(s), in order to do so.

3. Provide regular report summaries to agencies on EHRI training
   investment data and its reliability, in order to improve the transparency
   and reliability of federal training investment data.

4. Once federal training data reliability has been sufficiently improved,
   consistent with Executive Order No. 11348, use EHRI data to:

    a) counsel heads of agencies and other agency officials on the
       improvement of training, and

    b) assist agencies in developing sound programs and financial plans
       for training and provide advice, information, and assistance to
       agencies on planning and budgeting training programs.

In collaboration with the CHCO and CLO Councils, identify the best
existing courses that fulfill government-wide training requirements, such
as mandatory Equal Employment Opportunity training, or training in
common federal occupations, such as basic training in financial
management, and offer them to all agencies through HR University or
other appropriate platform to reduce costly and duplicative federal training
investments.




Page 47                                     GAO-12-878 Federal Training Investments
                     We provided a draft of this report to the departments of OPM, DHS, DOE,
Agency Comments      DOI, and VA for review and comment. OPM commented on our five
and Our Evaluation   recommendations to their agency, concurring with one recommendation,
                     partially concurring on three recommendations and not concurring with a
                     portion of one recommendation. OPM’s official comments are reprinted in
                     appendix IV. OPM, DOI, and VA provided technical comments, which we
                     incorporated into our report, as appropriate. DOE and DHS had no
                     comments.

                     OPM partially concurred with our first recommendation that it should
                     provide in existing or new guidance information on prioritizing federal
                     training investments agency-wide and factors agencies should consider
                     for comparing the merits of different delivery mechanisms. OPM stated
                     that its publications mentioned in our report already provide guidance on
                     necessary steps and specific factors agencies should consider when
                     prioritizing training investments. However, none of the guides that we
                     obtained or that OPM provided for our review contain a specific
                     discussion about ranking training investments based on key factors that
                     should be considered, such as expected demand for the investment from
                     internal sources, availability of resources to support the effort, potential
                     for increased revenue, risk of unfavorable consequences if investments
                     are not made or the potential to improve efficiency. OPM stated that, as
                     part of its effort to revise the Human Capital Assessment Framework
                     resources that it provides to agencies, OPM plans to include tools and
                     guidance on steps agencies can take to prioritize learning investments as
                     part of its strategic human capital planning. We did not change our
                     recommendation, which is based on OPM’s current guidance and
                     assistance. OPM’s reported future plan to provide more specific guidance
                     on prioritization has the potential to address our recommendation, when
                     implemented. OPM also agreed to provide further guidance regarding
                     what steps agencies should take and what factors they should consider in
                     comparing the merits of different delivery mechanisms and determining
                     the mix of mechanisms to use to ensure efficient and cost-effective
                     delivery of federal training.

                     OPM did not concur with the portion of our second recommendation
                     regarding working with the CHCO council to improve the reliability of
                     agency training investments by developing a common definition of what
                     should be documented as training. OPM stated that the definition of
                     training is clearly stated in 5 U.S.C Chapter 41 and OPM’s Draft Training
                     Policy Handbook and Guide for the Collection and Management of
                     Training Information outlines which training events should be documented
                     as training and reported to OPM. Consequently, OPM recommended that


                     Page 48                                   GAO-12-878 Federal Training Investments
we delete this task for OPM. OPM’s Guide for the Collection and
Management of Training Information states that all courses, workshops
and conferences paid for by the government; all federally mandated
training; and, all agency required training should be reported to OPM’s
EHRI system. It further states that agencies do not have to report training
that occurs spontaneously or casually/incidentally (e.g., reading a book,
having a discussion, web casts, briefings, etc.); training that has no
specified training goals; training where there are no ways to evaluate if
the training improved knowledge, skills, abilities or competencies; and,
training that was not paid for by the government. We agree that this
guidance should assist agencies in knowing which training to track and
report, and therefore have removed this task from the recommendation.
However, given the concerns raised by officials in our case example
agencies regarding inconsistencies in whether conferences and other
trainings are actually tracked, and recent events regarding spending at
such training, we modified our recommendation to suggest that OPM
work closely with CHCOs to ensure that this guidance is followed as it
addresses the other actions we recommend to improve reliable reporting.
OPM concurred with the other actions identified in the recommendation
which included working with the CHCO council to: develop policies to
strengthen the utilization of Standard Form-182 to document and report
training costs; encourage agencies through guidance and technical
assistance, to develop policies that require consistent reporting of training
data to their learning management systems; and encourage each agency
to assess its existing training information system(s) and identify whether it
is providing complete and reliable data and, if not, to develop approaches
to improve the system(s), in order to do so.

OPM partially concurred with our third recommendation that it should
provide regular report summaries to agencies on EHRI training
investment data and its reliability, in order to improve the transparency
and reliability of federal training investment data. OPM stated that it will
provide regular summaries to agencies on the training investment data
submitted to OPM to improve transparency. However, OPM stated that
these summaries will not directly lead to improved reliability of the data
because agencies must take action to improve the data in order to have
an effect on data reliability. OPM also noted that agencies currently have
the option of working with OPM to secure a subscription to Business
Objects—a reporting tool that will allow agencies to run reports of the
data they have provided to OPM and determine whether those data
accurately reflect what is occurring in their agencies. OPM recommended
that we revise our recommendation to read, “Provide regular report
summaries to agencies on EHRI training investment data in order to


Page 49                                   GAO-12-878 Federal Training Investments
improve the transparency of federal training investment data.” We agree
that agencies are ultimately responsible for making changes to their data
to improve its reliability. However, OPM plays an important role in the first
step of that process by reporting the current information that it has, so
that agencies can make corrections. We believe that this
recommendation along with our prior recommendation on steps OPM and
CHCOs can take to improve reliability will contribute to improving the
transparency and reliability of agency training data. Therefore, we did not
make changes to this recommendation.

OPM concurred with our fourth recommendation that it should counsel
heads of agencies and other agency officials on the improvement of
training; assist agencies in developing sound programs and financial
plans for training; and provide advice, information, and assistance to
agencies on planning and budgeting training programs using EHRI data,
once federal training data reliability has been sufficiently improved. OPM
stated that it will consult with agencies on possible improvements and
assistance on planning training programs once federal training data are
reliable.

OPM partially concurred with our fifth recommendation that it should, in
collaboration with the CHCO and CLO Councils, identify the best existing
courses that fulfill government-wide training requirements, such as
mandatory Equal Employment Opportunity training, or training in common
federal occupations, such as basic training in financial management, and
offer them to all agencies through HR University or another appropriate
platform to reduce costly and duplicative federal training investments.
OPM stated that it agrees and is already collaborating with the CHCO and
CLO Councils to identify, collect, and share existing mandatory courses
that fulfill government-wide training requirements (e.g., Plain Writing,
Telework. USERRA, Veterans Employment, Constitution Day) through
HR University or on OPM’s Federal Training and Development Wiki.
Therefore, OPM recommended that we revise the recommendation to
recognize that the expansion of mandatory training by HR University
would be a continuation of efforts they have started. We have revised the
recommendation to reflect this comment.




Page 50                                   GAO-12-878 Federal Training Investments
We are sending copies of this report to the Director of OPM. In addition,
this report will be available at no charge on the GAO website at
www.gao.gov. If you have any questions about this report, please contact
me at 202-512-2717 or jonesy@gao.gov. Contact points for our Offices of
Congressional Relations and Public Affairs may be found on the last page
of this report. GAO staff who made key contributions to this report are
listed in appendix V.

Sincerely yours,




Yvonne D. Jones
Director
Strategic Issues




Page 51                                 GAO-12-878 Federal Training Investments
Appendix I: Objectives, Scope, and
              Appendix I: Objectives, Scope, and
              Methodology



Methodology

              To better understand how federal training investment decisions are made
              and whether improvements are needed, you asked us to review the
              methods that agencies are using to establish their training investment
              strategies and Office of Personnel Management’s (OPM) training
              investment guidance to agencies. Accordingly, this review assesses the
              extent to which (1) chief human capital officers (CHCOs) of selected
              federal agencies have established processes to set and prioritize training
              investments that are aligned with leading practices; and (2) OPM’s
              guidance and assistance for developing training investment strategies
              align with these leading practices.

              For the purposes of this review, we define the key terms “training”,
              “development” and “agency-wide” in the following ways:

              •   Training is making available to employees planned and coordinated
                  educational programs of instruction in professional, technical, or other
                  fields that are or will be related to the employee’s job responsibilities.
                  Training can be accomplished through a variety of approaches, such
                  as classroom training, e-learning, and professional conferences that
                  are educational or instructional in nature. 1

              •   Development is generally considered to include training, structured
                  on-the-job learning experiences, and education. Developmental
                  programs can include experiences such as coaching, mentoring, or
                  rotational assignments.

              •   Agency-wide includes all components, sub-agencies or offices within
                  a cabinet department or independent agency.




              1
               This definition of training includes mandatory training that governs how employees
              conduct themselves when carrying out their responsibilities, such as mandatory equal
              employment opportunity training or information technology security training.




              Page 52                                          GAO-12-878 Federal Training Investments
Appendix I: Objectives, Scope, and
Methodology




For both objectives of the review, we compared OPM and CHCO practices
against eight federal training investment leading practices, which are based
on our prior studies, other expert studies, and statutory, regulatory, and
executive order training requirements. 2 (See table 1 at the beginning of this
report). OPM reviewed these criteria and agreed that they are practices
that agencies should be implementing to support effective training
investment decisions. They also informed us that, while some leading
practices are related to training program requirements contained in
statutory, regulatory, or executive order provisions, responses to our
questions about the leading practices are not an indication of whether
agencies are in compliance with these laws and regulations. 3

To obtain government-wide information on agency training investment
practices, through a questionnaire on their training investment practices
and processes, we obtained high-level information from members of the
27 agencies represented on the CHCO Council. 4 We provided a standard
set of questions to each CHCO to ensure we consistently captured their
responses to our questions on their training investment practices. We
then analyzed the results of the questionnaire to identify the main themes
and develop summary findings. Two of our analysts conducted this
analysis, placed CHCO responses into categories, and tallied the number
of responses in each category. A third analyst traced the responses back
to the original questionnaire and verified the appropriate categorization of
CHCOs’ responses. To characterize CHCOs views throughout this report,
we defined modifiers (e.g., “many”) to quantify users’ views as follows:


2
 All of these practices are described in GAO-04-546G. We also used several studies from
the Corporate Leadership Council.
3
 Appendix II includes a table that lists the laws, regulations, and executive order
provisions that are related to the leading practices.
4
 The CHCO Council was established with 25 members. However, the Director of OPM
may designate other members of the Council. Such additional members may include, but
are not limited to: (1) the CHCOs of other executive agencies and (2) members who are
designated on an ex officio basis and who may be invited to contribute to projects, as
particular skills and expertise are needed. The council agencies include the Departments
of Agriculture, Commerce, Defense, Education, Energy, Health and Human Services,
Homeland Security, Housing and Urban Development , Interior, Justice, Labor, State,
Treasury, Transportation, and Veterans Affairs; Environmental Protection Agency:
General Services Administration: National Aeronautics and Space Administration: National
Science Foundation: Nuclear Regulatory Commission: Office of Management and Budget:
OPM: Office of the Director of National Intelligence: Small Agency Council: Small
Business Administration: Social Security Administration: and U.S. Agency for International
Development. CHCOs from all 27 agencies responded to our questionnaire.




Page 53                                            GAO-12-878 Federal Training Investments
Appendix I: Objectives, Scope, and
Methodology




•   “nearly all” users represents 23 to 27 users,

•   “most” users represents 18 to 22 users,

•   “many” users represents 13 to 17 users,

•   “several” users represents 8 to 12 users,

•   “some” users represents 3 to 7 users, and

•   “few” users represents 0 to 3 users.

To obtain additional perspective and insights on the training investment
practices identified in the questionnaire, we discussed the responses with
CHCO and chief learning officers (CLO) councils. In addition, based on
the responses to the questionnaire and workforce size, we selected four
agencies (the Department of Homeland Security, Department of Veterans
Affairs, Department of the Interior, and Department of Energy) from which
to obtain illustrative examples of how they implemented the training
investment practices identified in the questionnaire. (See table 6 for
selection traits). As part of our review of agency practices, we also
obtained information on the steps that agencies are taking to identify and
prioritize investment allocations for training required to implement the
GPRA Modernization Act of 2010 (GPRAMA). 5




5
 GPRAMA mandated the Director of OPM to: (1) within 1 year of enactment, identify, in
consultation with the Performance Improvement Council, the key skills and competencies
needed by federal government personnel for developing goals, evaluating programs, and
analyzing and using performance information for the purpose of improving government
efficiency and effectiveness; (2) within 2 years of enactment, incorporate, as appropriate,
key skills and competencies into relevant position classifications; and (3) within 2 years of
enactment, work with each agency to incorporate the key skills identified into training for
relevant employees at each agency. Pub.L. No. 111-352, 124 Stat. 3866 (Jan. 4, 2011).




Page 54                                            GAO-12-878 Federal Training Investments
                                        Appendix I: Objectives, Scope, and
                                        Methodology




Table 6: Selection of Case Example Agencies

                                                                                 Agency-wide                                    Agency-wide
                                                                                 process for             Centralized or         process for
                                 Training              Sets a specific           prioritizing            decentralized          tracking cost
Agency                           oversight             level of training         training                training investment    and delivery
selected        Workforce size   structurea            investment                investments             processes              of training
DHS             Greater than     None                  No                        Yes                     both                   No
                100,000
VA              Greater than     Oversight             Yes                       Yes                     both                   Yes
                100,000          committee and
                                 learning board
DOE             Less than        Oversight             No                        Yes                     decentralized          Yes
                100,000          committee and
                                 learning board
DOI             Less than        Oversight             No                        No                      decentralized          No
                100,000          committee
                                        Source: GAO analysis of CHCO responses to a GAO questionnaire.
                                        a
                                          During our meetings with DHS, VA, DOI and DOE, officials clarified or changed their answers to
                                        some of the other responses regarding their oversight structure as well as prioritizing and tracking
                                        training investments agency-wide. Similarly, VA, DOE, and DOI reported having decentralized
                                        processes in their questionnaire. However, when we met with officials in the agency, they described
                                        have both some centralized and decentralized training investment processes.


                                        To identify and assess OPM’s oversight and guidance to agencies on
                                        training investment strategies, we reviewed OPM training guidance,
                                        relevant documentation on forums, workshop or other assistance, and
                                        oversight activities. In addition, we interviewed officials from OPM offices
                                        with primary responsibility for providing training policy guidance and
                                        technical assistance to agencies. We compared this information to the
                                        leading practices identified in table 6. We also identified and described
                                        the steps, that OPM has taken to identify the skills and training needed to
                                        implement performance management improvements, such those required
                                        by GPRAMA, as a foundation for future agency training investments.
                                        However, we did not assess the effectiveness of OPM’s efforts to identify
                                        GPRAMA-related skills and actions to develop related training

                                        Based on information obtained from agencies and OPM, we assessed
                                        which leading training investment practices were being implemented by
                                        agencies and addressed by OPM guidance and assistance. We also
                                        identified the challenges or limitations reported by agencies to
                                        implementing the practices, and opportunities for improvement in agency
                                        processes and related OPM guidance.




                                        Page 55                                                          GAO-12-878 Federal Training Investments
Appendix I: Objectives, Scope, and
Methodology




We conducted this performance audit from December 2011 to September
2012, in accordance with generally accepted government auditing
standards. Those standards require that we plan and perform the audit to
obtain sufficient, appropriate evidence to provide a reasonable basis for
our findings and conclusions based on our audit objectives. We believe
that the evidence obtained provides a reasonable basis for our findings
and conclusions based on our audit objectives.




Page 56                                 GAO-12-878 Federal Training Investments
Appendix II: Table of Statutes, Regulations,
                                              Appendix II: Table of Statutes, Regulations,
                                              and Executive Orders Related to Leading
                                              Practices


and Executive Orders Related to Leading
Practices

Leading Training Investment Practicesa                    Related Statutes, Executive Orders, and OPM Regulations
1.   Agencies should (a) Identify the appropriate level   •   Exec. Order No. 11348, section 303(e) requires agency heads to
     of investment to provide for training and                establish priorities for needed training, and provide for the use of funds
     development efforts and (b) prioritize funding so        and man-hours in accordance with these priorities.
     that the most important training needs are           •   5 C.F.R. § 410.201(c), in implementing the E.O., requires agency heads
     addressed first                                          (or designee(s)) to establish priorities for training employees and allocate
                                                              resources according to those priorities.
2.   Agencies should identify the most appropriate        There are no statutory, regulatory, or Executive Order requirements
     mix of centralized and decentralized approaches      directly related to this practice.
     for its training and development programs
3.   Agencies should consider government-wide             There are no statutory, regulatory, or Executive Order requirements
     reforms and other targeted initiatives to improve    directly related to this practice.
     management and performance when planning its
     training and development programs
4.   Agencies should have criteria for determining        There are no statutory, regulatory, or Executive Order requirements
     whether to design training and development           directly related to this practice.
     programs in-house or obtain these services from
     a contractor or other external source
5.   Agencies should compare the merits of different      There are no statutory, regulatory, or Executive Order requirements
     delivery mechanisms (such as classroom or            directly related to this practice.
     computer-based training) and determine what
     mix of mechanisms to use to ensure efficient and
     cost-effective delivery
6.   Agencies should track the cost and delivery of its   •   5 U.S.C. 4118 authorizes OPM to prescribe regulations—and in doing
     training and development programs                        so—to specifically provide for
                                                               •   the maintenance of necessary information concerning the general
                                                                   conduct of the training activities of each agency, and such other
                                                                   information as is necessary “to enable the President and Congress
                                                                   to discharge effectively their respective duties for supervision,
                                                                   control and review of these training programs.”
                                                               •   The submission of reports by agencies on results and effects of
                                                                   training programs and plans and economies resulting therefrom,
                                                                   including estimates of costs of training.
                                                               •   5 C.F.R. § 410.601(a) requires agencies to maintain records of
                                                                   training plans, expenditures, and activities in such form and manner
                                                                   as necessary to submit to OPM. Subsection (b) provides that
                                                                   beginning December 31, 2006, agencies are to report training data
                                                                   at such times and in such form as required for OPM’s government-
                                                                   wide Electronic Data Collection System.




                                              Page 57                                            GAO-12-878 Federal Training Investments
                                             Appendix II: Table of Statutes, Regulations,
                                             and Executive Orders Related to Leading
                                             Practices




Agencies should evaluate the benefits achieved               •      5 U.S.C . § 4103(c) requires the head of an agency to evaluate, on a
through training and development programs, including                regular basis, each program or plan established, operated, or maintained
                                                                                          b
improvements in individual and agency performance                   under subsection (a) with respect to accomplishing specific performance
                                                                    plans and strategic goals in performing the agency mission and modify
                                                                    such program or plan as needed to accomplish such plans and goals.
                                                             •      Exec. Order. No. 11348, section 303(a) requires agency heads to
                                                                    review at least annually the agency’s program to identify training needed
                                                                    to bring about more effective performance at the least possible cost.
                                                             •      5 C.F.R . § 410.202–provides that agencies must evaluate their training
                                                                    programs annually to determine how well such plans and programs
                                                                    contribute to mission accomplishment and meet organizational
                                                                    performance goals.
                                                             •      5 C.F.R. 410.201(d)(4), requires heads of agencies (or their designee(s))
                                                                    to develop and maintain plans and programs that assess periodically, but
                                                                    not less often than annually, the overall agency talent management
                                                                    program to identify training needs within the agency as required by
                                                                    section 303 of E.O. 11348.
Agencies should compare training investments,                There are no statutory, regulatory, or Executive Order requirements directly
methods, or outcomes with those of other                     related to this practice.
organizations to identify innovative approaches or
lessons learned
                                             Source: GAO Analysis of relevant laws, regulations, and Executive Orders.
                                             a
                                              These key training investment practices are part of the framework outlined in the GAO’s guide GAO,
                                             Human Capital: A Guide for Assessing Strategic Training and Development Efforts for the Federal
                                             Government, GAO-04-546G (Washington, D.C.: March 2004). This guide summarizes attributes of
                                             effective training and development programs and it is based on the GAO analysis of prior work, other
                                             related expert studies, and federal training requirements.
                                             b
                                              5 USC 4103(a) requires agency head to establish, operate, maintain, and evaluate a program or
                                             programs, and a plan or plans there under, for the training of employees.




                                             Page 58                                                                     GAO-12-878 Federal Training Investments
Appendix III: Illustrations of Building a Business Case
                  Appendix III: Illustrations of Building a
                  Business Case for Training from OPM’s Guide
                  to Strategically Planning Training and
for Training from OPM’s Guide to Strategically Planning
                  Measuring Results


Training and Measuring Results

                  Figure 1: OPM illustration of questions used to develop a business case for training




                  Page 59                                       GAO-12-878 Federal Training Investments
Appendix III: Illustrations of Building a
Business Case for Training from OPM’s Guide
to Strategically Planning Training and
Measuring Results




Figure 2: OPM illustration of developing a business case for training




Page 60                                       GAO-12-878 Federal Training Investments
Appendix IV: Comments from the Office of
              Appendix IV: Comments from the Office of
              Personnel Management



Personnel Management




              Page 61                                    GAO-12-878 Federal Training Investments
Appendix IV: Comments from the Office of
Personnel Management




Page 62                                    GAO-12-878 Federal Training Investments
Appendix IV: Comments from the Office of
Personnel Management




Page 63                                    GAO-12-878 Federal Training Investments
Appendix IV: Comments from the Office of
Personnel Management




Page 64                                    GAO-12-878 Federal Training Investments
Appendix IV: Comments from the Office of
Personnel Management




Page 65                                    GAO-12-878 Federal Training Investments
Appendix V: GAO Contact and Staff
                  Appendix V: GAO Contact and Staff
                  Acknowledgments



Acknowledgments

                  Yvonne J. Jones, Director (202) 512-2717 or Jonesy@gao.gov
GAO Contact
                  In addition to the contact named above, William Doherty (Assistant
Staff             Director), Latesha Love, Angela Leventis, and Karin Fangman made key
Acknowledgments   contributions to this report. Also contributing to this report were Benjamin
                  Crawford, Eric Gorman, and Natalie Maddox.




(450948)
                  Page 66                                   GAO-12-878 Federal Training Investments
GAO’s Mission         The Government Accountability Office, the audit, evaluation, and
                      investigative arm of Congress, exists to support Congress in meeting its
                      constitutional responsibilities and to help improve the performance and
                      accountability of the federal government for the American people. GAO
                      examines the use of public funds; evaluates federal programs and
                      policies; and provides analyses, recommendations, and other assistance
                      to help Congress make informed oversight, policy, and funding decisions.
                      GAO’s commitment to good government is reflected in its core values of
                      accountability, integrity, and reliability.

                      The fastest and easiest way to obtain copies of GAO documents at no
Obtaining Copies of   cost is through GAO’s website (http://www.gao.gov). Each weekday
GAO Reports and       afternoon, GAO posts on its website newly released reports, testimony,
                      and correspondence. To have GAO e-mail you a list of newly posted
Testimony             products, go to http://www.gao.gov and select “E-mail Updates.”

Order by Phone        The price of each GAO publication reflects GAO’s actual cost of
                      production and distribution and depends on the number of pages in the
                      publication and whether the publication is printed in color or black and
                      white. Pricing and ordering information is posted on GAO’s website,
                      http://www.gao.gov/ordering.htm.
                      Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
                      TDD (202) 512-2537.
                      Orders may be paid for using American Express, Discover Card,
                      MasterCard, Visa, check, or money order. Call for additional information.
                      Connect with GAO on Facebook, Flickr, Twitter, and YouTube.
Connect with GAO      Subscribe to our RSS Feeds or E-mail Updates. Listen to our Podcasts.
                      Visit GAO on the web at www.gao.gov.
                      Contact:
To Report Fraud,
Waste, and Abuse in   Website: http://www.gao.gov/fraudnet/fraudnet.htm
                      E-mail: fraudnet@gao.gov
Federal Programs      Automated answering system: (800) 424-5454 or (202) 512-7470

                      Katherine Siggerud, Managing Director, siggerudk@gao.gov, (202) 512-
Congressional         4400, U.S. Government Accountability Office, 441 G Street NW, Room
Relations             7125, Washington, DC 20548

                      Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800
Public Affairs        U.S. Government Accountability Office, 441 G Street NW, Room 7149
                      Washington, DC 20548




                        Please Print on Recycled Paper.