oversight

Best Practices: DOD Training Can Do More to Help Weapon System Programs Implement Best Practices

Published by the Government Accountability Office on 1999-08-16.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                   United States General Accounting Office

GAO                Report to the Chairman and Ranking
                   Minority Member, Subcommittee on
                   Readiness and Management Support,
                   Committee on Armed Services,
                   U.S. Senate
August 1999
                   BEST PRACTICES

                   DOD Training Can Do
                   More to Help Weapon
                   System Programs
                   Implement Best
                   Practices




GAO/NSIAD-99-206
United States General Accounting Office                                                    National Security and
Washington, D.C. 20548                                                              International Affairs Division



           B-280234                                                                                           Letter

           August 16, 1999

           The Honorable James Inhofe
           Chairman
           The Honorable Charles Robb
           Ranking Minority Member
           Subcommittee on Readiness and Management Support
           Committee on Armed Services
           United States Senate

           As you requested, this report assesses the extent the Department of Defense’s (DOD) training helps
           weapon system program offices apply best practices and whether such training can be of more help.
           We make recommendations to the Secretary of Defense on how DOD training can better support
           program offices in the application of best practices.

           We are sending copies of this report to the Honorable William S. Cohen, Secretary of Defense; the
           Honorable Louis Caldera, Secretary of the Army; the Honorable Richard Danzig, Secretary of the
           Navy; the Honorable F. Whitten Peters, Acting Secretary of the Air Force; the Honorable Jacob J. Lew,
           Director, Office of Management and Budget; and to interested congressional committees. We will also
           make copies available to others upon request.

           If you or your staff have any questions regarding this report, please call me at (202) 512-4841.
           Contacts and key contributors to this report are listed in appendix III.




           Katherine V. Schinasi
           Associate Director
           Defense Acquisitions Issues




                      Leter
Executive Summary



Purpose              The Department of Defense (DOD) plans to increase its annual
                     procurement investment to about $60 billion by fiscal year 2001. DOD has
                     high expectations from this investment: that new weapons will be better
                     yet less expensive than their predecessors and will be developed in half the
                     time. Essential to getting these kinds of outcomes will be the adaptation of
                     best commercial practices that have enabled leading commercial firms to
                     develop new products faster, cheaper, and better. DOD has begun a number
                     of acquisition reform initiatives based on commercial practices to help
                     foster these outcomes. Their success depends greatly on the extent to
                     which the program offices responsible for managing weapon acquisitions
                     can implement the practices on individual programs. Training provided to
                     the program offices serves as a key agent in both creating a culture that is
                     receptive to new practices and in providing the knowledge needed to
                     implement new practices at the workplace. At the request of the Chairman
                     and the Ranking Minority Member, Subcommittee on Readiness and
                     Management Support, Senate Committee on Armed Services, GAO
                     evaluated the role DOD training is playing in implementing best practices in
                     weapon system programs. This report addresses (1) the contribution DOD
                     training makes to program offices’ ability to apply best practices, (2) the
                     different methods used by DOD and leading commercial firms in training
                     on best practices, and (3) the strategic approaches that underlie DOD’s and
                     leading commercial firms’ training methods for best practices.



Background           GAO’s review focused on weapon system program offices because they
                     comprise a key component of DOD’s acquisition workforce. In planning,
                     managing, and executing acquisition programs, these program offices are
                     responsible for managing about $80 billion of DOD’s annual research,
                     development, and procurement funds. As an entry point for DOD
                     acquisitions, program offices greatly influence the work of the rest of the
                     acquisition workforce. The primary responsibility for training the
                     acquisition workforce falls within the Office of the Under Secretary of
                     Defense for Acquisition and Technology. In 1992, the Defense Acquisition
                     University, a consortium of 13 schools, was created to develop and provide
                     training for the acquisition workforce. Each service also has an acquisition
                     reform office that provides the workforce with the latest information about
                     practices and initiatives that apply to acquisitions. Based on personnel
                     reductions mandated by the National Defense Authorization Acts for Fiscal
                     Years 1996 and 1997, DOD expects the acquisition workforce of fiscal year
                     2000 to be 25 percent smaller than that of fiscal year 1995. Thus, training




             Leter   Page 2                                          GAO/NSIAD-99-206 Best Practices
                      Executive Summary




                      will become even more important as new authority and responsibility is
                      granted to those who remain in the workforce.

                      To determine the extent to which DOD training and other sources helped
                      program offices obtain the knowledge needed for implementing best
                      practices, GAO focused on five specific practices: cost as an independent
                      variable, integrated product teams, evaluation of contractors’ past
                      performance, setting performance specifications, and managing supplier
                      relationships. The first four are formal DOD initiatives that are based on
                      best commercial practices, while supplier relationships is a best practice
                      GAO has observed in leading commercial firms. GAO selected six program
                      offices considered by DOD as leaders in implementing one or more of the
                      practices. As such, they represented best case examples of marshaling
                      training and other resources needed to implement new acquisition
                      practices. The term “standard training” is used in this report to describe
                      the training provided by the Defense Acquisition University and the
                      services’ acquisition reform offices, as distinct from training that program
                      offices provide on their own.



Results in Brief      DOD’s standard training did not make a major contribution to the leading
                      program offices’ ability to implement best practices. In evaluating their key
                      sources of knowledge, none of the key officials from programs at the
                      forefront of implementing best practices ranked standard DOD training
                      first, with many ranking it last. DOD training either did not reach the right
                      people when it was needed or did not reach them at all. When training on
                      best practices was received, it did not contain the depth or practical
                      insights program office people needed to implement the practices. It was
                      primarily through their own efforts—learning on the job, finding external
                      training, or developing their own training program—that they attained the
                      knowledge needed to apply best practices. Thus, success depended on
                      their having the foresight to see what was needed, the ingenuity to find
                      good sources of knowledge, and the resources needed to attain that
                      knowledge. Replicating this approach broadly on other programs is
                      problematic. Managers may not see the relevance of a practice to their
                      programs and thus may not realize what training is needed. Others may not
                      be able to afford the needed training.

                      Leading commercial firms and DOD use different training methods to
                      implement key practices. Commercial firms use targeted, hands-on
                      methods to ensure that program offices are trained on key practices. Their
                      training organizations conduct front-end analyses to determine the



              Leter   Page 3                                          GAO/NSIAD-99-206 Best Practices
Executive Summary




programs’ training requirements and involve the program offices in
designing the training. Training is customized to meet the specific needs of
those implementing the practice. Company officials believe the targeted
method results in more useful training, which helps to improve outcomes
of the final product. DOD does not have a counterpart to this method.
DOD relies primarily on its standard training, including classroom courses,
videos, internet-based training, satellite broadcasts, and roadshows, to
inform staff on best practices. These methods were designed for functional
training, such as for engineers, and for increasing the awareness of new
practices. As such, they do not provide the depth or reach enough of the
right people at the right time to be of great help in implementing best
practices at program offices. Also, DOD does not systematically involve
program offices in the design of training.

The intensive training methods leading commercial firms employ on new
practices are the result of a strategic, institutionally driven approach to
implementation. These firms commit their resources and attention to a few
well-defined practices and make a significant front-end investment in the
training to be provided to the workforce. Also, the firms strive to create an
environment to put those responsible for implementing the practices in a
good position to succeed. DOD’s training methods for best practices do not
stem from such a strategic approach. DOD has promulgated as many as 40
acquisition management initiatives in the past few years without
communicating their relative priority to trainers or implementers. Often,
the initiatives have not been accompanied by clear guidance or by the
initial training needed for implementation. While DOD commits significant
resources to training, it does not make a uniform front-end investment to
ensure that program offices will succeed with the new practices. Since
1997, two studies commissioned by DOD have recommended ways to make
training organizations more effective in providing training of best practices.
These recommendations were not adopted in favor of a more traditional
training role. In June 1999, another DOD study proposed that DOD training
organizations become change agents and be modeled after their corporate
counterparts. If the latest study’s recommendations are adopted, DOD may
be in a much better position to provide the type of help program offices
need to successfully implement best practices.

GAO makes recommendations to the Secretary of Defense on how DOD’s
training on best practices can better support the needs of program offices.




Page 4                                           GAO/NSIAD-99-206 Best Practices
                              Executive Summary




Principal Findings

DOD Training Not a Major      Training often did not exist or was not provided when program officials
Catalyst for Best Practices   began to implement an individual practice. For example, when the Joint
                              Strike Fighter program office started to implement cost as an independent
                              variable, there were no guiding documents, and no one, including the
                              training community, knew what the initiative really meant. Some people
                              involved with implementing best practices were missed altogether by DOD
                              training offerings because they had not been required to take training since
                              before the initiatives began. Others, such as requirements authors, fell
                              outside the definition—and training curriculum—of the acquisition
                              workforce. In one case, program officials reduced contractor reporting
                              requirements to 2 items, in line with acquisition reform, only to have 40
                              more added by another office with approval authority over the contract.
                              Contractors are also essential to the application of best practices, but they
                              are not part of the DOD defined acquisition workforce or the training
                              offered. Consequently, they do not necessarily understand or know how to
                              implement new practices. Program officials noted that DOD’s standard
                              training typically provided a general awareness of the practices but not the
                              “how-to” knowledge needed for implementation. Training was not tailored
                              in such a way that individuals could see how the practices could be applied
                              to their program. Program officials also observed that in some cases
                              training suffered because instructors did not use up-to-date case studies
                              and were not current on new practices themselves. They also believed they
                              had little opportunity to influence the training they received from DOD.

                              Program officials used a combination of ways—generally outside of
                              standard DOD training offerings—to get the knowledge they needed to
                              apply best practices. Several program officials relied on the cumulative job
                              expertise of the staff and personal research to teach themselves how to
                              implement new initiatives. For example, on the Advanced Medium-Range
                              Air-to-Air Missile program, officials relied on their personal experiences to
                              know how to set contract specifications at a performance, rather than a
                              detailed, level. Officials also used their program funds to send staff to
                              nongovernment sources or to bring experts in. One program manager
                              sends people to outside training, such as university leadership courses, to
                              develop creative thinking skills. One of his managers said some of his best
                              training was from off-site sessions sponsored by the program office that
                              dealt with the people issues critical to making integrated product teams
                              work. For the Advanced Amphibious Assault Vehicle program, the prime



                              Page 5                                          GAO/NSIAD-99-206 Best Practices
                              Executive Summary




                              contractor was responsible for providing the training on integrated product
                              teams and cost as an independent variable. The contractor hired a third
                              party to develop a training program tailored to the Advanced Amphibious
                              Assault Vehicle program, and both program office and contractor staff
                              were taught together, on-site as a team.


DOD’s Training Methods for    For routine training, such as skill building, leading commercial firms have
Best Practices Do Not Go as   standard training offerings, including functional area courses and
                              instruction on corporatewide issues, such as communications or ethics.
Far as Leading Commercial
                              However, when implementing key practices—such as those that change
Firms’                        how product development and production are conducted—leading firms go
                              beyond standard training offerings. Commercial firms use a targeted,
                              hands-on approach to ensure program teams are put in a good position to
                              implement a new practice. The elements common to how leading firms
                              provide training on a key practice include (1) front-end analysis of program
                              teams’ needs and training requirements, (2) involvement of program teams
                              in key training decisions, (3) customized training to meet program team
                              specific needs, and (4) targeted training for the implementation of specific
                              practices. Program staff participate in and often influence a wide range of
                              training decisions, including the amount of training provided for certain job
                              descriptions, course topics, depth of course coverage, and identification of
                              the appropriate course recipients. The involvement of the program staff
                              has improved course depth, timeliness, and coverage of personnel in the
                              commercial firms.

                              In the Boeing Company, training representatives develop a partnership with
                              the program staff when a team is formulated to design and manufacture a
                              new airplane. The training organization forms “drop teams” to colocate
                              with the program staff to learn as much as possible about the business
                              process and the staff’s concerns and to determine what training is needed
                              to help the program staff implement a practice. Boeing officials stated that
                              training was instrumental to the implementation of key practices, such as
                              design build teams, on the 777 aircraft program. They noted that such
                              teams were at odds with the company’s culture because employees were
                              not accustomed to working in a team environment and sharing information
                              across functional areas. Training officials worked side by side with the
                              program staff to create a training program that provided team building and
                              conflict resolution techniques as well as new technical skills training. To
                              ensure all program staff were equally trained, employees were required to
                              complete training before they reported for duty on the program. The
                              professional employees—engineers and drafters—were required to



                              Page 6                                          GAO/NSIAD-99-206 Best Practices
Executive Summary




complete 120 hours of start-up training on several key 777 practices before
they were allowed to report for duty. Teams were often trained together at
the work location. Ford followed a similar approach when it implemented
the Ford Product Development System—a lean engineering process.

Responsibility for training on best practices is diffused among several DOD
organizations, including the Defense Acquisition University and the service
acquisition reform offices. However, GAO did not find an organization that
was able to tailor and help deliver training on best practices to the program
offices visited. Training provided by the university is designed primarily to
enable people in individual career fields or functions, such as engineering
and cost estimating, to meet professional certification requirements. The
university incorporates best practices topics into these functionally
oriented courses as drop-in modules that provide a survey of the topic, but
not in-depth coverage. Although program offices see a greater need for
training that cuts across functions to implement new practices and to
manage in a team environment, it is difficult for a person in one career field
to obtain training in another field. The usefulness of these courses for best
practices is further hampered by limited availability; according to an
official from the university, the member schools get about 10 percent of the
workforce into training each year.

DOD’s Acquisition Reform Communication Center and the acquisition
reform offices in the services communicate best practice information
through videos, periodic satellite broadcasts, roadshows, and Acquisition
Reform Week. These methods can reach more people than Defense
Acquisition University courses and are designed around practices—versus
functions—but are not tailored to specific program offices and are not
necessarily delivered at the time those implementing new practices or
initiatives need them. For example, roadshows, traveling multiday training
workshops provided to staff at a number of locations, typically provide
awareness training on the practices. DOD officials estimate that only 10 to
15 percent of the acquisition workforce attend the second day of
workshops, where more detailed training is provided. The annual
Acquisition Reform Weeks, which are a combination of satellite broadcasts
and local presentations, mainly provide awareness-level training. Neither
individual attendance nor the level of learning attained by attendees is
tracked by these methods.




Page 7                                           GAO/NSIAD-99-206 Best Practices
                              Executive Summary




Differences Between DOD       Leading commercial firms shared a common strategy for adopting and
and Commercial Training       deploying key new practices. First, the firms’ corporate management
                              committed to and adopted few key practices—seven or less—at any given
Reflect Different Strategic
                              time. In doing so, the companies were able to concentrate their attention
Approaches                    and target resources to implementing the practices. It also signaled the
                              importance of the practices to trainers and implementers. Second, the
                              firms assessed which staff should be included in the implementation. For
                              example, Ford’s training unit determined which engineering teams working
                              on vehicle lines could benefit from the new production development
                              system. Third, company leaders made implementing the practice
                              mandatory for the target population. Lastly, companies developed
                              well-defined learning objectives to better ensure that the target population
                              consistently understood how to apply a new practice to improve
                              production outcomes—the ultimate goal. According to officials from these
                              firms, it was a corporate responsibility to ensure that those implementing
                              the practice received the necessary training and other assistance to
                              succeed. It was for this reason that the companies made a significant
                              front-end investment to support the needs of program offices that would
                              implement key practices. Company officials also pointed out that training
                              is just one of the several components necessary for adopting new practices.
                              They stressed that providing strong leadership and the right environment
                              were key to ensuring the implementation of new practices and to
                              developing quality training.

                              DOD’s approach to implementing best practices is less structured and more
                              reliant on individual program offices to make the necessary training
                              investment. DOD policies on individual reform initiatives are typically
                              promulgated without indicating what components of the acquisition
                              workforce or which programs are expected to implement the practice. In
                              addition, the policies themselves are not always clear. For example,
                              although the initiative on cost as an independent variable was promulgated
                              in 1995, Advanced Amphibious Assault Vehicle program officials
                              developed training in 1997 to define the initiative for the program. In
                              November 1998, an Air Force workshop on the cost initiative reported that
                              it was still not well understood or widely implemented. DOD has proposed
                              over 40 reform initiatives since 1994, without an indication of relative
                              priority, leaving educators and implementers to decide what is important.
                              Program offices are not necessarily in a good position to sort through the
                              initiatives to focus on those that are the most important to the job at hand.
                              A service acquisition reform official observed that the combination of many
                              reform initiatives and unclear priorities causes the office to guess what is




                              Page 8                                          GAO/NSIAD-99-206 Best Practices
Executive Summary




the most important, which leads to emphasizing what is perceived to be
popular.

DOD is aware of the need to improve the means by which the acquisition
workforce receives and implements new initiatives. A 1997 study by a DOD
team and a 1998 study by the Logistics Management Institute pointed out
several weaknesses in the focus and delivery of DOD training. Weaknesses
included the need for the Defense Acquisition University to be more active
in implementing best practices and reforms, the tendency for the training
curriculum to be functionally stove-piped, and the need for the university
to have more involvement with the workforce—the recipients of training.
The studies made recommendations for significant organizational and
operational changes in the university, including that it should follow the
corporate university model of becoming a change agent and a proponent of
best practices and put more emphasis on cross-functional training. In
September 1998, the university proposed a revised structure, which has not
been approved. While the proposal offers some improvements, it stays
close to its roots of providing functional training. It does not reflect the
corporate university model, a broadened role as a change agent, or a closer
relationship with the acquisition workforce. In short, it does not
discernibly address key weaknesses in providing training of best practices.

In June 1999, a DOD study team chartered to identify training on
commercial business practices for the acquisition workforce concluded
that adopting the most effective commercial practices requires a cultural
and organizational transformation within DOD. The team proposed a
cross-functional plan for managing acquisitions that embraces best
practices and calls for “learning organizations that seek out and adopt best
practices that improve individual and organizational performance.” The
plan proposes new roles for several organizations in fostering the adoption
of best practices. Among these is the Defense Acquisition University. For
example, it recommends that the university be broadly recast to adopt the
corporate university model and become a change agent. This plan, while
not specific about the help that program offices would receive, does call for
a strategic approach that would make it more likely that DOD could
provide its program offices tailored training—more help—in implementing
best practices. However, the fact that the September 1998 and the June
1999 proposals are still vying for approval indicates that DOD has not yet
decided what role it wants acquisition training to play on best practices.




Page 9                                          GAO/NSIAD-99-206 Best Practices
                  Executive Summary




Recommendations   GAO makes several recommendations to the Secretary of Defense that are
                  intended to ensure that DOD’s approach to the training of new practices
                  better supports the needs of program offices by (1) developing a strategy
                  for a structured approach to training on new practices; (2) providing
                  tailored training assistance to program offices; and (3) improving the
                  standard training curriculum so that it is more timely, relevant, and
                  accessible. These recommendations appear in full in chapter 5.



Agency Comments   DOD concurred with the views expressed in the report and all of the
                  recommendations. A discussion of DOD’s actions appears in chapter 5.
                  DOD’s comments appear in appendix I.




                  Page 10                                       GAO/NSIAD-99-206 Best Practices
Executive Summary




Page 11             GAO/NSIAD-99-206 Best Practices
Contents



Letter                                                                                                1


Executive Summary                                                                                     2


Chapter 1                 Responsibilities for Training the Acquisition Workforce Within
                            DOD                                                                      16
Introduction              Objectives, Scope, and Methodology                                         21


Chapter 2                 DOD Training Did Not Reach the Right People at the Right Time              26
                          DOD Training Offerings Did Not Provide the Depth Needed to
DOD Training Is Not a       Implement Best Practices                                                 31
Major Catalyst for Best   Limitations of Standard Training Led Program Offices to Develop
                            Their Own Training Solutions                                             34
Practices

Chapter 3                 Commercial Firms Use Targeted, Hands-on Methods to Improve
                            Training Usefulness                                                      39
DOD Training Methods      DOD Does Not Target Training on Key Practices to Program Offices           45
for Best Practices Do
Not Go as Far as
Leading Commercial
Firms

Chapter 4                 Leading Firms’ Strategic Approach Better Ensures Implementation
                            of Key Practices                                                         50
DOD and Commercial        DOD’s Training on Best Practices Does Not Stem From a Strategic
Training Methods            Approach                                                                 52
                          DOD Proposals to Alter Training Reflect Conflicting Approaches             54
Reflect Different
Strategic Approaches
to New Practices

Chapter 5                 Conclusions                                                                59
                          Recommendations                                                            60
Conclusions and           Agency Comments and Our Evaluation                                         62
Recommendations


                          Page 12                                        GAO/NSIAD-99-206 Best Practices
                       Contents




Appendixes             Appendix I: Comments From the Department of Defense                        64
                       Appendix II: Description of Program Offices Visited                        67
                       Appendix III: GAO Contacts and Staff Acknowledgments                       69


Related GAO Products                                                                              72


Tables                 Table 1.1: The Defense Acquisition and Technology Workforce
                         as of April 1999                                                         18
                       Table 1.2: Best Practices Evaluated at Program Offices Visited             23


Figures                Figure 1.1: Organizations Responsible for Training
                         Development                                                              20
                       Figure 2.1: JSF                                                            28
                       Figure 2.2: JSTARS-JTT                                                     30
                       Figure 2.3: AMRAAM                                                         32
                       Figure 2.4: AAAV                                                           35
                       Figure 2.5: JASSM                                                          37
                       Figure 3.1: 777 Airplane                                                   42
                       Figure 3.2: Ford Focus                                                     43




                       Page 13                                        GAO/NSIAD-99-206 Best Practices
Contents




Abbreviations

AAAV         Advanced Amphibious Assault Vehicle
AMRAAM       Advanced Medium-Range Air-to-Air Missile
ARCC         Acquisition Reform Communications Center
BCIS         Battlefield Combat Identification System
CAIV         cost as an independent variable
DAU          Defense Acquisition University
DAWIA        Defense Acquisition Workforce Improvement Act
DOD          Department of Defense
FPDS         Ford Production Development System
IBM          International Business Machines
IPT          integrated product teams
JASSM        Joint Air-to-Surface Standoff Missile
JSF          Joint Strike Fighter
JSTARS-JTT   Joint Surveillance Target Attack Radar System-Joint Tactical
             Terminal



Page 14                                       GAO/NSIAD-99-206 Best Practices
Contents




Page 15    GAO/NSIAD-99-206 Best Practices
Chapter 1

Introduction                                                                                          Chapte1
                                                                                                            r




                        The Department of Defense (DOD) plans to increase its annual
                        procurement investment to $60 billion by fiscal year 2001. DOD has high
                        expectations from this investment: that new weapons will be better and
                        less expensive than their predecessors and will be developed in half the
                        time. Essential to getting these kinds of outcomes will be the adaptation of
                        best commercial practices that have enabled leading commercial firms to
                        develop new products faster, cheaper, and better. To help foster these
                        outcomes, DOD has begun a number of acquisition reform initiatives based
                        on commercial practices. Success depends greatly on the extent to which
                        the program offices responsible for managing weapon acquisitions can
                        implement the practices on individual programs.

                        The training DOD provides program office staff to help them implement
                        best practices should play a central role in getting the desired outcomes.
                        While first-hand experience and “learning by doing” are instrumental in
                        adopting new practices, training serves as a key agent in creating a culture
                        that is receptive to new practices and providing the knowledge to
                        implement the new practices at the workplace. The relationship between
                        training and implementing new practices was highlighted in a 1994 study of
                        300 “improvement-driven” organizations conducted by Coopers &
                        Lybrand—with the American Society of Quality Control, Rutgers University
                        Center for Public Productivity, and the National Institute of Canada. The
                        study found that training played a “critical, integrative role as driver of
                        cultural change, process alignment, job redesign, and continuous
                        improvement.” Organizations included in the study used training as the
                        vehicle for implementing and sustaining the changes at the level where the
                        work was done. The leading commercial firms we reviewed committed
                        substantial investments to the training on key practices, underscoring its
                        importance to getting the outcomes desired by the firms. The significant
                        numbers of reform initiatives that DOD has introduced, which involve
                        adopting a number of new practices in the acquisition of weapons, present
                        implementation challenges that training can help meet.



Responsibilities for    For nearly 50 years, the importance of an educated professional DOD
                        acquisition workforce has been emphasized by government leaders and
Training the            reflected in the work of key studies and reform commissions. The First
Acquisition Workforce   and Second Hoover Commissions (1949 and 1955), the Fitzhugh
                        Commission (1970), and the Commission on Government Procurement
Within DOD              (1972) all recognized the importance of high quality, well-educated
                        acquisition professionals to the successful operation of DOD. The Packard
                        Commission, which undertook a broad examination of DOD management



                        Page 16                                         GAO/NSIAD-99-206 Best Practices
Chapter 1
Introduction




practices and procedures, reported in its June 1986 report that the DOD
acquisition workforce was undertrained and inexperienced. One of its
recommendations was to improve the education and training of the
acquisition workforce for the purpose of enhancing the defense acquisition
process.

Based in part by the recommendations of the Packard Commission, the
Defense Acquisition Workforce Improvement Act (DAWIA) was passed in
1990 as part of the National Defense Authorization Act for Fiscal Year
1991.1 Its primary objective was to improve the DOD acquisition system by
enhancing the education, training, and career development of members of
the acquisition workforce. Accordingly, DAWIA established the Defense
Acquisition University (DAU) to provide for the professional educational
development and training of the DOD acquisition workforce. The act also
charged DOD officials with the responsibility to designate certain positions
as acquisition positions, to set qualification requirements, and to establish
policies and procedures for training the acquisition workforce.

The program offices that manage weapon system acquisitions are a key
component of DOD’s acquisition workforce. This workforce is generally
defined as those people who are responsible for managing the wide array of
DOD acquisitions, including contracting professionals, program managers,
engineers, scientists, logisticians, and other occupational fields, from the
earliest phases of basic research to the logistical support and disposal of
old systems. However, there have been several definitions of what
comprises the DOD acquisition workforce, which have led to varying
estimates of the workforce size, ranging from about 106,000 to 355,000
people. At the end of fiscal year 1997, DOD estimated the acquisition
workforce covered by DAWIA included approximately 106,000 positions, of
which about 90,000 were civilian. Its most recent definition places the
workforce, now called the acquisition and technology workforce, at about
150,000 people2 and includes people from science and technology
organizations. Table 1.1 lists the different acquisition and technology
workforce occupations and the number in each occupation based on this
definition.




1
    P. L. 101-510, Nov. 5, 1990.
2
 This definition was based on the study, “Identification of the Department of Defense Key Acquisition
and Technology Workforce,” April 1999, Jefferson Solutions, Washington, D. C.




Page 17                                                         GAO/NSIAD-99-206 Best Practices
Chapter 1
Introduction




Table 1.1: The Defense Acquisition and Technology Workforce as of April 1999
Acquisition and technology workforce occupations                                Total persons
Engineers (electronics, aerospace, mechanical, civil, and general)                       44,117
Contracting                                                                              19,387
Management                                                                               15,509
Business and industry                                                                    12,989
Communications and computers                                                              9,370
Administration and programs                                                               5,116
Scientists                                                                                4,476
Auditing                                                                                  3,692
Financial management                                                                      3,618
Procurement assistants                                                                    2,650
Mathematics and statistics                                                                2,400
Purchasing                                                                                2,158
Supply program management                                                                 1,753
Inventory management                                                                        944
Equipment specialists                                                                       858
General supply                                                                              326
Miscellaneous                                                                             3,698
Military                                                                                 16,378
Total                                                                                  149,439
Source: DOD.


Definitions aside, the DOD acquisition workforce has been undergoing
restructuring and downsizing. The National Defense Authorization Acts for
Fiscal Years 1996 and 1997 mandated reductions in the number of civilian
and military employees in acquisition organizations. DOD estimates that,
as a result, the acquisition workforce will be 25 percent smaller at the end
of fiscal year 2000 compared to fiscal year 1995. DOD understands that
effective training will become even more important as the workforce is
reduced and new authority and responsibility are granted to those who
remain. We have previously reported that decisions to restructure or
reduce this workforce should be linked to getting better outcomes from the
acquisition process; doing otherwise would miss an opportunity to address
the deep-seated causes of acquisition problems. 3


3
 Defense Acquisition Organizations: Linking Workforce Reductions With Better Program Outcomes
(GAO/T-NSIAD-97-140, Apr. 8, 1997).




Page 18                                                     GAO/NSIAD-99-206 Best Practices
Chapter 1
Introduction




Although weapon system program offices comprise a subset of this
workforce, they are a highly leveraged subset. In planning, managing, and
executing acquisition programs, program management offices are
responsible for about $80 billion of DOD’s annual research, development,
and procurement funds—about 30 percent of the fiscal year 1999 DOD
budget. They influence much of the work of the rest of the acquisition
workforce because they are bringing new equipment into the inventory,
which must be managed, budgeted for, maintained, and supplied. In a
sense, they are a point of entry for DOD acquisitions. In addition, these
offices are made up of a cross section of people that draw from most of the
acquisition and technology workforce occupations cited in table 1.1. As
such, they are a medium in which the training of new initiatives in different
career fields converge.

The primary responsibility for training the acquisition workforce in
general—and new initiatives in particular—falls within the Office of the
Under Secretary of Defense for Acquisition and Technology. That office is
responsible for setting the standards that the workforce must meet. DAU,
which is responsible for designing and conducting the training to meet
those standards, also reports to that office. These and other organizations
responsible for setting training standards and providing training are shown
in figure 1.1.




Page 19                                         GAO/NSIAD-99-206 Best Practices
                                         Chapter 1
                                         Introduction




Figure 1.1: Organizations Responsible for Training Development




                                          Under Secretary of Defense for
                                           Acquisition and Technology




        Director for Acquisition Education,
                                                                    Defense Acquisition University
        Training, and Career Development



         Functional Boards                                        Consortium of Schools
         • Acquisition management                                 • Defense Systems Management
         • Contracting                                              College
         • Technical management (e.g.,                            • Industrial College of the Armed
            engineering and testing                                 Forces
         •Business, cost estimating, and                          • Service Schools
           financial management




                                         The Under Secretary of Defense for Acquisition and Technology delegated
                                         responsibility for developing career paths and establishing educational
                                         standards to the Office of the Director of Acquisition Education, Training,
                                         and Career Development. That office is supported by four functional
                                         boards that have established experience, education, and training standards
                                         for the acquisition workforce and for professional certification levels in
                                         each of the acquisition career fields. A DOD manual, DOD 5000.52M,
                                         “Acquisition Career Development Program,” implements and prescribes
                                         procedures for career development of the acquisition workforce. The



                                         Page 20                                        GAO/NSIAD-99-206 Best Practices
                         Chapter 1
                         Introduction




                         manual establishes experience, education, and training standards for 3
                         certification levels in each of the 11 acquisition career fields. Level I is
                         basic or entry level, level II is the intermediate or journeyman level, and
                         level III represents the advanced or senior level. The specific training to
                         increase competency or attain higher certification levels within a career
                         field is often referred to as functional training. There are also standards for
                         specific acquisition workforce positions (such as program managers),
                         position categories, and membership in the acquisition corps.

                         DAU was created by DAWIA to develop the training curriculum to meet
                         these standards and to coordinate the efforts of its consortium of 13
                         DOD-wide and service schools that conduct the training courses. The
                         Defense Systems Management College, a DOD-wide school, is dedicated to
                         providing acquisition-related training. Other member schools, including
                         those run by individual military services, provide a variety of training, of
                         which acquisition training is only a part. While the service schools provide
                         information on new initiatives, they do not have primary responsibility for
                         educating the acquisition workforce. Each service also has an acquisition
                         reform office that helps make people aware of the latest practices and
                         initiatives that apply to acquisitions, although this office does not play a
                         significant role in designing or providing the training. The Acquisition
                         Reform Communications Center is an organization related to DAU that has
                         the mission of sharing knowledge about acquisition reform by providing
                         and disseminating information on how DOD is changing the way it acquires
                         goods and services.



Objectives, Scope, and   The Chairman and the Ranking Minority Member, Senate Committee on
                         Armed Services, Subcommittee on Readiness and Management Support,
Methodology              asked us to review how DOD is training its acquisition workforce to
                         implement best practices for acquiring weapon systems. The
                         Subcommittee’s request is part of a broader interest in seeing that best
                         practices are incorporated into DOD’s acquisition process as a way of
                         saving money for modernization, increasing efficiency, and improving
                         quality. The objectives of this report are to assess (1) the contribution DOD
                         training makes to program offices that are applying best practices, (2) the
                         different training methods DOD and leading commercial firms use in
                         providing training on practices, and (3) the different strategic approaches
                         that underlie the training methods DOD and leading commercial firms use
                         in implementing practices.




                         Page 21                                           GAO/NSIAD-99-206 Best Practices
Chapter 1
Introduction




To meet these objectives, we focused on five specific practices and
identified program offices recognized as being leaders in applying them.
We chose the five practices based on discussions with DOD and on our
previous work in the application of best practices to weapon acquisitions.
The first four are formal DOD initiatives that are based on best commercial
practices, while supplier relationships is a best practice we observed in
leading firms and in some leading weapon system programs.4 The selected
practices are the following:

• Cost as an independent variable (CAIV): An acquisition management
  practice in which aggressive life-cycle cost goals are achieved through
  trade-offs between requirements and performance.
• Integrated product teams (IPT): Teams composed of members from
  functional disciplines such as engineering, test, and contract
  management. All members contribute their particular expertise to team
  decisions and to resolve issues.
• Performance specifications: States requirements in terms of required
  results without stating the methods to achieve those results. They
  define the functional requirement for operation, interface, and
  interchange characteristics, and have criteria for verifying performance
  compliance.
• Past performance: Information on a contractor’s past performance on
  relevant prior work is used as a factor in source selection. The
  information is used to evaluate risk and the potential for future
  contractor success.
• Supplier relationships: A commercial practice in which maximum
  participation of suppliers and their suppliers is encouraged to promote
  product excellence. The best suppliers are selected and supported in a
  number of ways that ensure a mutually beneficial partnership.

We focused on weapon system program offices because of the significant
role they play in implementing best practices. Initially, we considered
gathering information from a cross section of program offices that had a
range of experience in implementing best practices. However, we found
that a program office that had little or no experience with a new practice
was not in a good position to comment on the training needed to implement
the practice. Consequently, we selected program offices that DOD
considered to be leaders in one or more of the five practices.


4
  See Best Practices: DOD Can Help Suppliers Contribute More to Weapon System Programs
(GAO/NSIAD-98-87, Mar. 1998).




Page 22                                                     GAO/NSIAD-99-206 Best Practices
Chapter 1
Introduction




We based our selections on extensive consultation with DOD officials to
ensure that the program offices had experience in implementing the
practices and thus were in a good position to comment on the training
resources that enabled them to implement the practices. (See app. II for a
description of the programs.) These program offices represented best case
examples in DOD for specific practices, although none of the programs was
considered a leader in all five practices. The program offices selected were
the Advanced Amphibious Assault Vehicle (AAAV), Advanced
Medium-Range Air-to-Air Missile (AMRAAM), Battlefield Combat
Identification System (BCIS), Joint Air-to-Surface Standoff Missile
(JASSM), Joint Strike Fighter (JSF) and Joint Surveillance Target Attack
Radar System-Joint Tactical Terminal (JSTARS-JTT). Table 1.2 shows
which of the program offices were considered leaders for the different
practices.



Table 1.2: Best Practices Evaluated at Program Offices
                                                Performance          Past      Supplier
                            CAIV           IPT specifications performance relationships
AAAV                            X           X              X
AMRAAM                          X                          X
BCIS                            X                          X
JASSM                                       X                           X              X
JSF                             X                          X
JSTARS-JTT                      X           X
Source: GAO analysis of DOD information.


At the program offices, we used structured questions to interview the key
people responsible for implementing an individual practice. Through the
interviews, we determined the various sources they used to develop the
knowledge needed to implement best practices and the extent to which the
training DOD provided to the program office contributed to this
knowledge.

To assess strategy and the methods DOD uses to train program office staff
on the use of best practices, we concentrated on the DAU training
organizations and resources established by DAWIA to provide for the
professional educational development and training of the DOD-wide
acquisition workforce. We also assessed how these resources met program
office needs in implementing best practices. In addition, we assessed other



Page 23                                                   GAO/NSIAD-99-206 Best Practices
Chapter 1
Introduction




acquisition training sources, including service and agency specific schools
and the services’ acquisition reform offices. However, our report focuses
on the training approach for best practices and is not intended to describe
all DOD training practices. We met with the Director of Acquisition,
Education, Training, and Career Development; the President of the DAU;
officials of the Defense Systems Management College; and representatives
from the Office of the Under Secretary of Defense for Acquisition and
Technology. We also met with and collected data from officials at the Air
Force, the Army, and the Navy. We reviewed the organizational structure
and responsibilities for DAU consortium members and service
organizations, the process for establishing the DAU training curriculum,
and survey data on DOD training recipients.

We evaluated the strategies and methods of commercial firms recognized
for their training excellence by examining how these leading companies
used training to implement new practices at their program management
organizations. To identify these companies, we conducted literature
searches, consulted with and collected data from professional associations,
and spoke with university faculty specializing in corporate organizations.
We visited the following four companies recognized as being leaders in the
area of training:

• The Boeing Company is the largest manufacturer of commercial
  jetliners and military aircraft with 234,000 employees worldwide.
  Boeing’s Employee Training and Development organization is
  responsible for training all Commercial Airplane Group employees,
  approximately 97,000. There are 500 to 600 employees in the office, with
  roughly 300 serving as trainers.
• Ford Motor Company is one of the largest U.S. manufacturers of
  automobiles, trucks and provider of automotive services with 345,000
  employees worldwide. Ford’s Product Development Process
  Leadership organization was created to provide assistance, including
  training, to engineers implementing the Ford Production Development
  System (FPDS).
• International Business Machines (IBM) Corporation is one of the
  world’s top providers of computer hardware and software with 290,000
  employees worldwide. Different internal organizations provide training
  that serves employees worldwide, including Learning Services and the
  Center for Excellence.
• Motorola is one of the world’s leading providers of wireless
  communications, semiconductors, advanced electronic systems, and
  services with over 150,000 employees worldwide. Motorola University



Page 24                                        GAO/NSIAD-99-206 Best Practices
Chapter 1
Introduction




   has a staff of over 600 operating through account managers that are
   assigned to each business unit to provide a “one-face” education and
   training contact.

These companies, recognized as industry leaders, place strong emphasis on
training. At these companies, we reviewed company documents and
training data and met with individuals responsible for designing and
developing programs to educate and train employees on major new
practices. We also met with representatives from major program offices
who were involved in key training decisions and were recipients of the
training. Our report highlights the best commercial training approaches for
implementing key new practices. As such, they are not intended to
describe all commercial training practices or suggest that commercial firms
are without flaws.

Finally, we reviewed several studies on DOD’s training organizations and
methods. We used these, as well as the previous information and analysis,
to determine the extent to which DOD’s proposals to reshape DAU and
continuous learning policy held potential for better delivery of training to
foster implementation of best practices by program offices.

We conducted our review between April 1998 and June 1999 in accordance
with generally accepted government auditing standards.




Page 25                                         GAO/NSIAD-99-206 Best Practices
Chapter 2

DOD Training Is Not a Major Catalyst for Best
Practices                                                                                               Chapte2
                                                                                                              r




                         Key officials from weapon system programs at the forefront of
                         implementing best practices did not find that standard DOD training
                         offerings provided the information they needed to apply the practices to
                         their programs. In evaluating their key sources of knowledge for
                         implementing best practices, none of the program officials ranked required
                         DOD training first, with many ranking it last. DOD training either did not
                         reach the right people when it was needed or did not reach them at all.
                         When training on best practices was received, it did not contain the depth
                         or practical insights program office staff needed to implement the
                         practices. It was primarily through their own efforts—learning on the job,
                         finding external training, or developing their own training program—that
                         they attained the knowledge needed to apply best practices.

                         Programs that became leaders in applying best practices did so primarily
                         because their managers realized that the practices were key to the
                         programs’ success. In so doing, the managers were able to identify what
                         knowledge they needed to apply the practices. Thus, their success
                         depended on having the foresight to see what was needed, the ingenuity to
                         find good sources of knowledge, including training, and the resources
                         needed to attain that knowledge. Replicating this approach broadly on
                         other programs is problematic. Other managers may not realize the
                         significance of a practice to the success of their programs and the need for
                         additional training. Also, they may be uncertain about testing new
                         initiatives on their programs. Some may recognize a practice’s importance
                         but be unable to fund their own training efforts and be left relying on
                         standard DOD training.



DOD Training Did Not     For training to facilitate the adoption of a new practice, it must be received
                         by those responsible when they are ready to begin implementing the
Reach the Right People   practice. Program officials stressed that on a new initiative or practice,
at the Right Time        training must begin when a practice is to be introduced. If training is
                         provided too late, people will be forced to devise their own means of
                         getting the knowledge needed to begin implementation or risk improper
                         implementation. If training is provided too soon, knowledge could fade
                         before it is applied at the workplace. For the programs we reviewed, those
                         responsible for managing or implementing best practices noted that in
                         some cases, training did not exist when they began implementing the
                         practices; in other cases, the training was received too late for the job at
                         hand. Some were missed altogether by DOD training offerings because
                         they were too senior to be required to take courses, training was not




                         Page 26                                          GAO/NSIAD-99-206 Best Practices
                             Chapter 2
                             DOD Training Is Not a Major Catalyst for
                             Best Practices




                             provided locally, or they fell outside the definition—and training
                             curriculum—of the acquisition workforce.


Training Came at the Wrong   Program office officials believed it is essential that training on new
Time for Some People         initiatives accompany implementation to explain what the initiatives are
                             and to come up with a common understanding and way to tackle
                             program-specific issues. For the program offices leading the way in best
                             practices, officials reported relevant DOD training was not offered at the
                             time implementation began. For example, the BCIS program applied
                             performance-based specifications soon after the 1994 DOD initiative was
                             announced. However, specific training on performance specifications had
                             not been developed. The AAAV program had already implemented
                             performance specifications without DOD provided training. A top manager
                             for JSF similarly reported that there was no training for program office
                             staff when they started to implement CAIV. There were no guiding
                             documents and no one, including the training community, knew what CAIV
                             was. Similarly, on the JSTARS-JTT program, CAIV training was not
                             available when the program office began applying the practice in 1995.
                             Now, some training is available, but it was not when it was needed for the
                             program. Nor was training available when the JASSM program office
                             began assessing contractor past performance. Training could have helped
                             avoid a protest of the prime contract award, according to an official.




                             Page 27                                         GAO/NSIAD-99-206 Best Practices
                  Chapter 2
                  DOD Training Is Not a Major Catalyst for
                  Best Practices




Figure 2.1: JSF




                  Implementation of CAIV on the JSF program began with a clean slate—no advance training.
                  Source: DOD, artist rendition.


                  Training can also come too early. A program official questioned the staff’s
                  ability to retain the information when training is not provided at a practical
                  time for the assignment. For example, he noted that symposiums are good




                  Page 28                                                     GAO/NSIAD-99-206 Best Practices
                             Chapter 2
                             DOD Training Is Not a Major Catalyst for
                             Best Practices




                             ideas, but people may not have an opportunity to apply the ideas at the time
                             and may not remember when they need to.


Key People Were Missed by    Applying best practices on a weapon system program involves reaching not
Training on Best Practices   only program office staff but also other members of the acquisition
                             workforce, DOD people outside the acquisition workforce, and
                             contractors. Experience on the programs we reviewed shows that it is
                             hard to reach all of these people with best practices information through
                             standard DOD training.

                             The promotion of many best practices took place after the implementation
                             of DAWIA course requirements so that senior people have not taken formal
                             acquisition training that includes exposure to best practices. Several of the
                             senior program officials we spoke with did not receive training on new
                             practices because they had not taken courses that incorporated best
                             practices. These officials had been acquisition officials for many years
                             before DAWIA and were grandfathered into their certification level. Their
                             job experiences were applied to meet professional certification
                             requirements and thus they did not have to take any DAWIA training. For
                             example, an official who helped establish JSF performance specifications
                             noted that he had his last class 3 years earlier, a contracting refresher
                             course. The course may have had a short section on performance
                             specifications, but he could not remember it.

                             Training misses some program office people because it is not offered
                             locally. According to program officials, training needs to be conducted at
                             the local level by subject matter when it is needed. This is especially true
                             for small programs, for which it is difficult to spare people from the
                             programs for long periods. One official from a small program said that
                             although senior management stresses the importance of training, it is
                             difficult to attend classes that run over 14 weeks. He added that these
                             classes have not been offered locally. Another program official asked why
                             DOD training organizations do not bring some of the harder to get courses
                             to the field, as is done by private organizations.




                             Page 29                                         GAO/NSIAD-99-206 Best Practices
Chapter 2
DOD Training Is Not a Major Catalyst for
Best Practices




Figure 2.2: JSTARS-JTT




JSTARS-JTT officials believe training could be improved by being provided on-site.
Source: DOD.


Training is not reaching people outside program offices that also play key
roles in successfully implementing best practices. For example, program
offices are typically supplemented with people from separate functional
organizations, such as engineering directorates. These people may not
receive training on new initiatives in their home organizations and can be
unfamiliar with the initiatives when they come to a program. People that
set requirements early in the process and those in logistics must also be
knowledgeable and committed for practices to be successful. Exercising
flexibility in requirements, for instance, is critical to the success of CAIV
and performance specifications. The need for training was extended
further to those that play a role in a weapon system’s approval. Officials
from one program reported that they had worked with a prime contractor
to streamline contract reporting requirements down to four items, in line
with acquisition reform. However, based on a review by a separate office
with approval authority over the contract, 40 contract requirements were



Page 30                                                        GAO/NSIAD-99-206 Best Practices
                      Chapter 2
                      DOD Training Is Not a Major Catalyst for
                      Best Practices




                      added, returning the contract to traditional reporting methods. The
                      program official said that the reviewing official did not know what the
                      program office was trying to accomplish and did not care.

                      Prime contractors and subcontractors are also essential to the application
                      of best practices but are not part of the DOD defined acquisition workforce
                      or otherwise targeted for training on best practices. Several program
                      officials we met with believed that some means of informing contractors is
                      needed to make them aware of important DOD initiatives. Contractors do
                      not necessarily understand or know how to implement new DOD
                      initiatives; for example, they may not know how an integrating IPT or
                      overarching IPT works. While DOD is not responsible for training
                      contractors, it does have to ensure that contractors understand best
                      practices, as well as give them an opportunity to help shape how these
                      practices are applied on programs. The AAAV program management
                      ensured their prime contractor was knowledgeable of best practices by
                      requiring training for contractor and program office staff in the prime
                      contract.



DOD Training          Program officials reported that standard DOD training did not prepare
                      them well for implementing the practices at the workplace. DOD’s training
Offerings Did Not     typically provided only an awareness of the practices, not the knowledge
Provide the Depth     that is needed for actual implementation. Also, the training was not
                      tailored to allow program offices to see how individual practices applied to
Needed to Implement   their specific programs. Further, program officials noted that trainers did
Best Practices        not have the practical experience to share and were not up to date with the
                      most recent examples of programs that had implementation experience.

                      Program officials stated that they need to go beyond the theoretical
                      concepts covered in most training courses. They believed that the “how to”
                      is missing on all the initiatives and that they need to know how to move
                      from traditional practices to the new practices. DOD courses were
                      described in general as too esoteric and not relevant to the tasks at hand.
                      For example, a JASSM official noted that a contractor’s past performance is
                      often equated with DOD’s Contractor Performance Assessment Rating
                      System, but it involves much more. The JASSM program office evaluated
                      contractors’ past products, and in doing so, learned about the quality of
                      their design, management, and production processes. Program officials
                      said they went well beyond the performance assessment rating system
                      covered in standard training courses. Nonetheless, they said that they
                      could not avoid a protest by the losing contractor. They believed they



                      Page 31                                         GAO/NSIAD-99-206 Best Practices
                     Chapter 2
                     DOD Training Is Not a Major Catalyst for
                     Best Practices




                     needed guidance on how to collect past performance information, which
                     was not covered by training.

                     Program officials believed, overall, that training should be designed more
                     for the customer because acquisitions are unique and that programs may
                     have different implementation issues based on program size, stage of the
                     acquisition, or type of weapon. A senior AAAV official said general courses
                     can provide information about individual practices, but to implement a
                     practice, the training must be tailored to the program. JSF and BCIS
                     officials concurred. Some program officials have observed that training
                     courses overemphasize the application of new practices for larger and
                     newer programs and believe that not enough coverage is given to teaching
                     how the new practices apply to programs that are smaller or older. For
                     example, officials noted that newer programs have an advantage in starting
                     with a clean slate and that training should also show how the practices
                     apply to older programs and the benefits to be obtained.



Figure 2.3: AMRAAM




                     AMRAAM officials believe training should help program managers in deciding how initiatives apply to
                     their particular circumstances.
                     Source: DOD.




                     Page 32                                                        GAO/NSIAD-99-206 Best Practices
Chapter 2
DOD Training Is Not a Major Catalyst for
Best Practices




Program officials also found the presentation methods for training on best
practices were not helpful in applying the practices in the workplace. They
believed that instructors lacked practical experience and current
information. For example, a JSF manager reported that the executive
course he took in 1997 on CAIV was not valuable because the instructors
had no experience with CAIV and did not know how to explain it. Another
program official thought that instructors were left in place too long and had
only old experiences to share. Officials noted that course material was
frequently out of date and sometimes incorrect. For example, one JASSM
program official attended an engineering course in which the instructor’s
answer on CAIV was wrong. Another said the acquisition reform example
the instructor used—from the official’s own program—was about 2 years
behind what was being applied on the program. The official brought up
more current examples of practices being used on the program, but the
instructor did not want to discuss them.

Training also suffered from limited use of case studies that would allow
students to see how an initiative in the abstract might apply to their own
programs. Officials from several programs added that current case
histories should show the application of acquisition principles in a program
context. Case studies would include the successes, as well as the pitfalls
and solutions. The case studies should be designed for the customer,
another said, and be applied at multiple levels. For example, programs
could benefit if training allowed students to play off the risks and benefits
of how new initiatives apply to their particular program. Real case
experiences help others visualize how practices could apply. BCIS officials
reported that during the presentation of their case study for the 1998
Acquisition Reform Week, they realized, as they explained what they did to
implement initiatives, that the audience gained better insight on the issues.
Another official said he shared his program’s experiences with those in his
class, but those in other classes would not have the benefit of his
experiences.

Program officials did not believe that there was an effective means for
providing feedback on the quality and usefulness of courses, such as the
need to update course materials, or course relevance. They would like to
see the students—the practitioners—have more impact on what
adjustments are needed to courses. The only means of feedback program
office people were aware of was the end of class survey form. However,
these surveys did not allow students to give in-depth feedback or ask the
questions that got to the larger issue of whether students’ training needs
were met.



Page 33                                         GAO/NSIAD-99-206 Best Practices
                             Chapter 2
                             DOD Training Is Not a Major Catalyst for
                             Best Practices




Limitations of               In evaluating their key sources of knowledge for implementing best
                             practices, none of the program officials ranked required DOD training first,
Standard Training Led        with many ranking it last. To obtain the knowledge needed to implement
Program Offices to           key practices, officials in leading programs developed their own solutions.
                             They had a vision of what they wanted to accomplish and they devised a
Develop Their Own            variety of methods, such as funding training beyond standard DOD
Training Solutions           offerings, creating their own internal training programs, and learning on the
                             job. Strong program managers, supported by the executives above them
                             and strong working relationships with their contractors, were key to the
                             implementation of the practices. Program managers of the leading
                             programs cautioned, however, that not all officials have the vision or the
                             resources to mirror this approach.


Implementation Success       All the leading programs had at least one element in common—strong
Tied to Vision and Support   leadership committed to implementing practices that would help their
                             programs succeed. Leaders were described as having vision and knowing
at All Levels
                             what they had to do to realize that vision. This included accepting the risks
                             for trying new approaches. For example, the original AAAV program
                             manager conceived of the unique approach of collocating the program
                             office and the contractor because he thought it was essential to making
                             IPTs work. IPTs became the backbone of the AAAV program.
                             Consequently, team training was viewed as so important that it was
                             incorporated into the prime contract as a requirement. Support for
                             collocation and the team approach came from top management of both the
                             contractor and the program office. According to program officials, it does
                             not make any difference how good the training is without management
                             support; junior people can come back from training with new ideas and
                             have them go nowhere if they do not get management support.




                             Page 34                                         GAO/NSIAD-99-206 Best Practices
                   Chapter 2
                   DOD Training Is Not a Major Catalyst for
                   Best Practices




Figure 2.4: AAAV




                   Training for AAAV was tailored to support contractor and program office collocation.
                   Source: DOD.


                   In turn, managers point to the support that they receive from their senior
                   management as an important factor in their ability to be successful. One
                   official described his manager as aggressive in his efforts to pave the way
                   for trying new practices. To do that, the senior manager said that he is
                   willing to go up against the established bureaucracy and provide cover for
                   his program managers to try new things. One program manager cited a
                   personal commitment to a reform-minded DOD official as part of his
                   motivation for making acquisition reform work. He said that with his
                   bosses’ support, he can do what he thinks is right on his program, such as
                   making past performance central in selecting a prime contractor. However,
                   he noted that other managers have not had that ability because they are not
                   supported when they propose doing things differently.

                   Program managers believe that a key element for adopting best practices
                   was developing trust or partnership with the contractor. An AMRAAM
                   manager believes the ability to make radical program changes has



                   Page 35                                                          GAO/NSIAD-99-206 Best Practices
                            Chapter 2
                            DOD Training Is Not a Major Catalyst for
                            Best Practices




                            depended on working closely with the prime contractor, and seeing that the
                            same relationship exists between the prime contractor and subcontractors.
                            The JASSM office organized its IPTs to mirror the structure of the
                            contractor’s IPTs, rather than along functional lines like most offices. A top
                            JASSM official credited the high level of trust between the program office
                            and the contractor, along with the program office’s creativity and
                            innovation, for enabling their use of new practices. This perspective is
                            essential to managing programs in today’s environment and to make teams
                            work, another official noted, but it is not taught in standard DOD training
                            offerings.

                            More managers would attempt to apply best practices if training
                            encouraged it, a program official said, particularly the training provided for
                            junior program managers. More general training is needed to support a
                            new program environment for adopting new practices that goes beyond the
                            current training emphasis on the mechanics of management and theory.
                            Program officials reported that new program manager training as recent as
                            1997 did not reflect the new approach they have practiced on their
                            programs. One believed that it was a lost opportunity for DOD to not
                            impress upon new managers their role in adopting new practices and the
                            potential benefits.


Leading Program Officials   With the support of senior management, program officials used a mixture
Innovated to Get Needed     of strategies—generally outside of standard DOD training offerings—to get
                            the knowledge they needed to apply best practices. A number of programs
Training
                            used their own resources, including cumulative job expertise of the staff
                            and personal research to learn how to implement new initiatives. For
                            example, JSF officials stated they learned how to apply CAIV by organizing
                            warfighters, engineers, and analysts together in a team. They learned as
                            they went, developing materials on their own. The AMRAAM program had
                            a young staff member gather information on CAIV from various sources
                            such as the internet and conferences. From that starting point, they
                            learned as they went along. In setting performance specifications, it was
                            the personal experience of AMRAAM officials, coupled with commitment,
                            that allowed the specifications to be set at a performance, rather than
                            detailed, level. Despite their own experiences, program officials did not
                            recommend the learn-as-you-go approach; everyone should not have to
                            reinvent the wheel.

                            Programs officials used their own program funds to go beyond standard
                            DOD training by sending staff to nongovernment training or to bring



                            Page 36                                          GAO/NSIAD-99-206 Best Practices
                    Chapter 2
                    DOD Training Is Not a Major Catalyst for
                    Best Practices




                    experts in. For example, frustrated with DOD training, the JASSM program
                    manager sends people to outside training, such as Harvard leadership
                    courses, to have them learn and grow beyond the basic training and
                    develop creative and innovative thinking. He has his staff take 80 hours of
                    general, nonfunctional training. Another manager said some of his best
                    training was from off-site sessions sponsored by JASSM that dealt with
                    people issues that were critical to making IPTs work. JSF officials took a
                    private sector course on performance-based specifications and used a
                    model developed by their contractor. BCIS officials hired an outside firm
                    to teach program officials on two occasions, both of which included a CAIV
                    component.



Figure 2.5: JASSM




                    JASSM officials sponsor training to develop staff leadership skills.
                    Source: DOD.


                    AAAV officials developed their own training program tailored to the
                    program’s work environment and training needs. A key characteristic of
                    the program is the collocation of government and contractor program



                    Page 37                                                           GAO/NSIAD-99-206 Best Practices
                               Chapter 2
                               DOD Training Is Not a Major Catalyst for
                               Best Practices




                               officials, making team dynamics an important factor. The prime contract
                               specified that teams must be used, and IPT training was charged to the
                               contract. The contractor hired a third party to develop a training program
                               tailored to the AAAV program, and program office, contractor, and
                               subcontractor staff were taught together, on-site as a team. Joint training
                               was used to establish a common culture for participants. Training was
                               given one time to the team initially and then to every new person. The team
                               training started with mandatory 10 hours of IPT training, with sessions
                               covering the AAAV program, trust, communications, and other team
                               dynamics. CAIV training was added as part of the training for
                               implementation within the team context.

                               The team approach to training was supported by other leading programs as
                               well. On the JASSM program, the manager reported that he just had his
                               entire office (about 30 staff) take 1 week of leadership training, which
                               included topics such as stress management, critical thinking, and
                               decision-making. JSF officials took the “train the trainer” approach and
                               developed a team of experts within JSF and then the experts trained
                               everyone that would be involved in implementing performance
                               specifications. The experts used multiple sources to train themselves, such
                               as published guidance and talking with their peers. They then developed
                               basic guidance for the staff, such as engineers, who needed a common
                               understanding on how to write requirements at a performance level.


Other Programs May Lack        Program officials reported that they were fortunate to have staff that could
Leading Programs’ Ability to   use their collective experiences to work through problems in implementing
                               initiatives. However, they noted that not all programs will have the same
Innovate
                               advantages. JASSM officials said top service officials handpicked program
                               officials for the program team because their personal characteristics
                               supported flexibility and creativity. Similarly, JSF officials reported that
                               the program office was staffed with multifaceted people, as well as strong,
                               senior management support and upfront money for training. They noted
                               that smaller programs may be staffed by junior officials with less
                               experience to draw upon or be unable to devote staff to research
                               information on how practices might apply to their situation. Limited
                               funding may also be an issue for some programs. As one program official
                               said, his program office could pay for external training, but for many
                               programs, the cost would be an impediment since training is one of the first
                               items cut in a program budget. Consequently, smaller programs might need
                               to rely more heavily on standard DOD training as their main source of
                               information.



                               Page 38                                         GAO/NSIAD-99-206 Best Practices
Chapter 3

DOD Training Methods for Best Practices Do
Not Go as Far as Leading Commercial Firms                                                             Chapte3
                                                                                                            r




                       Leading commercial firms and DOD apply different training methods to
                       implement key practices. Commercial firms use targeted, hands-on
                       methods that include conducting a front-end analysis to determine the
                       teams’ training requirements and regularly involve those implementing the
                       practices in making important training decisions to ensure program teams
                       are trained on key practices. Thus, the training is customized to meet the
                       specific needs of the teams. These methods also involve providing many
                       hours of training—beyond standard skill-based or functional training—
                       focused on the implementation of a single practice. Company officials
                       believe the targeted approach results in more useful training—improving
                       the personnel coverage, course depth, and timing—which helps to improve
                       outcomes of the final product.

                       In contrast, DOD training on best practices is delivered through traditional
                       DAWIA certification courses and vehicles such as videos and
                       computer-based training, which are limited in reaching the right people at
                       the right time and in providing the needed depth to implement best
                       practices. DOD does not have a counterpart to the commercial method of
                       providing customized, hands-on assistance to support program office staff
                       and other implementers of key practices. Although exceptions exist, there
                       is no systematic effort—or responsible organizations—within DOD to
                       directly assist key implementers to use new practices. Further, DOD does
                       not have a comprehensive means for allowing program staff and others to
                       influence training decisions in a way that could improve the relevance of
                       training.



Commercial Firms Use   For routine training, such as skill-building, leading commercial firms
                       provide standard training offerings, including functional area courses and
Targeted, Hands-on     instruction on corporatewide issues, such as communications or ethics.
Methods to Improve     However, when implementing key new practices—such as those that
                       change product development and production—the firms go beyond the
Training Usefulness    standard training offerings. Commercial firms use a targeted, hands-on
                       training approach to ensure program teams are in a good position to
                       implement a new practice. They provide numerous hours of training,
                       typically through a single company organization, targeted to the
                       implementation of a key practice. The practice-specific training hours are
                       targeted to the program teams most likely to implement the new practice.




                       Page 39                                         GAO/NSIAD-99-206 Best Practices
                     Chapter 3
                     DOD Training Methods for Best Practices Do
                     Not Go as Far as Leading Commercial Firms




                     The elements common to how the four leading firms provide training on a
                     key initiative or practice include

                     •   a front-end analysis of program teams’ needs and training requirements;
                     •   involvement of program teams in key training decisions;
                     •   customized training to meet program teams’ specific needs;
                     •   targeted training for the implementation of specific practices; and
                     •   improved training outcomes, including better course depth, timeliness,
                         and reach.

                     The training organizations of leading commercial firms conduct a front-end
                     analysis to determine the needs and training requirements of program
                     offices implementing new practices. The analysis is also used to identify
                     and address barriers each program office faces when implementing new
                     practices. According to the Director of the Benchmarking Forum for the
                     American Society for Training and Development, this type of analysis is
                     crucial for an organization to be able to institute performance-improving
                     measures. Using information from the front-end analysis, the training
                     organizations customize the training to ensure that it directly assists
                     program teams in implementing new practices. One company official told
                     us that training is costly and when it misses the mark, the company pays a
                     big price. Given the importance of training when implementing a key
                     practice, company officials believe that it is crucial to ensure that the
                     training is beneficial to the key implementers of the practice. To ensure that
                     the training will address the needs of the program teams, the training
                     organizations involve the staff in making important training decisions.
                     Program staff help decide the amount of training to be provided for certain
                     job positions, course objectives, and depth of course coverage. Company
                     officials believe their training approach, which includes program staff, has
                     resulted in the right amount of course depth, timeliness, and coverage of
                     personnel in the commercial firms. Following are descriptions of the
                     training methods employed by companies on key initiatives or new
                     practices.

The Boeing Company   Officials from Boeing’s Employee Training and Development organization
                     state that their primary goal is to support their customers—employees
                     assigned to the Commercial Airplane Group. The training representatives
                     develop a partnership with the staff from the beginning of the program to
                     design and manufacture a new airplane. The training representatives form
                     “drop teams” to collocate with the program to conduct a front-end analysis
                     and learn as much as possible about the business process and the staff’s




                     Page 40                                          GAO/NSIAD-99-206 Best Practices
Chapter 3
DOD Training Methods for Best Practices Do
Not Go as Far as Leading Commercial Firms




concerns. The analysis allows the drop team to determine what training is
needed to support the staff implementing new practices.

Boeing training officials said they worked side by side with the program
staff to create a training program that provided team building and conflict
resolution techniques and technical skills training that specifically focused
on improving work competencies that would change as a result of the 777’s
new digital environment. To ensure all 777 staff were equally trained,
employees were required to complete training before they reported to the
program. For example, the professional employees—engineers and
drafters—were required to complete 120 hours of start-up training on
several key 777 practices, including design build teams and computer-aided
three-dimensional interactive application1 software. Teams were often
trained together at the work location. Boeing officials stated that training
was instrumental to the implementation of key practices on the 777
program, such as design build teams—essentially IPTs. The officials stated
that design build teams were at odds with the company’s culture because
employees were not accustomed to working in a team environment and
sharing information across functional areas.




1
 This application is a computer-based design tool that allows designers the opportunity to view design
drawings and the interface of the millions of airplane parts as three dimensional.




Page 41                                                         GAO/NSIAD-99-206 Best Practices
                           Chapter 3
                           DOD Training Methods for Best Practices Do
                           Not Go as Far as Leading Commercial Firms




Figure 3.1: 777 Airplane




                           Boeing’s use of design build teams created a major culture change for the staff assigned to the 777
                           program.
                           Source: The Boeing Company.


                           Boeing officials believe their partnership approach improved their training.
                           Training representatives stated that the partnership resulted in program
                           management support, which ultimately led to acceptance from the program
                           staff. The representative stated that since Boeing has involved program
                           staff in decisions regarding training, course “no-show” rates have
                           decreased. A senior manager for program operations for the 777 program
                           stated that because he and other senior program leaders drove key training
                           decisions, the training was tailored to the staff’s needs and provided the
                           necessary skills and orientation to work in the new environment.




                           Page 42                                                         GAO/NSIAD-99-206 Best Practices
                     Chapter 3
                     DOD Training Methods for Best Practices Do
                     Not Go as Far as Leading Commercial Firms




Ford Motor Company   The Ford Motor Company created an organization, Product Development
                     Process Leadership, with the singular purpose of supporting its 100
                     program offices in designing new or modifying vehicle lines in the
                     implementation of the FPDS--a lean engineering process. The organization
                     provides internal communication regarding FPDS to the teams, and its
                     training representatives work with the teams to conduct an analysis to
                     learn first hand if impediments to FPDS implementation exist. Other
                     support, such as team coaching, is provided to facilitate the engineering
                     team’s implementation of FPDS in the workplace.



                     Figure 3.2: Ford Focus




                     Ford uses training to improve the timeliness and quality of new vehicle launches.
                     Source: Ford Motor Company.


                     Ford officials told us that their training focus is to provide practical skills
                     just in time—that is, when it coincides with the need to apply the skills on



                     Page 43                                                         GAO/NSIAD-99-206 Best Practices
                      Chapter 3
                      DOD Training Methods for Best Practices Do
                      Not Go as Far as Leading Commercial Firms




                      the job. Ford provides over 50 hours of training to instruct engineers on
                      how to do their jobs using the FPDS process. To that end, Ford uses
                      internal subject matter experts—engineers who have been on teams
                      designing new or modifying existing vehicle lines—to help tailor FPDS
                      course topics to make them relevant to the work environment. Ford
                      officials stated that the subject matter experts understand the details of the
                      FPDS and are needed to ensure that the training developed is practical at
                      the working level. According to the manager responsible for training
                      employees to launch new vehicles at Ford, his office pulls together the
                      training that is necessary to get the job done. To further improve
                      practicality and timeliness, the manager stated that on-the-job-training at
                      the worksite is used to the extent possible.

IBM                   IBM’s Center for Excellence provides in-house education consultants to
                      personally understand the business situation and product. The consultants
                      interview business unit staff to assess the staff’s training needs and identify
                      inhibitors to implementing new practices. The assessments can take
                      2 weeks to 4 months, depending on the size of the business unit. An IBM
                      trainer for Object Orientation Project—a companywide software
                      development practice—explained that providing practical training is one of
                      the Center’s guiding principles. To achieve this goal, the company first
                      stresses the importance of using experienced instructors who have
                      practical knowledge in “doing” what they are teaching. Second, employees
                      are assigned to an Object Orientation Project before they take the
                      5- to 7-week immersion training course. According to the IBM trainer, this
                      requirement has improved practicality because the students have better
                      knowledge retention, as opposed to having to wait 6 months to apply the
                      information. Lastly, to further enhance practicality and relevance, IBM
                      integrates case studies with real examples related to the student’s next
                      assignment into the Object Orientation Project training. According to an
                      official from IBM’s Center of Excellence, most training is provided and
                      tailored to entire work teams or, at a minimum, to individuals with common
                      responsibilities.

Motorola University   Motorola University focuses on providing training and education solutions
                      for its business units. The university recently began to assign an account
                      management team to consult and advise senior leadership for each
                      business sector in order to anticipate and provide appropriate training.
                      The management team partners with the business unit staff to identify their
                      training needs for implementing new practices, such as Five Nines—an
                      effort to improve product reliability to the level of 99.999 percent. The
                      account team works with the business units to assess their training needs



                      Page 44                                           GAO/NSIAD-99-206 Best Practices
                            Chapter 3
                            DOD Training Methods for Best Practices Do
                            Not Go as Far as Leading Commercial Firms




                            and develop a plan to meet those needs. This partnership enables the
                            university to customize its training to the specific needs of the various
                            types of engineers, such as software engineers, within the business units. A
                            software engineer stated that the university provides separate training
                            courses tailored to meet the different and often distinct needs within the
                            engineering community implementing Five Nines. For example, a software
                            engineer could receive up to 120 hours of training targeted to implementing
                            Five Nines. Although the account management structure is relatively new,
                            the engineer observed that most business unit staff are reacting favorably
                            to the effort because the coordinated approach provides one-stop
                            shopping.



DOD Does Not Target         DOD does not have a counterpart to the commercial hands-on approach for
                            directly assisting key implementers of the best practices. DOD relies
Training on Key             primarily on its standard training, including DAWIA courses, augmented by
Practices to Program        videos, internet-based training, satellite broadcasts, and roadshows, to
                            inform staff on best practices. These venues were designed to focus on
Offices                     functional training and to increase the awareness of new practices. As
                            such, they do not provide the necessary depth or reach enough of the right
                            people at the right time to be of help in implementing best practices at
                            program offices. Responsibility for training on best practices is diffused
                            among several DOD organizations, including DAU and the service
                            acquisition reform offices. We did not find an organization that was able to
                            tailor and help deliver training on best practices to the program offices we
                            visited. Furthermore, DOD does not systematically involve program office
                            staff and other implementers in key decisions regarding best practices
                            training. Currently, no feedback mechanism exists to determine the effect
                            of DOD’s training on the implementation of new practices at the program
                            office level.


DAWIA Training Not Well     DAU training is designed primarily for employees seeking career level
Suited as a Primary Means   certification in the acquisition workforce, as required by the DAWIA
                            standards. DAU incorporates best practice topics into the DAWIA courses
for Conveying Best
                            as drop-in modules that provide a survey of the topic. The information
Practices                   conveyed is enough to provide a general awareness of the concept but not
                            enough to implement the practice at the workplace. While this approach
                            may provide sufficient information for the target audience—newer
                            acquisition employees—it has inherent limitations when it comes to
                            providing best practices’ implementation training to the entire acquisition
                            workforce. For example, DAU courses have been developed by functional


                            Page 45                                         GAO/NSIAD-99-206 Best Practices
Chapter 3
DOD Training Methods for Best Practices Do
Not Go as Far as Leading Commercial Firms




boards that teach skill-based competencies for functional career fields
versus best practices. As a result, DAU courses are primarily aligned along
career fields, such as engineering and cost estimating. Because personnel
assigned to a particular functional area are given priority for training in that
function, program officials told us that it is difficult for personnel outside
of their areas to take courses. On one program, an official noted that
because of limited space, only 5 to 10 percent of the program staff can take
training outside of their functional areas each year.

One program manager believed that the functional training, while
important to career fields, was no longer as relevant to the role of today’s
program office. He noted that the combination of best practices,
delegation of key responsibilities to contractors, and fewer staff has altered
the program office’s role. In his opinion, the program office used to be
closely involved with managing the design of a weapon system and
double-checking the prime contractor’s work. In this role, the program
office was organized and operated along functional lines, and functional
training was relevant to how a program office operated. He observed that
today the program office is not as involved with the designing of the
weapon system, nor is it able to mirror the contractor’s functions. Rather,
the program office must be expert at what the government can control—
which the program manager referred to as key leverage points. These
leverage points include the requirement trade-off process, the selection of a
prime contractor, and the establishment of key relationships that enable
the program office staff to have insight into the contractor’s actual
progress. Program office staff, working in an IPT environment, must have
knowledge of multiple career fields and work in a cross-functional setting.
As a result, he believes that functional training alone no longer covers the
things most critical to a program manager. The kind of training that is
needed—such as on the leverage points—must be obtained elsewhere by
the program office.

The usefulness of the DAU courses is further hampered by limited
availability, which restricts program offices from receiving training when
needed. According to a DAU school representative, the consortia of
schools can train about 10 percent of the workforce each year.
Furthermore, more senior staff may have limited exposure to best
practices because the majority of them have already met training
requirements and are not taking the certification courses that introduce
best practices. While these staff are not prohibited from taking
certification courses as part of their continuing education requirements,
availability is limited and priority is given to individuals seeking



Page 46                                           GAO/NSIAD-99-206 Best Practices
                               Chapter 3
                               DOD Training Methods for Best Practices Do
                               Not Go as Far as Leading Commercial Firms




                               certification. These inherent limitations are consistent with the training
                               shortfalls noted by program office staff in chapter 2.


Other Training Methods Are     DAU’s Acquisition Reform Communication Center (ARCC) is a key avenue
More Dedicated to Best         for disseminating information on the best practices to the acquisition
                               workforce. ARCC provides training through videos and periodic satellite
Practices but Are
                               broadcasts on a variety of best practice topics. However, ARCC does not
Awareness-oriented             track attendees and has no assurance that the workforce adequately
                               receives the training. For those that do attend, the introductory nature of
                               the training may not provide the depth or specificity to implement the
                               practice at the workplace or in a time frame that is helpful.

                               The acquisition reform offices in the services communicate best practice
                               information through acquisition reform courses, periodic satellite
                               broadcasts, and informational videos, which are sufficient for broad
                               exposure on best practices but again are of limited depth for practical
                               application. Roadshows, traveling multiday training workshops provided to
                               staff at a number of locations, and Acquisition Reform Week, designated by
                               DOD as an opportunity for all service organizations to cease their normal
                               operations and focus on acquisition and logistics reform initiatives, are also
                               used to provide best practices training to a wide range of the acquisition
                               workforce. These methods also have limitations in depth and workforce
                               coverage. For example, roadshows typically provide awareness training on
                               the practices and do not provide in-depth information needed for
                               implementation at the workplace. A program official believes that only
                               10 to 15 percent of the acquisition staff attend the second day of roadshow
                               workshops, where more detailed training is provided. The annual
                               Acquisition Reform Weeks also provide awareness level training. Neither
                               method is tailored to specific program offices or provides assurance that it
                               is delivered at the time most needed by the workforce.


DOD Training Organizations     DOD does not have organizations that are comparable to those in
Not Set Up to Help Design      commercial firms and that work with program offices in identifying best
                               practices applicable to a particular program, designing an approach to
or Deliver Tailored Training
                               training the program office staff, or delivering the training to the program
to Program Offices             offices. DOD training organizations—those that deliver DAWIA and
                               acquisition reform training—are not set up or have the resources to deliver
                               best practices quickly and easily to program staff, and then ensure
                               implementation at the working level.




                               Page 47                                          GAO/NSIAD-99-206 Best Practices
Chapter 3
DOD Training Methods for Best Practices Do
Not Go as Far as Leading Commercial Firms




The DAU consortium integrates best practices topics into DAWIA
certification or acquisition reform training but does not develop training
specifically for implementing the practices. Although DAU has a
substantial full-time faculty, the faculty members’ main priority has been to
teach in the classroom. One course director informed us that faculty
members are evaluated on the basis of hours of training provided in the
classroom, which implicitly discourages work outside the classroom, such
as consulting with program offices. Also, little or no consultation occurs
between the course designers and the implementers on how to best
implement a practice at the program office level. The Director of DOD’s
International and Commercial Systems Acquisition unit stated that DOD’s
current structure for defining and developing training courses does not
have clear accountability to ensure that training on best practices is
provided at the program level. Several program office staff informed us
that they have not been given the opportunity to provide input to influence
the content of the courses. Instructors for the courses, who do have
frequent contact with program office staff, also believed that their ability to
influence the content of the courses was limited. DOD does use internet
site surveys to obtain staff feedback on acquisition reform training, which
could help tailor the courses more to the needs of the program offices.
However, program office staff believed the survey forms were inadequate
for addressing the specific training needs of individual program offices.

Organizations with more direct responsibility for providing training on best
practices are staffed to the level needed to design distance learning
courses, such as web-based training, satellite broadcasts, and instructional
videos. For example, the services’ acquisition reform offices each employ
from 8 to 20 people, not nearly enough to provide on-site training or
in-depth consulting to individual program offices. Some DOD organizations
have recognized the value of providing training tailored to specific program
offices and are attempting to go beyond traditional training roles.
However, these are largely ad hoc efforts that do not go as far as
commercial methods. For example, the Navy is creating Total Ownership
Cost teams, that are to advise program offices on the use of CAIV. Navy
officials stated that due to limited resources, the teams will only reach
about 15 of the 300 program offices per year. The Air Force plans to begin
using training support teams to address best practices issues from the
program office perspective. However, the teams have not yet begun and
full operation is not scheduled to begin until fiscal year 2001. Also, an Air
Force official stated that another team, created to address acquisition
reform training issues, will not involve program managers, which raises the
question of whether program office needs will be adequately considered.



Page 48                                           GAO/NSIAD-99-206 Best Practices
Chapter 4

DOD and Commercial Training Methods
Reflect Different Strategic Approaches to New
Practices                                                                                       Chapte4
                                                                                                      r




               The intensive, hands-on training methods leading commercial firms employ
               on new practices are the result of a systematic, institutionally driven
               approach to implementation. These firms commit to and concentrate their
               resources and attention on a few well-defined practices and make a
               significant front-end investment in providing the training to the primary
               implementers. Additionally, the firms strive to create a supportive
               environment to put the implementers of the practice in a good position to
               succeed. Their objective is to have successful implementation at the work
               level; training methods are shaped to meet that objective. DOD’s training
               methods for best practices do not benefit from a similarly strategic
               approach for deploying new best practices and provide little assurance that
               the practices are consistently implemented in the workplace. DOD has
               promulgated numerous initiatives in the past few years without
               communicating their relative priority to trainers and implementers. Often,
               the initiatives were not accompanied by clear guidance or by the initial
               training needed for implementation. While DOD commits significant
               resources to training, it does not make a uniform front-end investment to
               ensure the implementers of new practices will succeed. This approach
               depends more on the individual program offices to recognize the need for
               and to make this investment.

               Recent DOD actions reflect its recognition that training improvements are
               needed. Two plans proposing divergent improvements to the training of
               the acquisition workforce are now being considered. One plan does not
               focus on best practices training specifically, offering instead an incremental
               approach to changing the DAU structure. The other targets
               implementation of best practices directly, with a broad scope of efforts that
               would require substantial changes from the current approach to training.
               The latter plan is much more aligned with the commercial approach to
               training described in this report. However, if it is adopted, it faces many
               challenges to providing better training for program offices. Another recent
               DOD action—creation of a continuous learning policy—may provide more
               training, especially for more senior members of the acquisition workforce,
               but it does not ensure that best practices will be included.




               Page 49                                          GAO/NSIAD-99-206 Best Practices
                        Chapter 4
                        DOD and Commercial Training Methods
                        Reflect Different Strategic Approaches to
                        New Practices




Leading Firms’          Although the four companies we contacted initiated very different types of
                        practices, they exhibited a similar strategic approach to ensuring that the
Strategic Approach      key practices were implemented. The elements of this approach can be
Better Ensures          summarized as
Implementation of Key   • clear vision,
Practices               • adoption of few practices at any one time,
                        • assessment and identification of target population for implementation,
                        • well-defined goals,
                        • significant investment in training and other vehicles to aid
                          implementation, and
                        • supportive environment to facilitate implementation.

                        In general, we found that the firms shared a common strategy for
                        implementing key practices that were intended to change company culture.
                        First, the firms’ corporate management committed to and adopted seven or
                        less key practices at any given time. One firm we contacted adopted only
                        one or two key new practices, which enabled the company to concentrate
                        its attention and target resources, including training, to the implementation
                        of the practice, and signaled the importance of the practices to trainers and
                        implementers. Second, for practices that were not companywide, the firms
                        assessed which staff should be included in the implementation. For
                        example, Ford’s training unit assessed each existing program team working
                        on vehicle lines beyond the initial development phase to determine if the
                        team could benefit from adopting portions of FPDS or if some older teams
                        should be excluded. An IBM official stated that while the company
                        currently promotes six practices, not all will apply to every segment of the
                        employee population. Third, once the target population was identified,
                        company leaders made the implementation of the practice mandatory. For
                        example, the Ford Motor Company required that all new vehicle lines built
                        for the year 2000 and beyond use FPDS. Lastly, to further assist in the
                        implementation, the companies developed well-defined goals to better
                        ensure that the target population consistently understood how to apply a
                        new practice to improve product production—the ultimate goal.

                        The firms we contacted ensured that the implementers of the practice
                        received the assistance necessary to succeed. Consequently, these
                        companies made a significant investment, including a comprehensive
                        front-end effort, to support the needs of program offices that would
                        implement the practice. According to a Ford training official, if the
                        initiative is important enough to the organization, then the investment is



                        Page 50                                         GAO/NSIAD-99-206 Best Practices
                         Chapter 4
                         DOD and Commercial Training Methods
                         Reflect Different Strategic Approaches to
                         New Practices




                         justified. Other companies agreed with that way of thinking. Boeing’s
                         Learning Program Development Director summarized the corporate
                         training strategy for implementing new practices as one that includes a
                         clearly stated vision or mission statement, well-defined goals, and enablers,
                         such as training and good processes, to support the implementers. This
                         philosophy enabled Boeing to take a year to develop the training program
                         tailored to the 777 program—which was intended to change the corporate
                         culture and encourage employees to rethink how they did their jobs. Both
                         Boeing training and program officials believe that the training investment
                         resulted in the successful implementation of the key 777 program
                         practices.

                         While the company officials acknowledged that training was instrumental
                         in the implementation of the key practices, nearly every official we spoke
                         with stated that training is just one of the necessary components. They
                         noted that creating the right environment is key to the successful
                         implementation of new practices and that the quality of training was
                         dependent on this environment. Company officials stressed that strong
                         leadership is often another key. An IBM official stated that, at inception,
                         top leaders need to provide sufficient funding for training, well-defined
                         expectations, clear direction, oversight, continued interest, and incentives
                         to ensure that the new practices are possible to implement. The manager
                         for the 777 program stated that Boeing’s management works in teams—a
                         key practice. He believed that it was management’s ability to lead by
                         example that helped to prevent a return to the former functional way of
                         operating. These companies believe that other factors, such as an
                         accommodating organizational structure, good internal communication,
                         consistent application, and supportive technology, are needed to foster the
                         implementation of key new practices.



DOD’s Training on Best   DOD does not have a uniform or defined strategic approach to
                         implementing best practices. Accordingly, training methods are not shaped
Practices Does Not       by the same drive for implementation found in leading commercial firms.
Stem From a Strategic    DOD’s approach commonly begins with policy memorandums widely
                         distributed to the entire acquisition workforce. However, the policy
Approach                 provides little specific guidance on how or which program offices should
                         implement the practice. Since 1994, DOD has proposed close to 40
                         acquisition reform initiatives without an indication of relative priority. The
                         absence of priorities makes it difficult to not only develop training courses
                         but also to determine which courses are most appropriate for the needs of
                         program offices.



                         Page 51                                          GAO/NSIAD-99-206 Best Practices
                           Chapter 4
                           DOD and Commercial Training Methods
                           Reflect Different Strategic Approaches to
                           New Practices




DOD’s Policy on Best       DOD’s policy on implementing the best practices is promulgated without a
Practices Is Not Coupled   strategy or specificity as to whom within the acquisition workforce is
                           expected to implement the practices or which program offices are in the
With a Strategy for
                           best position, in a program life cycle, to implement the practices. Typically,
Implementation             the policy is not promulgated in conjunction with a detailed plan that
                           outlines the support and training required by the program offices tasked
                           with implementing the practice. Similarly, a 1997 Coopers & Lybrand study
                           on acquisition reform implementation found inconsistencies in DOD’s
                           implementation of new practices. It stated that these inconsistencies may
                           have resulted from a lack of an integrated, cohesive DOD strategic plan to
                           put them in context to each other and to larger strategic goals. While DOD
                           has made some progress in establishing such goals through its Government
                           Performance Reporting Act plans, organizations responsible for training
                           and implementing the practices have not experienced the effects.

                           In addition, the policies themselves are not always clear. For example,
                           although the CAIV initiative was promulgated in 1995, training officials
                           reported they did not have a clear understanding of what CAIV means.
                           AAAV program officials said they developed training for 1997 to clarify the
                           CAIV concept for the program. A November 1998 Air Force workshop on
                           CAIV reported that the concept is still not well understood or widely
                           implemented.

                           Similarly, the IPT policy states that IPTs should be implemented “when it
                           makes sense.” The policy for the use of past performance data states that
                           the practice should be implemented to the “degree possible.” While a fair
                           degree of latitude is good for implementers, some DOD officials believe
                           that this level of generality is insufficient to ensure implementation of the
                           practice. For example, a training official observed that policies are not well
                           defined because of the desire to bring new practices quickly to the
                           workforce and so some are deployed when they are not ready for
                           implementation. According to a DAU official, unclear and general policies
                           can contribute to ineffective training because it makes it difficult to
                           develop a course of instruction that teaches a unified, cohesive way to
                           implement a policy. A program official noted that general policy
                           statements provide little guidance or assistance in the program offices’
                           implementation of the practice, even though program offices are often the
                           primary implementers of best practices.




                           Page 52                                          GAO/NSIAD-99-206 Best Practices
                           Chapter 4
                           DOD and Commercial Training Methods
                           Reflect Different Strategic Approaches to
                           New Practices




DOD Attempts to            In addition to IPTs, performance specifications, CAIV, and past
Implement Many Practices   performance, DOD has proposed numerous reform initiatives since 1994.
                           In 1997, the Secretary of Defense proposed several new reform initiatives
Without Prioritization
                           under the umbrella of the Defense Reform Initiative. These initiatives have
                           been promulgated without an indication of relative priority, leaving
                           educators and implementers to decide on what is important. The lack of
                           prioritization makes it difficult to determine training needs, especially
                           when the number of initiatives continues to increase. While the prolific
                           offering of new ideas can stimulate innovation, the absence of priorities
                           makes it difficult to focus training on the specific initiatives that are the
                           most important to implement. Moreover, program management offices are
                           not necessarily in a good position to sort through the initiatives to focus on
                           those that are the most important to the job at hand.

                           Several DOD officials expressed concerns regarding the number of
                           initiatives without prioritization. For example, a DAU training official
                           noted that it is impractical to expect to train the entire workforce on all of
                           the initiatives and that the Office of the Secretary of Defense should set the
                           priorities for implementation of initiatives. A DAU consultant echoed the
                           view that there are too many new reform initiatives and that DOD needs to
                           set priorities on which initiatives to address. A service acquisition reform
                           official observed that the large numbers of best practices promoted by
                           DOD overwhelms the services. This official told us that the three services’
                           acquisition reform offices meet quarterly to share progress on acquisition
                           reform initiatives. He believes these meetings could serve as an
                           opportunity to prioritize the practices. However, the current focus of the
                           meetings is unclear and the offices’ charters are under review. Another
                           acquisition reform office representative stated that the combination of a
                           large number of reform initiatives and unclear priorities puts the office in
                           the position of having to guess at what is the most important. This leads to
                           emphasizing what is perceived to be popular, he added.

                           Other officials explained that due to the frequency of new initiatives being
                           introduced, training courses should be regularly revisited. However,
                           training personnel have not been able to consistently keep up with the pace
                           of change. For example, a service acquisition reform official stated that it
                           requires at least a full year to determine the elements of a new initiative and
                           then the challenge of how to train the initiative still exists. A 1997 DOD
                           study team that reviewed DOD’s acquisition education and training
                           structure and process found that curriculum development often lagged
                           behind the rapidly changing policy requirements. The study also noted that
                           instructors were not up to date on current acquisition policies and that



                           Page 53                                           GAO/NSIAD-99-206 Best Practices
                         Chapter 4
                         DOD and Commercial Training Methods
                         Reflect Different Strategic Approaches to
                         New Practices




                         because they were out of touch with DOD policy makers, the instructors
                         found it difficult to develop up-to-date training courses.



DOD Proposals to Alter   DOD is aware of the need to improve the means by which the acquisition
                         workforce receives and implements new initiatives. Two proposals being
Training Reflect         considered by the Under Secretary of Defense for Acquisition and
Conflicting              Technology that are intended to improve workforce training, and a new
                         continuing education policy has been put in place. The first proposal is a
Approaches               DAU transition plan for improving DAU’s structure. The plan is based on a
                         1997 study by a DOD team and a 1998 study by the Logistics Management
                         Institute. Both studies confirmed strengths in the DAU approach,
                         particularly relating to functional or skill-based training and the
                         weaknesses in some of the same areas that surfaced in our work. The
                         studies made recommendations for significant organizational and
                         operational changes in DAU to better meet the educational needs of the
                         acquisition workforce. However, a more moderate approach was adopted
                         in preparing the transition plan for DAU. While the proposed structure
                         offers improvements, it does not discernibly address key weaknesses in the
                         training of best practices. The plan has yet to be approved. The second
                         proposal, a report from the DOD Section 912(c) Commercial Business
                         Environment Group directly advocates incorporating best practices and is
                         much broader in recommending changes. It proposes that DOD’s
                         acquisition structure be transformed into a team type of organization with
                         members drawn from across the DOD procurement enterprise. A new
                         learning organization would be created specifically to support accelerated
                         change.

                         DOD has also instituted a continuous learning policy, which should help
                         people obtain current training, even if they are already fully certified.
                         However, the required levels of continuous learning can be met in many
                         ways and do not place any particular emphasis on training in best
                         practices. These DOD efforts are a step in the right direction for seeing
                         that program offices have the information they need, but none focus on the
                         program offices’ needs. Given the pivotal role they play in weapons
                         acquisition, a more direct approach may be required.




                         Page 54                                       GAO/NSIAD-99-206 Best Practices
                         Chapter 4
                         DOD and Commercial Training Methods
                         Reflect Different Strategic Approaches to
                         New Practices




DAU Transition Plan Is   The DOD transition plan for restructuring DAU was based on
Largely Silent on        recommendations of a 1997 process action team that DOD chartered to
                         create a clear vision and a structure for future workforce training and
Improvements to Best
                         education. Several findings of the study related to training for new
Practices Training       initiatives and practices, including (1) the existing curriculum design
                         process was functionally driven, did not address the needs of people
                         outside the functional career fields, and did not lend itself to successful
                         development of cross-functional curricula; (2) a mechanism was needed to
                         validate and prioritize requirements before they are submitted; (3) faculty
                         members did not incorporate up-to-date case studies; (4) DAU schools
                         used procedures for measuring performance that focused on student
                         satisfaction with a course, not whether the user community’s acquisition
                         education requirements were being met; and (5) DAU’s use of technology-
                         based learning was insufficient.

                         The team recommended that DAU be replaced with a unified Defense
                         Acquisition Institute that would have responsibility for development and
                         delivery of acquisition training, the consortium be streamlined to reduce
                         duplication, and lines of authority between DAU and the members be
                         strengthened. The team envisioned the institute as, among other things, (1)
                         fostering innovation and facilitating reform and continuous improvements,
                         (2) having a clear message of support from DOD acquisition leadership that
                         education is the key to meeting DOD goals, (3) identifying acquisition
                         training needs that are cross-functional areas or that are multifunctional in
                         nature, (4) focusing on the needs of the learner, and (5) employing a
                         process to validate the effectiveness of the training received. However, a
                         dissenting opinion offered by one of the team members to evolve the
                         current DAU organization structure through a series of near-term and
                         long-term actions was accepted by the Under Secretary of Defense.

                         Subsequently, the Logistic Management Institute conducted a review for
                         DOD and developed a structural reorganization for DAU that was to be
                         used as the starting point for the transition plan. The Institute noted that
                         DAU training was generally skill and specialty oriented, with limited
                         overlap among “stove-piped” career fields. It recommended that the new
                         structure be designed along the lines of a corporate university. As such, its
                         curriculum would include cross-functional training and business executive
                         development and act as an agent of change, being more involved with the
                         workforce in determining how changes would be carried out.

                         The DAU president submitted the draft transition plan for restructuring in
                         September 1998. As of June 1999, DOD had not approved the plan. The



                         Page 55                                         GAO/NSIAD-99-206 Best Practices
                            Chapter 4
                            DOD and Commercial Training Methods
                            Reflect Different Strategic Approaches to
                            New Practices




                            plan has several features that offer promise for more effective training
                            overall. These include (1) a unified structure in which DAU has full
                            responsibility for ensuring that the overall course structure meets the
                            needs of the acquisition workforce, (2) consolidation of the consortium
                            schools into four main campuses under the direct authority of DAU, (3) an
                            increased emphasis on distance learning techniques, (4) steps to improve
                            the qualifications of faculty, and (5) replacement of functional boards with
                            functional IPTs that have representation from both functional experts and
                            instructors and that will design curricula together.

                            Other features of the plan raise doubts about its ability to provide a
                            strategic approach to shape training that will help implementation of best
                            practices at program offices. The transition team that prepared the plan
                            concluded that DAU was unique and could not be matched with other
                            institutions, like corporate universities. Thus, the corporate university
                            model recommended by the Institute was not adopted. While the
                            curriculum may benefit from the functional IPTs, the plan does not mention
                            the need for cross-functional or best practices training or for DAU to
                            accept the role of a change agent. It is silent on prioritizing initiatives or
                            improving feedback. While it is possible that under the proposed plan DAU
                            could play a more active role in the design and delivery of tailored training
                            to the workplace, the plan does not that suggest that, other than increased
                            distance learning, training will be offered in a manner substantially
                            different from what has been traditionally offered or that the relationship
                            between DAU and the acquisition community will be any closer.


Commercial Business         The National Defense Authorization Act for Fiscal Year 1998, section 912,
Environment Plan Proposes   directed the Secretary of Defense to submit to Congress an implementation
                            plan to streamline the acquisition organizations, workforce, and
Major Changes to Hasten
                            infrastructure. As part of that mandate, the Deputy Under Secretary of
Reform                      Defense for Acquisition and Technology formed a team to develop a plan
                            for ensuring that new practices would be incorporated by DOD. The draft
                            plan, dated June 1999, concludes that adopting the most effective
                            commercial practices requires a cultural and organizational transformation
                            within DOD. The draft proposes a very broad approach to making this
                            change, which includes a training regime to implement key commercial
                            business practices and a change model to accelerate the implementation of
                            new practices. The model calls for establishing goals and a scorecard to
                            measure whether they are met. To inform everyone of the goals, a tiered
                            approach is envisioned, starting with senior leaders, rolling down to teams
                            with specific reform target goals. The teams are to report on their



                            Page 56                                          GAO/NSIAD-99-206 Best Practices
                             Chapter 4
                             DOD and Commercial Training Methods
                             Reflect Different Strategic Approaches to
                             New Practices




                             outcomes within 60 days. A mission support office will support the teams
                             as well as capture results across DOD and make them available to others.
                             This office is to be led by a senior DOD official and staffed by management
                             and facilitation experts, and possibly outside contractors.

                             The plan proposes a team approach for managing acquisitions that
                             embraces best practices and maintains “learning organizations that seek
                             out and adopt best practices that improve individual and organizational
                             performance.” The plan also calls for new organizational roles in adopting
                             of best practices. Among these is DAU. The plan recommends that DAU be
                             broadly recast to adopt the corporate university model and become a
                             change agent. This plan, while not specific about the help that program
                             offices would receive, calls for a strategic approach that would make it
                             more likely that DOD could provide its program offices tailored training—
                             more help—in implementing best practices. Nonetheless, translating this
                             plan into the type of training and other help program offices need to
                             implement best practices is challenging. Key elements, such as the
                             cascading of goals from senior levels on down and the revision of what
                             constitutes teams, are major undertakings and challenge long established
                             patterns of interaction. How the mission support office would reach
                             specific program offices is unclear, as is how it would interface with DAU.
                             Also, whether initiatives are to be prioritized and how DAU would be
                             restructured to provide functional training and become a change agent for
                             new practices remains to be delineated.


Continuous Learning Policy   The objective of DOD’s recently issued continuous learning policy is to
Does Not Ensure Best         augment the acquisition training standards for career field certification of
                             acquisition professionals. The policy requires all civilian and military
Practices Training
                             members of the acquisition workforce to complete 80 hours of continued
                             education within a 2-year period. As such, it is a good step for reaching
                             more senior people in the acquisition workforce who would normally not
                             receive this kind of training because they are fully certified. The policy also
                             places increased emphasis on distance learning, which may address some
                             of the timeliness and availability issues raised by program office officials in
                             obtaining useful best practice information. One distance learning course
                             that deals directly with a best practice was jointly developed last year by
                             the National Contract Management Association and the National
                             Association of Purchasing Management at the request of DOD. It is an
                             internet-based course to integrate best commercial practices for managing
                             suppliers into DOD acquisitions. It entitles participants to 24 hours of




                             Page 57                                           GAO/NSIAD-99-206 Best Practices
Chapter 4
DOD and Commercial Training Methods
Reflect Different Strategic Approaches to
New Practices




continuous education credit and participants are encouraged to take the
course as teams.

While the policy does not preclude staff from receiving training on new
practices, given the array of options for meeting requirements, it does little
to ensure that best practices training will be received. An earlier version of
the policy recommended 16 units of acquisition reform training that could
have included best practices, but the requirement was deleted. The policy
encourages acquisition reform training but allows for requirements to be
met in many ways—including courses, professional activities, and on the
job experience—which may or may not include best practices information.
According to a training official, crediting acquisition workforce members
for almost anything dilutes the policy.

Officials of one program noted that no one advises them on what courses to
take. They can take courses or attend conferences that they believe would
benefit them, but they believed this may be inadequate for ensuring that
senior officials are aware of new ideas. Another program official said that
her program office could meet continuing education requirements by
mandatory attendance of Acquisition Reform Week. While this exposure
would increase awareness of new practices, such training was
characterized as providing just a “flavor” for the initiative—not providing
enough information to implement the practice.




Page 58                                          GAO/NSIAD-99-206 Best Practices
Chapter 5

Conclusions and Recommendations                                                               Chapte5
                                                                                                    r




Conclusions   Leading commercial firms and weapon system program offices have
              successfully implemented best practices or new initiatives by taking a
              tailored, on-site training approach. Such an approach is a concentrated,
              up-front effort that creates an environment for accepting the practice
              within a program and provides the impetus and know-how to apply the
              practice at the workplace. It is warranted by the investment and
              importance that large programs represent. The difference is that within
              DOD, the approach is allowed to happen, while leading commercial firms
              ensure it happens. These firms ensure that initiatives are successfully
              implemented at the program level by customizing the initiatives to fit the
              programs, working with program office staff to develop training that meets
              the needs of the program, and providing on-site assistance to the program.
              This approach would not necessarily be used every time by commercial
              firms to train their entire workforce or to impart every skill; it is instead an
              intensive approach they reserve for key initiatives and individual programs.
              It stems from an overall training strategy that restricts the number of
              initiatives that are brought to the workforce, ensures that the initiatives
              have well-defined objectives, and makes training organizations responsible
              for seeing that policy is turned into practice at the program level. Little
              regarding implementation is left to chance.

              Within DOD, this approach can be taken if a program’s management has the
              insight to recognize the importance of an initiative, the necessary resources
              needed to implement the initiative at the work site, and support from top
              management. Even though DOD has drawn its reform initiatives from the
              practices of leading commercial firms, it has not adopted the
              accompanying training strategy to ensure a concerted effort is made to
              implement key initiatives at the program level. As a result, more is left to
              chance. In the past 4 years, over 40 initiatives have been introduced to the
              DOD acquisition workforce, without delineating which are the most
              important for weapon system programs. Several organizations are
              responsible for training and each service has an office that promotes
              acquisition reform initiatives. Although these organizations are doing
              valuable work and are contributing to workforce training, it is unclear how,
              collectively, they can drive key initiatives to the program office level.

              DAU’s main tool for training the acquisition workforce—functional
              training—is aimed at increasing the expertise of people in their career
              fields, but it does not have the throughput and ability to reach the program
              office as a work unit, nor the practical depth needed to imbed an initiative
              at the program office. Other tools, such as satellite broadcasts, distance



              Page 59                                           GAO/NSIAD-99-206 Best Practices
                  Chapter 5
                  Conclusions and Recommendations




                  learning, and Acquisition Reform Week, make program office staff aware of
                  new initiatives, but they do not provide the depth or environment to
                  implement them because DOD’s training organizations and methods have
                  not been designed to provide this kind of support.

                  DOD recognizes the importance of training as a tool that can help infuse
                  best practices into weapon acquisitions and has several draft plans to
                  reorganize how training is delivered. However, it is not evident that the
                  planning and methods called for by the DAU transition plan will change the
                  way such training is conceived and delivered to weapon system program
                  offices. The Commercial Business Environment Plan does call for a
                  strategic approach that would make it more likely that DOD could provide
                  its program offices tailored training—more help—in implementing best
                  practices. The fact that neither plan has been approved indicates that DOD
                  has not decided what role it wants acquisition training to play on best
                  practices.

                  Whether through existing organizations or through new ones, DOD needs
                  to increase its capability to provide tailored training of specific initiatives at
                  the program office level. Developing this additional capability will require
                  a strategy for implementation and collaboration with program offices on
                  the design of training. Without a concerted approach to foster the
                  implementation of best practices by providing for customized training at
                  the program offices, better outcomes in weapon programs will be more
                  difficult to achieve. The recommendations that follow address ways DOD
                  can provide tailored training on selected practices to program offices. The
                  recommendations are made without prejudice toward the functional and
                  other training DOD provides to the acquisition workforce.



Recommendations   We recommend that the Secretary of Defense develop a strategy for
                  delivering targeted training on selected new practices to program offices to
                  ensure the practices will be implemented. This strategy should
                  accomplish the following.

                  • Identify those initiatives most worthy of a targeted training investment
                    by screening the numerous initiatives to separate those for which a
                    general awareness is sufficient from those that warrant a targeted
                    approach. Those to be targeted should be relatively few in number at
                    any given point in time. The process of setting priorities would be of a
                    continuing nature and would benefit from the input of current program
                    office members.



                  Page 60                                             GAO/NSIAD-99-206 Best Practices
Chapter 5
Conclusions and Recommendations




• Decide whether DAU is to play a more traditional role, as called for by
  the DAU Transition Plan, or a more proactive role, as called for by the
  Commercial Business Environment Plan and clearly communicate to
  DAU and other providers of DOD training their responsibilities in
  supporting a targeted approach to training.
• Identify the key organizations and people that are critical to the program
  offices’ ability to implement best practices, including those not currently
  defined as part of the acquisition workforce and contractors, as the
  potential audience of targeted training.

We also recommend that the Secretary make the Under Secretary of
Defense for Acquisition and Technology responsible for taking steps to
institutionalize the methods for tailoring training on key initiatives that
have been shown to be successful by leading program offices and
commercial firms. This involves having proactive organizations and the
tools to inform, prepare, and assist program offices to implement the
initiatives most applicable to their programs. This approach should include
the following.

• Making new program managers aware of the initiatives that could
  significantly affect the outcomes of their programs and the role the
  managers play in having their staffs trained to implement those
  initiatives. This could be done through existing program managers’
  courses.
• Working with individual program managers to tailor the initiatives to
  their programs.
• Developing an approach, in conjunction with the program managers, to
  create a culture—and necessary incentives—within the program office
  to make it receptive for adopting the initiatives.
• Helping the program managers determine the best methods for making
  initial and sustaining resource investments in training the program
  office staff. This help could consist of providing advice, assisting in the
  identification of experts in the needed areas, and culling lessons learned
  from other programs that have implemented the initiative.

To introduce and reinforce new practices, we recommend that the
Secretary require the Under Secretary of Defense for Acquisition and
Technology more effectively use existing training vehicles to

• incorporate new practices more quickly,
• better reflect the changing role of program managers,




Page 61                                         GAO/NSIAD-99-206 Best Practices
                      Chapter 5
                      Conclusions and Recommendations




                      • ensure that instructors’ knowledge keeps pace with the latest practices,
                        and
                      • provide more case study material that gives current implementation
                        examples for a variety of situations.



Agency Comments and   DOD concurred with a draft of this report and all of its recommendations
                      (see app. I). DOD noted that it was taking steps to develop a strategy for
Our Evaluation        delivering targeted training on selected new practices to program offices,
                      consistent with our report. It cited the Commercial Business Environment
                      study team’s vision for accelerating cultural change within the acquisition
                      community, in conjunction with a strategy of delivering team training, to
                      implement best practices. DOD stated that it anticipates adopting and
                      launching many of the study’s recommendations without delay. DOD also
                      recognized that DAU should augment its capability, consistent with the best
                      practices of the private sector corporate universities, by embracing the role
                      of “change agent” and by designating “performance consultants” who focus
                      on developing tailored training to meet program team needs.

                      DOD noted that it has also begun the restructuring of DAU to provide a
                      more centralized and integrated education and training program that will
                      also be integrated with the initiatives identified by the Commercial
                      Business Environment report. We agree that the DOD study frames a
                      strategic approach that would make it more likely that DOD could provide
                      its weapon system program offices tailored training to help them
                      implement new practices. We are also encouraged by the speed with which
                      DOD is implementing the study’s recommendations. We do note that the
                      study is not specific about the help that program offices would receive and
                      that translating the study into such specifics will be a significant challenge.
                      As DOD proceeds with implementation, we reiterate the importance of
                      screening the numerous initiatives to identify those most worthy of
                      targeted investments and to involve the staff of current program offices in
                      setting these priorities.

                      In agreeing on the need for targeted training, DOD noted that DAU’s
                      communications arm will work to provide for immediate delivery of
                      tailored training on key initiatives to meet the needs of program offices.
                      DAU’s current communications arm—the ARCC—provides training
                      primarily by distributing videos and periodic satellite broadcasts. As such,
                      ARCC may not be well-poised to take on the responsibilities of a proactive
                      organization with the tools needed to inform, prepare, and assist program
                      offices to help them implement the initiatives most applicable to their



                      Page 62                                           GAO/NSIAD-99-206 Best Practices
Chapter 5
Conclusions and Recommendations




programs, as we have recommended. Regardless of which organization is
assigned the responsibility, it is important that the organization go beyond
the traditional approach of making standard training available and instead
work with program offices to (1) tailor the initiatives to their programs;
(2) develop an approach to help make their staffs receptive to adopting
new practices; and (3) help the program managers—on site, if necessary—
determine the best methods for making the investments in training the
program office staff.

Regarding making more effective use of existing training vehicles, DOD
stated that during a review of core curriculum requirements, it will be
completely reviewing the principal program management training toward
implementing a fully integrated strategy. It noted that the strategic plan
will address the development of tailored case study materials, practical
exercises, and assessment criteria for the adoption of new practices. We
believe that DOD’s proposed review has the potential to more effectively
promote best practices training. In addition to these actions, as DOD
conducts its review, it needs to have existing training vehicles incorporate
new practices more quickly, better reflect the changing role of program
managers, and ensure that instructors’ knowledge keeps pace with the
latest practices. Greater involvement of weapon system program office
staff in the design and content of these training vehicles could link the
training more closely with the job at hand. While DOD plans to focus its
continuous learning activities on key acquisition reforms, currently,
members of the acquisition workforce do not necessarily have to take
training on acquisition reform to meet continuous learning requirements.




Page 63                                         GAO/NSIAD-99-206 Best Practices
Appendix I

Comments From the Department of Defense                     Appenx
                                                                 Idi




             Page 64          GAO/NSIAD-99-206 Best Practices
Appendix I
Comments From the Department of Defense




Page 65                                   GAO/NSIAD-99-206 Best Practices
Appendix I
Comments From the Department of Defense




Page 66                                   GAO/NSIAD-99-206 Best Practices
Appendix II

Description of Program Offices Visited                                                                                         AppenIx
                                                                                                                                     di




                        Following is a description of each of the program offices we visited.



Advanced Amphibious     The Advanced Amphibious Assault Vehicle (AAAV) is to be used by the
                        Marine Corps to provide high-speed transportation of troops from ships
Assault Vehicle         located beyond the horizon to the beaches. It will also provide armor
                        protection, land mobility, and fire support during combat operations on the
                        shore. It is a category I1 major acquisition program for the Marine Corps.
                        The program office is staffed by about 75 people.



Advanced                The Advanced Medium-Range Air-to-Air Missile (AMRAAM) is a new
                        generation radar-guided missile that fighter aircraft are to use against
Medium-Range            enemy aircraft. It can be launched beyond visual range, day or night and in
Air-to-Air Missile      all weather. The missile is used on the Air Force F-15 and F-16 and the
                        Navy F-14 and F/A-18. It is a category I acquisition program for the Air
                        Force. The AMRAAM program started over 20 years ago. About 70 people
                        staff the program office.



Battlefield Combat      The Battlefield Combat Identification System (BCIS) is a millimeter wave
                        electronic, question, and answer combat identification system capable of
Identification System   identifying friendly ground combat vehicles. The BCIS interrogation is
                        triggered automatically by activation of the shooter, which sends an
                        encrypted query message to the targeted vehicle. If the targeted vehicle is
                        friendly and equipped with BCIS, its transponder answers with an
                        encrypted friend message that is illuminated in the shooter’s sights. It is a
                        category II2 acquisition managed by the Army. The program office is
                        staffed by 16 core people and 30 additional people who are assigned from
                        other government organizations or contractor support.




                        1
                         Category I programs are defined as major defense acquisition programs estimated to cost over
                        $355 million for research, development, test, and evaluation, or have procurement costs of more than
                        $2.135 billion.
                        2
                         Category II programs are defined as acquisition programs estimated to cost over $75 million for
                        research, development, test and evaluation, or have procurement costs of more than $300 million.




                        Page 67                                                         GAO/NSIAD-99-206 Best Practices
                         Appendix II
                         Description of Program Offices Visited




Joint Air-to-Surface     The Joint Air-to-Surface Standoff Missile (JASSM) is an autonomous,
                         long-range cruise missile to be capable of launch from outside area
Standoff Missile         defenses to hit ground targets. It is to be launched from a wide range of
                         bomber, attack, and fighter fixed-wing aircraft. It is a joint Air Force and
                         Navy category I major acquisition program.



Joint Strike Fighter     The Joint Strike Fighter (JSF) program’s objective is to develop and field an
                         affordable, highly common family of next generation multirole strike
                         fighter aircraft for the Navy, the Air Force, the Marine Corps, and U.S.
                         allies. The focus of the program is affordability. It is a joint service
                         category I major acquisition program with Navy as lead. About 100 people
                         staff the program office.



Joint Surveillance and   The Joint Surveillance and Target Attack Radar System– Joint Tactical
                         Terminal (JSTARS-JTT) is to provide warfighters with near real-time
Target Attack Radar      tactical intelligence and targeting information. It is a terminal that supplies
System–Joint Tactical    the critical data link from various intelligence sources to battle managers
                         across all services. The terminal is integrated into other weapon systems
Terminal                 and is to be mounted in fixed- and rotary-wing aircraft, surface ships, and
                         fixed or mobile ground platforms and vehicles. It is managed by the Army
                         and is a category III 3 program. About 10 people staff the program office.




                         3
                          Category III programs are defined as those acquisitions programs that do not meet the criteria for
                         categories I or II.




                         Page 68                                                          GAO/NSIAD-99-206 Best Practices
Appendix III

GAO Contacts and Staff Acknowledgments                                                        AppeInIx
                                                                                                     di




GAO Contacts      Louis J. Rodrigues (202) 512-4841
                  Paul L. Francis (202) 512-2811



Acknowledgments   In addition to those named above, Charles Cannon, Monica L. Kelly,
                  Gordon W. Lusby, Elisabeth G. Ryan, Sally Shipman, and Yelena K.
                  Thompson made key contributions to this report.




                  Page 69                                       GAO/NSIAD-99-206 Best Practices
Page 70   GAO/NSIAD-99-206 Best Practices
Page 71   GAO/NSIAD-99-206 Best Practices
Related GAO Products


                   Best Practices: Commercial Quality Assurance Practices Offer
                   Improvements for DOD (GAO/NSIAD-96-162, Aug. 26, 1996).

                   Major Acquisitions: Significant Changes Underway in DOD’s Earned Value
                   Management Process (GAO/NSIAD-97-108, May 5, 1997).

                   Best Practices: Successful Application to Weapon Acquisitions Requires
                   Changes in DOD’s Environment (GAO/NSIAD-98-56, Feb. 24, 1998).

                   Best Practices: DOD Can Help Suppliers Contribute More to Weapon
                   System Programs (GAO/NSIAD-98-87, Mar. 17, 1998).

                   Defense Acquisition: Improved Program Outcomes Are Possible
                   (GAO/T-NSIAD-98-123, Mar. 18, 1998).

                   Defense Acquisition: Best Commercial Practices Can Improve Program
                   Outcomes (GAO/T-NSIAD-99-116, Mar. 17, 1999).

                   Best Practices: Better Management of Technology Development Can
                   Improve Weapon System Outcomes (GAO/NSIAD-99-162, July 30, 1999).




(707343)   Leter   Page 72                                       GAO/NSIAD-99-206 Best Practices
Ordering Information

The first copy of each GAO report and testimony is free.
Additional copies are $2 each. Orders should be sent to the
following address, accompanied by a check or money order made
out to the Superintendent of Documents, when necessary, VISA and
MasterCard credit cards are accepted, also.

Orders for 100 or more copies to be mailed to a single address are
discounted 25 percent.

Orders by mail:

U.S. General Accounting Office
P.O. Box 37050
Washington, DC 20013

or visit:

Room 1100
700 4th St. NW (corner of 4th and G Sts. NW)
U.S. General Accounting Office
Washington, DC

Orders may also be placed by calling (202) 512-6000
or by using fax number (202) 512-6061, or TDD (202) 512-2537.

Each day, GAO issues a list of newly available reports and
testimony. To receive facsimile copies of the daily list or any list
from the past 30 days, please call (202) 512-6000 using a touchtone
phone. A recorded menu will provide information on how to obtain
these lists.

For information on how to access GAO reports on the INTERNET,
send an e-mail message with “info” in the body to:

info@www.gao.gov

or visit GAO’s World Wide Web Home Page at:

http://www.gao.gov
United States                       Bulk Rate
General Accounting Office      Postage & Fees Paid
Washington, D.C. 20548-0001           GAO
                                 Permit No. GI00
Official Business
Penalty for Private Use $300

Address Correction Requested