United States General Accounting Office GAO Report to the Chairman and Ranking Minority Member, Subcommittee on Readiness and Management Support, Committee on Armed Services, U.S. Senate August 1999 BEST PRACTICES DOD Training Can Do More to Help Weapon System Programs Implement Best Practices GAO/NSIAD-99-206 United States General Accounting Office National Security and Washington, D.C. 20548 International Affairs Division B-280234 Letter August 16, 1999 The Honorable James Inhofe Chairman The Honorable Charles Robb Ranking Minority Member Subcommittee on Readiness and Management Support Committee on Armed Services United States Senate As you requested, this report assesses the extent the Department of Defense’s (DOD) training helps weapon system program offices apply best practices and whether such training can be of more help. We make recommendations to the Secretary of Defense on how DOD training can better support program offices in the application of best practices. We are sending copies of this report to the Honorable William S. Cohen, Secretary of Defense; the Honorable Louis Caldera, Secretary of the Army; the Honorable Richard Danzig, Secretary of the Navy; the Honorable F. Whitten Peters, Acting Secretary of the Air Force; the Honorable Jacob J. Lew, Director, Office of Management and Budget; and to interested congressional committees. We will also make copies available to others upon request. If you or your staff have any questions regarding this report, please call me at (202) 512-4841. Contacts and key contributors to this report are listed in appendix III. Katherine V. Schinasi Associate Director Defense Acquisitions Issues Leter Executive Summary Purpose The Department of Defense (DOD) plans to increase its annual procurement investment to about $60 billion by fiscal year 2001. DOD has high expectations from this investment: that new weapons will be better yet less expensive than their predecessors and will be developed in half the time. Essential to getting these kinds of outcomes will be the adaptation of best commercial practices that have enabled leading commercial firms to develop new products faster, cheaper, and better. DOD has begun a number of acquisition reform initiatives based on commercial practices to help foster these outcomes. Their success depends greatly on the extent to which the program offices responsible for managing weapon acquisitions can implement the practices on individual programs. Training provided to the program offices serves as a key agent in both creating a culture that is receptive to new practices and in providing the knowledge needed to implement new practices at the workplace. At the request of the Chairman and the Ranking Minority Member, Subcommittee on Readiness and Management Support, Senate Committee on Armed Services, GAO evaluated the role DOD training is playing in implementing best practices in weapon system programs. This report addresses (1) the contribution DOD training makes to program offices’ ability to apply best practices, (2) the different methods used by DOD and leading commercial firms in training on best practices, and (3) the strategic approaches that underlie DOD’s and leading commercial firms’ training methods for best practices. Background GAO’s review focused on weapon system program offices because they comprise a key component of DOD’s acquisition workforce. In planning, managing, and executing acquisition programs, these program offices are responsible for managing about $80 billion of DOD’s annual research, development, and procurement funds. As an entry point for DOD acquisitions, program offices greatly influence the work of the rest of the acquisition workforce. The primary responsibility for training the acquisition workforce falls within the Office of the Under Secretary of Defense for Acquisition and Technology. In 1992, the Defense Acquisition University, a consortium of 13 schools, was created to develop and provide training for the acquisition workforce. Each service also has an acquisition reform office that provides the workforce with the latest information about practices and initiatives that apply to acquisitions. Based on personnel reductions mandated by the National Defense Authorization Acts for Fiscal Years 1996 and 1997, DOD expects the acquisition workforce of fiscal year 2000 to be 25 percent smaller than that of fiscal year 1995. Thus, training Leter Page 2 GAO/NSIAD-99-206 Best Practices Executive Summary will become even more important as new authority and responsibility is granted to those who remain in the workforce. To determine the extent to which DOD training and other sources helped program offices obtain the knowledge needed for implementing best practices, GAO focused on five specific practices: cost as an independent variable, integrated product teams, evaluation of contractors’ past performance, setting performance specifications, and managing supplier relationships. The first four are formal DOD initiatives that are based on best commercial practices, while supplier relationships is a best practice GAO has observed in leading commercial firms. GAO selected six program offices considered by DOD as leaders in implementing one or more of the practices. As such, they represented best case examples of marshaling training and other resources needed to implement new acquisition practices. The term “standard training” is used in this report to describe the training provided by the Defense Acquisition University and the services’ acquisition reform offices, as distinct from training that program offices provide on their own. Results in Brief DOD’s standard training did not make a major contribution to the leading program offices’ ability to implement best practices. In evaluating their key sources of knowledge, none of the key officials from programs at the forefront of implementing best practices ranked standard DOD training first, with many ranking it last. DOD training either did not reach the right people when it was needed or did not reach them at all. When training on best practices was received, it did not contain the depth or practical insights program office people needed to implement the practices. It was primarily through their own efforts—learning on the job, finding external training, or developing their own training program—that they attained the knowledge needed to apply best practices. Thus, success depended on their having the foresight to see what was needed, the ingenuity to find good sources of knowledge, and the resources needed to attain that knowledge. Replicating this approach broadly on other programs is problematic. Managers may not see the relevance of a practice to their programs and thus may not realize what training is needed. Others may not be able to afford the needed training. Leading commercial firms and DOD use different training methods to implement key practices. Commercial firms use targeted, hands-on methods to ensure that program offices are trained on key practices. Their training organizations conduct front-end analyses to determine the Leter Page 3 GAO/NSIAD-99-206 Best Practices Executive Summary programs’ training requirements and involve the program offices in designing the training. Training is customized to meet the specific needs of those implementing the practice. Company officials believe the targeted method results in more useful training, which helps to improve outcomes of the final product. DOD does not have a counterpart to this method. DOD relies primarily on its standard training, including classroom courses, videos, internet-based training, satellite broadcasts, and roadshows, to inform staff on best practices. These methods were designed for functional training, such as for engineers, and for increasing the awareness of new practices. As such, they do not provide the depth or reach enough of the right people at the right time to be of great help in implementing best practices at program offices. Also, DOD does not systematically involve program offices in the design of training. The intensive training methods leading commercial firms employ on new practices are the result of a strategic, institutionally driven approach to implementation. These firms commit their resources and attention to a few well-defined practices and make a significant front-end investment in the training to be provided to the workforce. Also, the firms strive to create an environment to put those responsible for implementing the practices in a good position to succeed. DOD’s training methods for best practices do not stem from such a strategic approach. DOD has promulgated as many as 40 acquisition management initiatives in the past few years without communicating their relative priority to trainers or implementers. Often, the initiatives have not been accompanied by clear guidance or by the initial training needed for implementation. While DOD commits significant resources to training, it does not make a uniform front-end investment to ensure that program offices will succeed with the new practices. Since 1997, two studies commissioned by DOD have recommended ways to make training organizations more effective in providing training of best practices. These recommendations were not adopted in favor of a more traditional training role. In June 1999, another DOD study proposed that DOD training organizations become change agents and be modeled after their corporate counterparts. If the latest study’s recommendations are adopted, DOD may be in a much better position to provide the type of help program offices need to successfully implement best practices. GAO makes recommendations to the Secretary of Defense on how DOD’s training on best practices can better support the needs of program offices. Page 4 GAO/NSIAD-99-206 Best Practices Executive Summary Principal Findings DOD Training Not a Major Training often did not exist or was not provided when program officials Catalyst for Best Practices began to implement an individual practice. For example, when the Joint Strike Fighter program office started to implement cost as an independent variable, there were no guiding documents, and no one, including the training community, knew what the initiative really meant. Some people involved with implementing best practices were missed altogether by DOD training offerings because they had not been required to take training since before the initiatives began. Others, such as requirements authors, fell outside the definition—and training curriculum—of the acquisition workforce. In one case, program officials reduced contractor reporting requirements to 2 items, in line with acquisition reform, only to have 40 more added by another office with approval authority over the contract. Contractors are also essential to the application of best practices, but they are not part of the DOD defined acquisition workforce or the training offered. Consequently, they do not necessarily understand or know how to implement new practices. Program officials noted that DOD’s standard training typically provided a general awareness of the practices but not the “how-to” knowledge needed for implementation. Training was not tailored in such a way that individuals could see how the practices could be applied to their program. Program officials also observed that in some cases training suffered because instructors did not use up-to-date case studies and were not current on new practices themselves. They also believed they had little opportunity to influence the training they received from DOD. Program officials used a combination of ways—generally outside of standard DOD training offerings—to get the knowledge they needed to apply best practices. Several program officials relied on the cumulative job expertise of the staff and personal research to teach themselves how to implement new initiatives. For example, on the Advanced Medium-Range Air-to-Air Missile program, officials relied on their personal experiences to know how to set contract specifications at a performance, rather than a detailed, level. Officials also used their program funds to send staff to nongovernment sources or to bring experts in. One program manager sends people to outside training, such as university leadership courses, to develop creative thinking skills. One of his managers said some of his best training was from off-site sessions sponsored by the program office that dealt with the people issues critical to making integrated product teams work. For the Advanced Amphibious Assault Vehicle program, the prime Page 5 GAO/NSIAD-99-206 Best Practices Executive Summary contractor was responsible for providing the training on integrated product teams and cost as an independent variable. The contractor hired a third party to develop a training program tailored to the Advanced Amphibious Assault Vehicle program, and both program office and contractor staff were taught together, on-site as a team. DOD’s Training Methods for For routine training, such as skill building, leading commercial firms have Best Practices Do Not Go as standard training offerings, including functional area courses and instruction on corporatewide issues, such as communications or ethics. Far as Leading Commercial However, when implementing key practices—such as those that change Firms’ how product development and production are conducted—leading firms go beyond standard training offerings. Commercial firms use a targeted, hands-on approach to ensure program teams are put in a good position to implement a new practice. The elements common to how leading firms provide training on a key practice include (1) front-end analysis of program teams’ needs and training requirements, (2) involvement of program teams in key training decisions, (3) customized training to meet program team specific needs, and (4) targeted training for the implementation of specific practices. Program staff participate in and often influence a wide range of training decisions, including the amount of training provided for certain job descriptions, course topics, depth of course coverage, and identification of the appropriate course recipients. The involvement of the program staff has improved course depth, timeliness, and coverage of personnel in the commercial firms. In the Boeing Company, training representatives develop a partnership with the program staff when a team is formulated to design and manufacture a new airplane. The training organization forms “drop teams” to colocate with the program staff to learn as much as possible about the business process and the staff’s concerns and to determine what training is needed to help the program staff implement a practice. Boeing officials stated that training was instrumental to the implementation of key practices, such as design build teams, on the 777 aircraft program. They noted that such teams were at odds with the company’s culture because employees were not accustomed to working in a team environment and sharing information across functional areas. Training officials worked side by side with the program staff to create a training program that provided team building and conflict resolution techniques as well as new technical skills training. To ensure all program staff were equally trained, employees were required to complete training before they reported for duty on the program. The professional employees—engineers and drafters—were required to Page 6 GAO/NSIAD-99-206 Best Practices Executive Summary complete 120 hours of start-up training on several key 777 practices before they were allowed to report for duty. Teams were often trained together at the work location. Ford followed a similar approach when it implemented the Ford Product Development System—a lean engineering process. Responsibility for training on best practices is diffused among several DOD organizations, including the Defense Acquisition University and the service acquisition reform offices. However, GAO did not find an organization that was able to tailor and help deliver training on best practices to the program offices visited. Training provided by the university is designed primarily to enable people in individual career fields or functions, such as engineering and cost estimating, to meet professional certification requirements. The university incorporates best practices topics into these functionally oriented courses as drop-in modules that provide a survey of the topic, but not in-depth coverage. Although program offices see a greater need for training that cuts across functions to implement new practices and to manage in a team environment, it is difficult for a person in one career field to obtain training in another field. The usefulness of these courses for best practices is further hampered by limited availability; according to an official from the university, the member schools get about 10 percent of the workforce into training each year. DOD’s Acquisition Reform Communication Center and the acquisition reform offices in the services communicate best practice information through videos, periodic satellite broadcasts, roadshows, and Acquisition Reform Week. These methods can reach more people than Defense Acquisition University courses and are designed around practices—versus functions—but are not tailored to specific program offices and are not necessarily delivered at the time those implementing new practices or initiatives need them. For example, roadshows, traveling multiday training workshops provided to staff at a number of locations, typically provide awareness training on the practices. DOD officials estimate that only 10 to 15 percent of the acquisition workforce attend the second day of workshops, where more detailed training is provided. The annual Acquisition Reform Weeks, which are a combination of satellite broadcasts and local presentations, mainly provide awareness-level training. Neither individual attendance nor the level of learning attained by attendees is tracked by these methods. Page 7 GAO/NSIAD-99-206 Best Practices Executive Summary Differences Between DOD Leading commercial firms shared a common strategy for adopting and and Commercial Training deploying key new practices. First, the firms’ corporate management committed to and adopted few key practices—seven or less—at any given Reflect Different Strategic time. In doing so, the companies were able to concentrate their attention Approaches and target resources to implementing the practices. It also signaled the importance of the practices to trainers and implementers. Second, the firms assessed which staff should be included in the implementation. For example, Ford’s training unit determined which engineering teams working on vehicle lines could benefit from the new production development system. Third, company leaders made implementing the practice mandatory for the target population. Lastly, companies developed well-defined learning objectives to better ensure that the target population consistently understood how to apply a new practice to improve production outcomes—the ultimate goal. According to officials from these firms, it was a corporate responsibility to ensure that those implementing the practice received the necessary training and other assistance to succeed. It was for this reason that the companies made a significant front-end investment to support the needs of program offices that would implement key practices. Company officials also pointed out that training is just one of the several components necessary for adopting new practices. They stressed that providing strong leadership and the right environment were key to ensuring the implementation of new practices and to developing quality training. DOD’s approach to implementing best practices is less structured and more reliant on individual program offices to make the necessary training investment. DOD policies on individual reform initiatives are typically promulgated without indicating what components of the acquisition workforce or which programs are expected to implement the practice. In addition, the policies themselves are not always clear. For example, although the initiative on cost as an independent variable was promulgated in 1995, Advanced Amphibious Assault Vehicle program officials developed training in 1997 to define the initiative for the program. In November 1998, an Air Force workshop on the cost initiative reported that it was still not well understood or widely implemented. DOD has proposed over 40 reform initiatives since 1994, without an indication of relative priority, leaving educators and implementers to decide what is important. Program offices are not necessarily in a good position to sort through the initiatives to focus on those that are the most important to the job at hand. A service acquisition reform official observed that the combination of many reform initiatives and unclear priorities causes the office to guess what is Page 8 GAO/NSIAD-99-206 Best Practices Executive Summary the most important, which leads to emphasizing what is perceived to be popular. DOD is aware of the need to improve the means by which the acquisition workforce receives and implements new initiatives. A 1997 study by a DOD team and a 1998 study by the Logistics Management Institute pointed out several weaknesses in the focus and delivery of DOD training. Weaknesses included the need for the Defense Acquisition University to be more active in implementing best practices and reforms, the tendency for the training curriculum to be functionally stove-piped, and the need for the university to have more involvement with the workforce—the recipients of training. The studies made recommendations for significant organizational and operational changes in the university, including that it should follow the corporate university model of becoming a change agent and a proponent of best practices and put more emphasis on cross-functional training. In September 1998, the university proposed a revised structure, which has not been approved. While the proposal offers some improvements, it stays close to its roots of providing functional training. It does not reflect the corporate university model, a broadened role as a change agent, or a closer relationship with the acquisition workforce. In short, it does not discernibly address key weaknesses in providing training of best practices. In June 1999, a DOD study team chartered to identify training on commercial business practices for the acquisition workforce concluded that adopting the most effective commercial practices requires a cultural and organizational transformation within DOD. The team proposed a cross-functional plan for managing acquisitions that embraces best practices and calls for “learning organizations that seek out and adopt best practices that improve individual and organizational performance.” The plan proposes new roles for several organizations in fostering the adoption of best practices. Among these is the Defense Acquisition University. For example, it recommends that the university be broadly recast to adopt the corporate university model and become a change agent. This plan, while not specific about the help that program offices would receive, does call for a strategic approach that would make it more likely that DOD could provide its program offices tailored training—more help—in implementing best practices. However, the fact that the September 1998 and the June 1999 proposals are still vying for approval indicates that DOD has not yet decided what role it wants acquisition training to play on best practices. Page 9 GAO/NSIAD-99-206 Best Practices Executive Summary Recommendations GAO makes several recommendations to the Secretary of Defense that are intended to ensure that DOD’s approach to the training of new practices better supports the needs of program offices by (1) developing a strategy for a structured approach to training on new practices; (2) providing tailored training assistance to program offices; and (3) improving the standard training curriculum so that it is more timely, relevant, and accessible. These recommendations appear in full in chapter 5. Agency Comments DOD concurred with the views expressed in the report and all of the recommendations. A discussion of DOD’s actions appears in chapter 5. DOD’s comments appear in appendix I. Page 10 GAO/NSIAD-99-206 Best Practices Executive Summary Page 11 GAO/NSIAD-99-206 Best Practices Contents Letter 1 Executive Summary 2 Chapter 1 Responsibilities for Training the Acquisition Workforce Within DOD 16 Introduction Objectives, Scope, and Methodology 21 Chapter 2 DOD Training Did Not Reach the Right People at the Right Time 26 DOD Training Offerings Did Not Provide the Depth Needed to DOD Training Is Not a Implement Best Practices 31 Major Catalyst for Best Limitations of Standard Training Led Program Offices to Develop Their Own Training Solutions 34 Practices Chapter 3 Commercial Firms Use Targeted, Hands-on Methods to Improve Training Usefulness 39 DOD Training Methods DOD Does Not Target Training on Key Practices to Program Offices 45 for Best Practices Do Not Go as Far as Leading Commercial Firms Chapter 4 Leading Firms’ Strategic Approach Better Ensures Implementation of Key Practices 50 DOD and Commercial DOD’s Training on Best Practices Does Not Stem From a Strategic Training Methods Approach 52 DOD Proposals to Alter Training Reflect Conflicting Approaches 54 Reflect Different Strategic Approaches to New Practices Chapter 5 Conclusions 59 Recommendations 60 Conclusions and Agency Comments and Our Evaluation 62 Recommendations Page 12 GAO/NSIAD-99-206 Best Practices Contents Appendixes Appendix I: Comments From the Department of Defense 64 Appendix II: Description of Program Offices Visited 67 Appendix III: GAO Contacts and Staff Acknowledgments 69 Related GAO Products 72 Tables Table 1.1: The Defense Acquisition and Technology Workforce as of April 1999 18 Table 1.2: Best Practices Evaluated at Program Offices Visited 23 Figures Figure 1.1: Organizations Responsible for Training Development 20 Figure 2.1: JSF 28 Figure 2.2: JSTARS-JTT 30 Figure 2.3: AMRAAM 32 Figure 2.4: AAAV 35 Figure 2.5: JASSM 37 Figure 3.1: 777 Airplane 42 Figure 3.2: Ford Focus 43 Page 13 GAO/NSIAD-99-206 Best Practices Contents Abbreviations AAAV Advanced Amphibious Assault Vehicle AMRAAM Advanced Medium-Range Air-to-Air Missile ARCC Acquisition Reform Communications Center BCIS Battlefield Combat Identification System CAIV cost as an independent variable DAU Defense Acquisition University DAWIA Defense Acquisition Workforce Improvement Act DOD Department of Defense FPDS Ford Production Development System IBM International Business Machines IPT integrated product teams JASSM Joint Air-to-Surface Standoff Missile JSF Joint Strike Fighter JSTARS-JTT Joint Surveillance Target Attack Radar System-Joint Tactical Terminal Page 14 GAO/NSIAD-99-206 Best Practices Contents Page 15 GAO/NSIAD-99-206 Best Practices Chapter 1 Introduction Chapte1 r The Department of Defense (DOD) plans to increase its annual procurement investment to $60 billion by fiscal year 2001. DOD has high expectations from this investment: that new weapons will be better and less expensive than their predecessors and will be developed in half the time. Essential to getting these kinds of outcomes will be the adaptation of best commercial practices that have enabled leading commercial firms to develop new products faster, cheaper, and better. To help foster these outcomes, DOD has begun a number of acquisition reform initiatives based on commercial practices. Success depends greatly on the extent to which the program offices responsible for managing weapon acquisitions can implement the practices on individual programs. The training DOD provides program office staff to help them implement best practices should play a central role in getting the desired outcomes. While first-hand experience and “learning by doing” are instrumental in adopting new practices, training serves as a key agent in creating a culture that is receptive to new practices and providing the knowledge to implement the new practices at the workplace. The relationship between training and implementing new practices was highlighted in a 1994 study of 300 “improvement-driven” organizations conducted by Coopers & Lybrand—with the American Society of Quality Control, Rutgers University Center for Public Productivity, and the National Institute of Canada. The study found that training played a “critical, integrative role as driver of cultural change, process alignment, job redesign, and continuous improvement.” Organizations included in the study used training as the vehicle for implementing and sustaining the changes at the level where the work was done. The leading commercial firms we reviewed committed substantial investments to the training on key practices, underscoring its importance to getting the outcomes desired by the firms. The significant numbers of reform initiatives that DOD has introduced, which involve adopting a number of new practices in the acquisition of weapons, present implementation challenges that training can help meet. Responsibilities for For nearly 50 years, the importance of an educated professional DOD acquisition workforce has been emphasized by government leaders and Training the reflected in the work of key studies and reform commissions. The First Acquisition Workforce and Second Hoover Commissions (1949 and 1955), the Fitzhugh Commission (1970), and the Commission on Government Procurement Within DOD (1972) all recognized the importance of high quality, well-educated acquisition professionals to the successful operation of DOD. The Packard Commission, which undertook a broad examination of DOD management Page 16 GAO/NSIAD-99-206 Best Practices Chapter 1 Introduction practices and procedures, reported in its June 1986 report that the DOD acquisition workforce was undertrained and inexperienced. One of its recommendations was to improve the education and training of the acquisition workforce for the purpose of enhancing the defense acquisition process. Based in part by the recommendations of the Packard Commission, the Defense Acquisition Workforce Improvement Act (DAWIA) was passed in 1990 as part of the National Defense Authorization Act for Fiscal Year 1991.1 Its primary objective was to improve the DOD acquisition system by enhancing the education, training, and career development of members of the acquisition workforce. Accordingly, DAWIA established the Defense Acquisition University (DAU) to provide for the professional educational development and training of the DOD acquisition workforce. The act also charged DOD officials with the responsibility to designate certain positions as acquisition positions, to set qualification requirements, and to establish policies and procedures for training the acquisition workforce. The program offices that manage weapon system acquisitions are a key component of DOD’s acquisition workforce. This workforce is generally defined as those people who are responsible for managing the wide array of DOD acquisitions, including contracting professionals, program managers, engineers, scientists, logisticians, and other occupational fields, from the earliest phases of basic research to the logistical support and disposal of old systems. However, there have been several definitions of what comprises the DOD acquisition workforce, which have led to varying estimates of the workforce size, ranging from about 106,000 to 355,000 people. At the end of fiscal year 1997, DOD estimated the acquisition workforce covered by DAWIA included approximately 106,000 positions, of which about 90,000 were civilian. Its most recent definition places the workforce, now called the acquisition and technology workforce, at about 150,000 people2 and includes people from science and technology organizations. Table 1.1 lists the different acquisition and technology workforce occupations and the number in each occupation based on this definition. 1 P. L. 101-510, Nov. 5, 1990. 2 This definition was based on the study, “Identification of the Department of Defense Key Acquisition and Technology Workforce,” April 1999, Jefferson Solutions, Washington, D. C. Page 17 GAO/NSIAD-99-206 Best Practices Chapter 1 Introduction Table 1.1: The Defense Acquisition and Technology Workforce as of April 1999 Acquisition and technology workforce occupations Total persons Engineers (electronics, aerospace, mechanical, civil, and general) 44,117 Contracting 19,387 Management 15,509 Business and industry 12,989 Communications and computers 9,370 Administration and programs 5,116 Scientists 4,476 Auditing 3,692 Financial management 3,618 Procurement assistants 2,650 Mathematics and statistics 2,400 Purchasing 2,158 Supply program management 1,753 Inventory management 944 Equipment specialists 858 General supply 326 Miscellaneous 3,698 Military 16,378 Total 149,439 Source: DOD. Definitions aside, the DOD acquisition workforce has been undergoing restructuring and downsizing. The National Defense Authorization Acts for Fiscal Years 1996 and 1997 mandated reductions in the number of civilian and military employees in acquisition organizations. DOD estimates that, as a result, the acquisition workforce will be 25 percent smaller at the end of fiscal year 2000 compared to fiscal year 1995. DOD understands that effective training will become even more important as the workforce is reduced and new authority and responsibility are granted to those who remain. We have previously reported that decisions to restructure or reduce this workforce should be linked to getting better outcomes from the acquisition process; doing otherwise would miss an opportunity to address the deep-seated causes of acquisition problems. 3 3 Defense Acquisition Organizations: Linking Workforce Reductions With Better Program Outcomes (GAO/T-NSIAD-97-140, Apr. 8, 1997). Page 18 GAO/NSIAD-99-206 Best Practices Chapter 1 Introduction Although weapon system program offices comprise a subset of this workforce, they are a highly leveraged subset. In planning, managing, and executing acquisition programs, program management offices are responsible for about $80 billion of DOD’s annual research, development, and procurement funds—about 30 percent of the fiscal year 1999 DOD budget. They influence much of the work of the rest of the acquisition workforce because they are bringing new equipment into the inventory, which must be managed, budgeted for, maintained, and supplied. In a sense, they are a point of entry for DOD acquisitions. In addition, these offices are made up of a cross section of people that draw from most of the acquisition and technology workforce occupations cited in table 1.1. As such, they are a medium in which the training of new initiatives in different career fields converge. The primary responsibility for training the acquisition workforce in general—and new initiatives in particular—falls within the Office of the Under Secretary of Defense for Acquisition and Technology. That office is responsible for setting the standards that the workforce must meet. DAU, which is responsible for designing and conducting the training to meet those standards, also reports to that office. These and other organizations responsible for setting training standards and providing training are shown in figure 1.1. Page 19 GAO/NSIAD-99-206 Best Practices Chapter 1 Introduction Figure 1.1: Organizations Responsible for Training Development Under Secretary of Defense for Acquisition and Technology Director for Acquisition Education, Defense Acquisition University Training, and Career Development Functional Boards Consortium of Schools • Acquisition management • Defense Systems Management • Contracting College • Technical management (e.g., • Industrial College of the Armed engineering and testing Forces •Business, cost estimating, and • Service Schools financial management The Under Secretary of Defense for Acquisition and Technology delegated responsibility for developing career paths and establishing educational standards to the Office of the Director of Acquisition Education, Training, and Career Development. That office is supported by four functional boards that have established experience, education, and training standards for the acquisition workforce and for professional certification levels in each of the acquisition career fields. A DOD manual, DOD 5000.52M, “Acquisition Career Development Program,” implements and prescribes procedures for career development of the acquisition workforce. The Page 20 GAO/NSIAD-99-206 Best Practices Chapter 1 Introduction manual establishes experience, education, and training standards for 3 certification levels in each of the 11 acquisition career fields. Level I is basic or entry level, level II is the intermediate or journeyman level, and level III represents the advanced or senior level. The specific training to increase competency or attain higher certification levels within a career field is often referred to as functional training. There are also standards for specific acquisition workforce positions (such as program managers), position categories, and membership in the acquisition corps. DAU was created by DAWIA to develop the training curriculum to meet these standards and to coordinate the efforts of its consortium of 13 DOD-wide and service schools that conduct the training courses. The Defense Systems Management College, a DOD-wide school, is dedicated to providing acquisition-related training. Other member schools, including those run by individual military services, provide a variety of training, of which acquisition training is only a part. While the service schools provide information on new initiatives, they do not have primary responsibility for educating the acquisition workforce. Each service also has an acquisition reform office that helps make people aware of the latest practices and initiatives that apply to acquisitions, although this office does not play a significant role in designing or providing the training. The Acquisition Reform Communications Center is an organization related to DAU that has the mission of sharing knowledge about acquisition reform by providing and disseminating information on how DOD is changing the way it acquires goods and services. Objectives, Scope, and The Chairman and the Ranking Minority Member, Senate Committee on Armed Services, Subcommittee on Readiness and Management Support, Methodology asked us to review how DOD is training its acquisition workforce to implement best practices for acquiring weapon systems. The Subcommittee’s request is part of a broader interest in seeing that best practices are incorporated into DOD’s acquisition process as a way of saving money for modernization, increasing efficiency, and improving quality. The objectives of this report are to assess (1) the contribution DOD training makes to program offices that are applying best practices, (2) the different training methods DOD and leading commercial firms use in providing training on practices, and (3) the different strategic approaches that underlie the training methods DOD and leading commercial firms use in implementing practices. Page 21 GAO/NSIAD-99-206 Best Practices Chapter 1 Introduction To meet these objectives, we focused on five specific practices and identified program offices recognized as being leaders in applying them. We chose the five practices based on discussions with DOD and on our previous work in the application of best practices to weapon acquisitions. The first four are formal DOD initiatives that are based on best commercial practices, while supplier relationships is a best practice we observed in leading firms and in some leading weapon system programs.4 The selected practices are the following: • Cost as an independent variable (CAIV): An acquisition management practice in which aggressive life-cycle cost goals are achieved through trade-offs between requirements and performance. • Integrated product teams (IPT): Teams composed of members from functional disciplines such as engineering, test, and contract management. All members contribute their particular expertise to team decisions and to resolve issues. • Performance specifications: States requirements in terms of required results without stating the methods to achieve those results. They define the functional requirement for operation, interface, and interchange characteristics, and have criteria for verifying performance compliance. • Past performance: Information on a contractor’s past performance on relevant prior work is used as a factor in source selection. The information is used to evaluate risk and the potential for future contractor success. • Supplier relationships: A commercial practice in which maximum participation of suppliers and their suppliers is encouraged to promote product excellence. The best suppliers are selected and supported in a number of ways that ensure a mutually beneficial partnership. We focused on weapon system program offices because of the significant role they play in implementing best practices. Initially, we considered gathering information from a cross section of program offices that had a range of experience in implementing best practices. However, we found that a program office that had little or no experience with a new practice was not in a good position to comment on the training needed to implement the practice. Consequently, we selected program offices that DOD considered to be leaders in one or more of the five practices. 4 See Best Practices: DOD Can Help Suppliers Contribute More to Weapon System Programs (GAO/NSIAD-98-87, Mar. 1998). Page 22 GAO/NSIAD-99-206 Best Practices Chapter 1 Introduction We based our selections on extensive consultation with DOD officials to ensure that the program offices had experience in implementing the practices and thus were in a good position to comment on the training resources that enabled them to implement the practices. (See app. II for a description of the programs.) These program offices represented best case examples in DOD for specific practices, although none of the programs was considered a leader in all five practices. The program offices selected were the Advanced Amphibious Assault Vehicle (AAAV), Advanced Medium-Range Air-to-Air Missile (AMRAAM), Battlefield Combat Identification System (BCIS), Joint Air-to-Surface Standoff Missile (JASSM), Joint Strike Fighter (JSF) and Joint Surveillance Target Attack Radar System-Joint Tactical Terminal (JSTARS-JTT). Table 1.2 shows which of the program offices were considered leaders for the different practices. Table 1.2: Best Practices Evaluated at Program Offices Performance Past Supplier CAIV IPT specifications performance relationships AAAV X X X AMRAAM X X BCIS X X JASSM X X X JSF X X JSTARS-JTT X X Source: GAO analysis of DOD information. At the program offices, we used structured questions to interview the key people responsible for implementing an individual practice. Through the interviews, we determined the various sources they used to develop the knowledge needed to implement best practices and the extent to which the training DOD provided to the program office contributed to this knowledge. To assess strategy and the methods DOD uses to train program office staff on the use of best practices, we concentrated on the DAU training organizations and resources established by DAWIA to provide for the professional educational development and training of the DOD-wide acquisition workforce. We also assessed how these resources met program office needs in implementing best practices. In addition, we assessed other Page 23 GAO/NSIAD-99-206 Best Practices Chapter 1 Introduction acquisition training sources, including service and agency specific schools and the services’ acquisition reform offices. However, our report focuses on the training approach for best practices and is not intended to describe all DOD training practices. We met with the Director of Acquisition, Education, Training, and Career Development; the President of the DAU; officials of the Defense Systems Management College; and representatives from the Office of the Under Secretary of Defense for Acquisition and Technology. We also met with and collected data from officials at the Air Force, the Army, and the Navy. We reviewed the organizational structure and responsibilities for DAU consortium members and service organizations, the process for establishing the DAU training curriculum, and survey data on DOD training recipients. We evaluated the strategies and methods of commercial firms recognized for their training excellence by examining how these leading companies used training to implement new practices at their program management organizations. To identify these companies, we conducted literature searches, consulted with and collected data from professional associations, and spoke with university faculty specializing in corporate organizations. We visited the following four companies recognized as being leaders in the area of training: • The Boeing Company is the largest manufacturer of commercial jetliners and military aircraft with 234,000 employees worldwide. Boeing’s Employee Training and Development organization is responsible for training all Commercial Airplane Group employees, approximately 97,000. There are 500 to 600 employees in the office, with roughly 300 serving as trainers. • Ford Motor Company is one of the largest U.S. manufacturers of automobiles, trucks and provider of automotive services with 345,000 employees worldwide. Ford’s Product Development Process Leadership organization was created to provide assistance, including training, to engineers implementing the Ford Production Development System (FPDS). • International Business Machines (IBM) Corporation is one of the world’s top providers of computer hardware and software with 290,000 employees worldwide. Different internal organizations provide training that serves employees worldwide, including Learning Services and the Center for Excellence. • Motorola is one of the world’s leading providers of wireless communications, semiconductors, advanced electronic systems, and services with over 150,000 employees worldwide. Motorola University Page 24 GAO/NSIAD-99-206 Best Practices Chapter 1 Introduction has a staff of over 600 operating through account managers that are assigned to each business unit to provide a “one-face” education and training contact. These companies, recognized as industry leaders, place strong emphasis on training. At these companies, we reviewed company documents and training data and met with individuals responsible for designing and developing programs to educate and train employees on major new practices. We also met with representatives from major program offices who were involved in key training decisions and were recipients of the training. Our report highlights the best commercial training approaches for implementing key new practices. As such, they are not intended to describe all commercial training practices or suggest that commercial firms are without flaws. Finally, we reviewed several studies on DOD’s training organizations and methods. We used these, as well as the previous information and analysis, to determine the extent to which DOD’s proposals to reshape DAU and continuous learning policy held potential for better delivery of training to foster implementation of best practices by program offices. We conducted our review between April 1998 and June 1999 in accordance with generally accepted government auditing standards. Page 25 GAO/NSIAD-99-206 Best Practices Chapter 2 DOD Training Is Not a Major Catalyst for Best Practices Chapte2 r Key officials from weapon system programs at the forefront of implementing best practices did not find that standard DOD training offerings provided the information they needed to apply the practices to their programs. In evaluating their key sources of knowledge for implementing best practices, none of the program officials ranked required DOD training first, with many ranking it last. DOD training either did not reach the right people when it was needed or did not reach them at all. When training on best practices was received, it did not contain the depth or practical insights program office staff needed to implement the practices. It was primarily through their own efforts—learning on the job, finding external training, or developing their own training program—that they attained the knowledge needed to apply best practices. Programs that became leaders in applying best practices did so primarily because their managers realized that the practices were key to the programs’ success. In so doing, the managers were able to identify what knowledge they needed to apply the practices. Thus, their success depended on having the foresight to see what was needed, the ingenuity to find good sources of knowledge, including training, and the resources needed to attain that knowledge. Replicating this approach broadly on other programs is problematic. Other managers may not realize the significance of a practice to the success of their programs and the need for additional training. Also, they may be uncertain about testing new initiatives on their programs. Some may recognize a practice’s importance but be unable to fund their own training efforts and be left relying on standard DOD training. DOD Training Did Not For training to facilitate the adoption of a new practice, it must be received by those responsible when they are ready to begin implementing the Reach the Right People practice. Program officials stressed that on a new initiative or practice, at the Right Time training must begin when a practice is to be introduced. If training is provided too late, people will be forced to devise their own means of getting the knowledge needed to begin implementation or risk improper implementation. If training is provided too soon, knowledge could fade before it is applied at the workplace. For the programs we reviewed, those responsible for managing or implementing best practices noted that in some cases, training did not exist when they began implementing the practices; in other cases, the training was received too late for the job at hand. Some were missed altogether by DOD training offerings because they were too senior to be required to take courses, training was not Page 26 GAO/NSIAD-99-206 Best Practices Chapter 2 DOD Training Is Not a Major Catalyst for Best Practices provided locally, or they fell outside the definition—and training curriculum—of the acquisition workforce. Training Came at the Wrong Program office officials believed it is essential that training on new Time for Some People initiatives accompany implementation to explain what the initiatives are and to come up with a common understanding and way to tackle program-specific issues. For the program offices leading the way in best practices, officials reported relevant DOD training was not offered at the time implementation began. For example, the BCIS program applied performance-based specifications soon after the 1994 DOD initiative was announced. However, specific training on performance specifications had not been developed. The AAAV program had already implemented performance specifications without DOD provided training. A top manager for JSF similarly reported that there was no training for program office staff when they started to implement CAIV. There were no guiding documents and no one, including the training community, knew what CAIV was. Similarly, on the JSTARS-JTT program, CAIV training was not available when the program office began applying the practice in 1995. Now, some training is available, but it was not when it was needed for the program. Nor was training available when the JASSM program office began assessing contractor past performance. Training could have helped avoid a protest of the prime contract award, according to an official. Page 27 GAO/NSIAD-99-206 Best Practices Chapter 2 DOD Training Is Not a Major Catalyst for Best Practices Figure 2.1: JSF Implementation of CAIV on the JSF program began with a clean slate—no advance training. Source: DOD, artist rendition. Training can also come too early. A program official questioned the staff’s ability to retain the information when training is not provided at a practical time for the assignment. For example, he noted that symposiums are good Page 28 GAO/NSIAD-99-206 Best Practices Chapter 2 DOD Training Is Not a Major Catalyst for Best Practices ideas, but people may not have an opportunity to apply the ideas at the time and may not remember when they need to. Key People Were Missed by Applying best practices on a weapon system program involves reaching not Training on Best Practices only program office staff but also other members of the acquisition workforce, DOD people outside the acquisition workforce, and contractors. Experience on the programs we reviewed shows that it is hard to reach all of these people with best practices information through standard DOD training. The promotion of many best practices took place after the implementation of DAWIA course requirements so that senior people have not taken formal acquisition training that includes exposure to best practices. Several of the senior program officials we spoke with did not receive training on new practices because they had not taken courses that incorporated best practices. These officials had been acquisition officials for many years before DAWIA and were grandfathered into their certification level. Their job experiences were applied to meet professional certification requirements and thus they did not have to take any DAWIA training. For example, an official who helped establish JSF performance specifications noted that he had his last class 3 years earlier, a contracting refresher course. The course may have had a short section on performance specifications, but he could not remember it. Training misses some program office people because it is not offered locally. According to program officials, training needs to be conducted at the local level by subject matter when it is needed. This is especially true for small programs, for which it is difficult to spare people from the programs for long periods. One official from a small program said that although senior management stresses the importance of training, it is difficult to attend classes that run over 14 weeks. He added that these classes have not been offered locally. Another program official asked why DOD training organizations do not bring some of the harder to get courses to the field, as is done by private organizations. Page 29 GAO/NSIAD-99-206 Best Practices Chapter 2 DOD Training Is Not a Major Catalyst for Best Practices Figure 2.2: JSTARS-JTT JSTARS-JTT officials believe training could be improved by being provided on-site. Source: DOD. Training is not reaching people outside program offices that also play key roles in successfully implementing best practices. For example, program offices are typically supplemented with people from separate functional organizations, such as engineering directorates. These people may not receive training on new initiatives in their home organizations and can be unfamiliar with the initiatives when they come to a program. People that set requirements early in the process and those in logistics must also be knowledgeable and committed for practices to be successful. Exercising flexibility in requirements, for instance, is critical to the success of CAIV and performance specifications. The need for training was extended further to those that play a role in a weapon system’s approval. Officials from one program reported that they had worked with a prime contractor to streamline contract reporting requirements down to four items, in line with acquisition reform. However, based on a review by a separate office with approval authority over the contract, 40 contract requirements were Page 30 GAO/NSIAD-99-206 Best Practices Chapter 2 DOD Training Is Not a Major Catalyst for Best Practices added, returning the contract to traditional reporting methods. The program official said that the reviewing official did not know what the program office was trying to accomplish and did not care. Prime contractors and subcontractors are also essential to the application of best practices but are not part of the DOD defined acquisition workforce or otherwise targeted for training on best practices. Several program officials we met with believed that some means of informing contractors is needed to make them aware of important DOD initiatives. Contractors do not necessarily understand or know how to implement new DOD initiatives; for example, they may not know how an integrating IPT or overarching IPT works. While DOD is not responsible for training contractors, it does have to ensure that contractors understand best practices, as well as give them an opportunity to help shape how these practices are applied on programs. The AAAV program management ensured their prime contractor was knowledgeable of best practices by requiring training for contractor and program office staff in the prime contract. DOD Training Program officials reported that standard DOD training did not prepare them well for implementing the practices at the workplace. DOD’s training Offerings Did Not typically provided only an awareness of the practices, not the knowledge Provide the Depth that is needed for actual implementation. Also, the training was not tailored to allow program offices to see how individual practices applied to Needed to Implement their specific programs. Further, program officials noted that trainers did Best Practices not have the practical experience to share and were not up to date with the most recent examples of programs that had implementation experience. Program officials stated that they need to go beyond the theoretical concepts covered in most training courses. They believed that the “how to” is missing on all the initiatives and that they need to know how to move from traditional practices to the new practices. DOD courses were described in general as too esoteric and not relevant to the tasks at hand. For example, a JASSM official noted that a contractor’s past performance is often equated with DOD’s Contractor Performance Assessment Rating System, but it involves much more. The JASSM program office evaluated contractors’ past products, and in doing so, learned about the quality of their design, management, and production processes. Program officials said they went well beyond the performance assessment rating system covered in standard training courses. Nonetheless, they said that they could not avoid a protest by the losing contractor. They believed they Page 31 GAO/NSIAD-99-206 Best Practices Chapter 2 DOD Training Is Not a Major Catalyst for Best Practices needed guidance on how to collect past performance information, which was not covered by training. Program officials believed, overall, that training should be designed more for the customer because acquisitions are unique and that programs may have different implementation issues based on program size, stage of the acquisition, or type of weapon. A senior AAAV official said general courses can provide information about individual practices, but to implement a practice, the training must be tailored to the program. JSF and BCIS officials concurred. Some program officials have observed that training courses overemphasize the application of new practices for larger and newer programs and believe that not enough coverage is given to teaching how the new practices apply to programs that are smaller or older. For example, officials noted that newer programs have an advantage in starting with a clean slate and that training should also show how the practices apply to older programs and the benefits to be obtained. Figure 2.3: AMRAAM AMRAAM officials believe training should help program managers in deciding how initiatives apply to their particular circumstances. Source: DOD. Page 32 GAO/NSIAD-99-206 Best Practices Chapter 2 DOD Training Is Not a Major Catalyst for Best Practices Program officials also found the presentation methods for training on best practices were not helpful in applying the practices in the workplace. They believed that instructors lacked practical experience and current information. For example, a JSF manager reported that the executive course he took in 1997 on CAIV was not valuable because the instructors had no experience with CAIV and did not know how to explain it. Another program official thought that instructors were left in place too long and had only old experiences to share. Officials noted that course material was frequently out of date and sometimes incorrect. For example, one JASSM program official attended an engineering course in which the instructor’s answer on CAIV was wrong. Another said the acquisition reform example the instructor used—from the official’s own program—was about 2 years behind what was being applied on the program. The official brought up more current examples of practices being used on the program, but the instructor did not want to discuss them. Training also suffered from limited use of case studies that would allow students to see how an initiative in the abstract might apply to their own programs. Officials from several programs added that current case histories should show the application of acquisition principles in a program context. Case studies would include the successes, as well as the pitfalls and solutions. The case studies should be designed for the customer, another said, and be applied at multiple levels. For example, programs could benefit if training allowed students to play off the risks and benefits of how new initiatives apply to their particular program. Real case experiences help others visualize how practices could apply. BCIS officials reported that during the presentation of their case study for the 1998 Acquisition Reform Week, they realized, as they explained what they did to implement initiatives, that the audience gained better insight on the issues. Another official said he shared his program’s experiences with those in his class, but those in other classes would not have the benefit of his experiences. Program officials did not believe that there was an effective means for providing feedback on the quality and usefulness of courses, such as the need to update course materials, or course relevance. They would like to see the students—the practitioners—have more impact on what adjustments are needed to courses. The only means of feedback program office people were aware of was the end of class survey form. However, these surveys did not allow students to give in-depth feedback or ask the questions that got to the larger issue of whether students’ training needs were met. Page 33 GAO/NSIAD-99-206 Best Practices Chapter 2 DOD Training Is Not a Major Catalyst for Best Practices Limitations of In evaluating their key sources of knowledge for implementing best practices, none of the program officials ranked required DOD training first, Standard Training Led with many ranking it last. To obtain the knowledge needed to implement Program Offices to key practices, officials in leading programs developed their own solutions. They had a vision of what they wanted to accomplish and they devised a Develop Their Own variety of methods, such as funding training beyond standard DOD Training Solutions offerings, creating their own internal training programs, and learning on the job. Strong program managers, supported by the executives above them and strong working relationships with their contractors, were key to the implementation of the practices. Program managers of the leading programs cautioned, however, that not all officials have the vision or the resources to mirror this approach. Implementation Success All the leading programs had at least one element in common—strong Tied to Vision and Support leadership committed to implementing practices that would help their programs succeed. Leaders were described as having vision and knowing at All Levels what they had to do to realize that vision. This included accepting the risks for trying new approaches. For example, the original AAAV program manager conceived of the unique approach of collocating the program office and the contractor because he thought it was essential to making IPTs work. IPTs became the backbone of the AAAV program. Consequently, team training was viewed as so important that it was incorporated into the prime contract as a requirement. Support for collocation and the team approach came from top management of both the contractor and the program office. According to program officials, it does not make any difference how good the training is without management support; junior people can come back from training with new ideas and have them go nowhere if they do not get management support. Page 34 GAO/NSIAD-99-206 Best Practices Chapter 2 DOD Training Is Not a Major Catalyst for Best Practices Figure 2.4: AAAV Training for AAAV was tailored to support contractor and program office collocation. Source: DOD. In turn, managers point to the support that they receive from their senior management as an important factor in their ability to be successful. One official described his manager as aggressive in his efforts to pave the way for trying new practices. To do that, the senior manager said that he is willing to go up against the established bureaucracy and provide cover for his program managers to try new things. One program manager cited a personal commitment to a reform-minded DOD official as part of his motivation for making acquisition reform work. He said that with his bosses’ support, he can do what he thinks is right on his program, such as making past performance central in selecting a prime contractor. However, he noted that other managers have not had that ability because they are not supported when they propose doing things differently. Program managers believe that a key element for adopting best practices was developing trust or partnership with the contractor. An AMRAAM manager believes the ability to make radical program changes has Page 35 GAO/NSIAD-99-206 Best Practices Chapter 2 DOD Training Is Not a Major Catalyst for Best Practices depended on working closely with the prime contractor, and seeing that the same relationship exists between the prime contractor and subcontractors. The JASSM office organized its IPTs to mirror the structure of the contractor’s IPTs, rather than along functional lines like most offices. A top JASSM official credited the high level of trust between the program office and the contractor, along with the program office’s creativity and innovation, for enabling their use of new practices. This perspective is essential to managing programs in today’s environment and to make teams work, another official noted, but it is not taught in standard DOD training offerings. More managers would attempt to apply best practices if training encouraged it, a program official said, particularly the training provided for junior program managers. More general training is needed to support a new program environment for adopting new practices that goes beyond the current training emphasis on the mechanics of management and theory. Program officials reported that new program manager training as recent as 1997 did not reflect the new approach they have practiced on their programs. One believed that it was a lost opportunity for DOD to not impress upon new managers their role in adopting new practices and the potential benefits. Leading Program Officials With the support of senior management, program officials used a mixture Innovated to Get Needed of strategies—generally outside of standard DOD training offerings—to get the knowledge they needed to apply best practices. A number of programs Training used their own resources, including cumulative job expertise of the staff and personal research to learn how to implement new initiatives. For example, JSF officials stated they learned how to apply CAIV by organizing warfighters, engineers, and analysts together in a team. They learned as they went, developing materials on their own. The AMRAAM program had a young staff member gather information on CAIV from various sources such as the internet and conferences. From that starting point, they learned as they went along. In setting performance specifications, it was the personal experience of AMRAAM officials, coupled with commitment, that allowed the specifications to be set at a performance, rather than detailed, level. Despite their own experiences, program officials did not recommend the learn-as-you-go approach; everyone should not have to reinvent the wheel. Programs officials used their own program funds to go beyond standard DOD training by sending staff to nongovernment training or to bring Page 36 GAO/NSIAD-99-206 Best Practices Chapter 2 DOD Training Is Not a Major Catalyst for Best Practices experts in. For example, frustrated with DOD training, the JASSM program manager sends people to outside training, such as Harvard leadership courses, to have them learn and grow beyond the basic training and develop creative and innovative thinking. He has his staff take 80 hours of general, nonfunctional training. Another manager said some of his best training was from off-site sessions sponsored by JASSM that dealt with people issues that were critical to making IPTs work. JSF officials took a private sector course on performance-based specifications and used a model developed by their contractor. BCIS officials hired an outside firm to teach program officials on two occasions, both of which included a CAIV component. Figure 2.5: JASSM JASSM officials sponsor training to develop staff leadership skills. Source: DOD. AAAV officials developed their own training program tailored to the program’s work environment and training needs. A key characteristic of the program is the collocation of government and contractor program Page 37 GAO/NSIAD-99-206 Best Practices Chapter 2 DOD Training Is Not a Major Catalyst for Best Practices officials, making team dynamics an important factor. The prime contract specified that teams must be used, and IPT training was charged to the contract. The contractor hired a third party to develop a training program tailored to the AAAV program, and program office, contractor, and subcontractor staff were taught together, on-site as a team. Joint training was used to establish a common culture for participants. Training was given one time to the team initially and then to every new person. The team training started with mandatory 10 hours of IPT training, with sessions covering the AAAV program, trust, communications, and other team dynamics. CAIV training was added as part of the training for implementation within the team context. The team approach to training was supported by other leading programs as well. On the JASSM program, the manager reported that he just had his entire office (about 30 staff) take 1 week of leadership training, which included topics such as stress management, critical thinking, and decision-making. JSF officials took the “train the trainer” approach and developed a team of experts within JSF and then the experts trained everyone that would be involved in implementing performance specifications. The experts used multiple sources to train themselves, such as published guidance and talking with their peers. They then developed basic guidance for the staff, such as engineers, who needed a common understanding on how to write requirements at a performance level. Other Programs May Lack Program officials reported that they were fortunate to have staff that could Leading Programs’ Ability to use their collective experiences to work through problems in implementing initiatives. However, they noted that not all programs will have the same Innovate advantages. JASSM officials said top service officials handpicked program officials for the program team because their personal characteristics supported flexibility and creativity. Similarly, JSF officials reported that the program office was staffed with multifaceted people, as well as strong, senior management support and upfront money for training. They noted that smaller programs may be staffed by junior officials with less experience to draw upon or be unable to devote staff to research information on how practices might apply to their situation. Limited funding may also be an issue for some programs. As one program official said, his program office could pay for external training, but for many programs, the cost would be an impediment since training is one of the first items cut in a program budget. Consequently, smaller programs might need to rely more heavily on standard DOD training as their main source of information. Page 38 GAO/NSIAD-99-206 Best Practices Chapter 3 DOD Training Methods for Best Practices Do Not Go as Far as Leading Commercial Firms Chapte3 r Leading commercial firms and DOD apply different training methods to implement key practices. Commercial firms use targeted, hands-on methods that include conducting a front-end analysis to determine the teams’ training requirements and regularly involve those implementing the practices in making important training decisions to ensure program teams are trained on key practices. Thus, the training is customized to meet the specific needs of the teams. These methods also involve providing many hours of training—beyond standard skill-based or functional training— focused on the implementation of a single practice. Company officials believe the targeted approach results in more useful training—improving the personnel coverage, course depth, and timing—which helps to improve outcomes of the final product. In contrast, DOD training on best practices is delivered through traditional DAWIA certification courses and vehicles such as videos and computer-based training, which are limited in reaching the right people at the right time and in providing the needed depth to implement best practices. DOD does not have a counterpart to the commercial method of providing customized, hands-on assistance to support program office staff and other implementers of key practices. Although exceptions exist, there is no systematic effort—or responsible organizations—within DOD to directly assist key implementers to use new practices. Further, DOD does not have a comprehensive means for allowing program staff and others to influence training decisions in a way that could improve the relevance of training. Commercial Firms Use For routine training, such as skill-building, leading commercial firms provide standard training offerings, including functional area courses and Targeted, Hands-on instruction on corporatewide issues, such as communications or ethics. Methods to Improve However, when implementing key new practices—such as those that change product development and production—the firms go beyond the Training Usefulness standard training offerings. Commercial firms use a targeted, hands-on training approach to ensure program teams are in a good position to implement a new practice. They provide numerous hours of training, typically through a single company organization, targeted to the implementation of a key practice. The practice-specific training hours are targeted to the program teams most likely to implement the new practice. Page 39 GAO/NSIAD-99-206 Best Practices Chapter 3 DOD Training Methods for Best Practices Do Not Go as Far as Leading Commercial Firms The elements common to how the four leading firms provide training on a key initiative or practice include • a front-end analysis of program teams’ needs and training requirements; • involvement of program teams in key training decisions; • customized training to meet program teams’ specific needs; • targeted training for the implementation of specific practices; and • improved training outcomes, including better course depth, timeliness, and reach. The training organizations of leading commercial firms conduct a front-end analysis to determine the needs and training requirements of program offices implementing new practices. The analysis is also used to identify and address barriers each program office faces when implementing new practices. According to the Director of the Benchmarking Forum for the American Society for Training and Development, this type of analysis is crucial for an organization to be able to institute performance-improving measures. Using information from the front-end analysis, the training organizations customize the training to ensure that it directly assists program teams in implementing new practices. One company official told us that training is costly and when it misses the mark, the company pays a big price. Given the importance of training when implementing a key practice, company officials believe that it is crucial to ensure that the training is beneficial to the key implementers of the practice. To ensure that the training will address the needs of the program teams, the training organizations involve the staff in making important training decisions. Program staff help decide the amount of training to be provided for certain job positions, course objectives, and depth of course coverage. Company officials believe their training approach, which includes program staff, has resulted in the right amount of course depth, timeliness, and coverage of personnel in the commercial firms. Following are descriptions of the training methods employed by companies on key initiatives or new practices. The Boeing Company Officials from Boeing’s Employee Training and Development organization state that their primary goal is to support their customers—employees assigned to the Commercial Airplane Group. The training representatives develop a partnership with the staff from the beginning of the program to design and manufacture a new airplane. The training representatives form “drop teams” to collocate with the program to conduct a front-end analysis and learn as much as possible about the business process and the staff’s Page 40 GAO/NSIAD-99-206 Best Practices Chapter 3 DOD Training Methods for Best Practices Do Not Go as Far as Leading Commercial Firms concerns. The analysis allows the drop team to determine what training is needed to support the staff implementing new practices. Boeing training officials said they worked side by side with the program staff to create a training program that provided team building and conflict resolution techniques and technical skills training that specifically focused on improving work competencies that would change as a result of the 777’s new digital environment. To ensure all 777 staff were equally trained, employees were required to complete training before they reported to the program. For example, the professional employees—engineers and drafters—were required to complete 120 hours of start-up training on several key 777 practices, including design build teams and computer-aided three-dimensional interactive application1 software. Teams were often trained together at the work location. Boeing officials stated that training was instrumental to the implementation of key practices on the 777 program, such as design build teams—essentially IPTs. The officials stated that design build teams were at odds with the company’s culture because employees were not accustomed to working in a team environment and sharing information across functional areas. 1 This application is a computer-based design tool that allows designers the opportunity to view design drawings and the interface of the millions of airplane parts as three dimensional. Page 41 GAO/NSIAD-99-206 Best Practices Chapter 3 DOD Training Methods for Best Practices Do Not Go as Far as Leading Commercial Firms Figure 3.1: 777 Airplane Boeing’s use of design build teams created a major culture change for the staff assigned to the 777 program. Source: The Boeing Company. Boeing officials believe their partnership approach improved their training. Training representatives stated that the partnership resulted in program management support, which ultimately led to acceptance from the program staff. The representative stated that since Boeing has involved program staff in decisions regarding training, course “no-show” rates have decreased. A senior manager for program operations for the 777 program stated that because he and other senior program leaders drove key training decisions, the training was tailored to the staff’s needs and provided the necessary skills and orientation to work in the new environment. Page 42 GAO/NSIAD-99-206 Best Practices Chapter 3 DOD Training Methods for Best Practices Do Not Go as Far as Leading Commercial Firms Ford Motor Company The Ford Motor Company created an organization, Product Development Process Leadership, with the singular purpose of supporting its 100 program offices in designing new or modifying vehicle lines in the implementation of the FPDS--a lean engineering process. The organization provides internal communication regarding FPDS to the teams, and its training representatives work with the teams to conduct an analysis to learn first hand if impediments to FPDS implementation exist. Other support, such as team coaching, is provided to facilitate the engineering team’s implementation of FPDS in the workplace. Figure 3.2: Ford Focus Ford uses training to improve the timeliness and quality of new vehicle launches. Source: Ford Motor Company. Ford officials told us that their training focus is to provide practical skills just in time—that is, when it coincides with the need to apply the skills on Page 43 GAO/NSIAD-99-206 Best Practices Chapter 3 DOD Training Methods for Best Practices Do Not Go as Far as Leading Commercial Firms the job. Ford provides over 50 hours of training to instruct engineers on how to do their jobs using the FPDS process. To that end, Ford uses internal subject matter experts—engineers who have been on teams designing new or modifying existing vehicle lines—to help tailor FPDS course topics to make them relevant to the work environment. Ford officials stated that the subject matter experts understand the details of the FPDS and are needed to ensure that the training developed is practical at the working level. According to the manager responsible for training employees to launch new vehicles at Ford, his office pulls together the training that is necessary to get the job done. To further improve practicality and timeliness, the manager stated that on-the-job-training at the worksite is used to the extent possible. IBM IBM’s Center for Excellence provides in-house education consultants to personally understand the business situation and product. The consultants interview business unit staff to assess the staff’s training needs and identify inhibitors to implementing new practices. The assessments can take 2 weeks to 4 months, depending on the size of the business unit. An IBM trainer for Object Orientation Project—a companywide software development practice—explained that providing practical training is one of the Center’s guiding principles. To achieve this goal, the company first stresses the importance of using experienced instructors who have practical knowledge in “doing” what they are teaching. Second, employees are assigned to an Object Orientation Project before they take the 5- to 7-week immersion training course. According to the IBM trainer, this requirement has improved practicality because the students have better knowledge retention, as opposed to having to wait 6 months to apply the information. Lastly, to further enhance practicality and relevance, IBM integrates case studies with real examples related to the student’s next assignment into the Object Orientation Project training. According to an official from IBM’s Center of Excellence, most training is provided and tailored to entire work teams or, at a minimum, to individuals with common responsibilities. Motorola University Motorola University focuses on providing training and education solutions for its business units. The university recently began to assign an account management team to consult and advise senior leadership for each business sector in order to anticipate and provide appropriate training. The management team partners with the business unit staff to identify their training needs for implementing new practices, such as Five Nines—an effort to improve product reliability to the level of 99.999 percent. The account team works with the business units to assess their training needs Page 44 GAO/NSIAD-99-206 Best Practices Chapter 3 DOD Training Methods for Best Practices Do Not Go as Far as Leading Commercial Firms and develop a plan to meet those needs. This partnership enables the university to customize its training to the specific needs of the various types of engineers, such as software engineers, within the business units. A software engineer stated that the university provides separate training courses tailored to meet the different and often distinct needs within the engineering community implementing Five Nines. For example, a software engineer could receive up to 120 hours of training targeted to implementing Five Nines. Although the account management structure is relatively new, the engineer observed that most business unit staff are reacting favorably to the effort because the coordinated approach provides one-stop shopping. DOD Does Not Target DOD does not have a counterpart to the commercial hands-on approach for directly assisting key implementers of the best practices. DOD relies Training on Key primarily on its standard training, including DAWIA courses, augmented by Practices to Program videos, internet-based training, satellite broadcasts, and roadshows, to inform staff on best practices. These venues were designed to focus on Offices functional training and to increase the awareness of new practices. As such, they do not provide the necessary depth or reach enough of the right people at the right time to be of help in implementing best practices at program offices. Responsibility for training on best practices is diffused among several DOD organizations, including DAU and the service acquisition reform offices. We did not find an organization that was able to tailor and help deliver training on best practices to the program offices we visited. Furthermore, DOD does not systematically involve program office staff and other implementers in key decisions regarding best practices training. Currently, no feedback mechanism exists to determine the effect of DOD’s training on the implementation of new practices at the program office level. DAWIA Training Not Well DAU training is designed primarily for employees seeking career level Suited as a Primary Means certification in the acquisition workforce, as required by the DAWIA standards. DAU incorporates best practice topics into the DAWIA courses for Conveying Best as drop-in modules that provide a survey of the topic. The information Practices conveyed is enough to provide a general awareness of the concept but not enough to implement the practice at the workplace. While this approach may provide sufficient information for the target audience—newer acquisition employees—it has inherent limitations when it comes to providing best practices’ implementation training to the entire acquisition workforce. For example, DAU courses have been developed by functional Page 45 GAO/NSIAD-99-206 Best Practices Chapter 3 DOD Training Methods for Best Practices Do Not Go as Far as Leading Commercial Firms boards that teach skill-based competencies for functional career fields versus best practices. As a result, DAU courses are primarily aligned along career fields, such as engineering and cost estimating. Because personnel assigned to a particular functional area are given priority for training in that function, program officials told us that it is difficult for personnel outside of their areas to take courses. On one program, an official noted that because of limited space, only 5 to 10 percent of the program staff can take training outside of their functional areas each year. One program manager believed that the functional training, while important to career fields, was no longer as relevant to the role of today’s program office. He noted that the combination of best practices, delegation of key responsibilities to contractors, and fewer staff has altered the program office’s role. In his opinion, the program office used to be closely involved with managing the design of a weapon system and double-checking the prime contractor’s work. In this role, the program office was organized and operated along functional lines, and functional training was relevant to how a program office operated. He observed that today the program office is not as involved with the designing of the weapon system, nor is it able to mirror the contractor’s functions. Rather, the program office must be expert at what the government can control— which the program manager referred to as key leverage points. These leverage points include the requirement trade-off process, the selection of a prime contractor, and the establishment of key relationships that enable the program office staff to have insight into the contractor’s actual progress. Program office staff, working in an IPT environment, must have knowledge of multiple career fields and work in a cross-functional setting. As a result, he believes that functional training alone no longer covers the things most critical to a program manager. The kind of training that is needed—such as on the leverage points—must be obtained elsewhere by the program office. The usefulness of the DAU courses is further hampered by limited availability, which restricts program offices from receiving training when needed. According to a DAU school representative, the consortia of schools can train about 10 percent of the workforce each year. Furthermore, more senior staff may have limited exposure to best practices because the majority of them have already met training requirements and are not taking the certification courses that introduce best practices. While these staff are not prohibited from taking certification courses as part of their continuing education requirements, availability is limited and priority is given to individuals seeking Page 46 GAO/NSIAD-99-206 Best Practices Chapter 3 DOD Training Methods for Best Practices Do Not Go as Far as Leading Commercial Firms certification. These inherent limitations are consistent with the training shortfalls noted by program office staff in chapter 2. Other Training Methods Are DAU’s Acquisition Reform Communication Center (ARCC) is a key avenue More Dedicated to Best for disseminating information on the best practices to the acquisition workforce. ARCC provides training through videos and periodic satellite Practices but Are broadcasts on a variety of best practice topics. However, ARCC does not Awareness-oriented track attendees and has no assurance that the workforce adequately receives the training. For those that do attend, the introductory nature of the training may not provide the depth or specificity to implement the practice at the workplace or in a time frame that is helpful. The acquisition reform offices in the services communicate best practice information through acquisition reform courses, periodic satellite broadcasts, and informational videos, which are sufficient for broad exposure on best practices but again are of limited depth for practical application. Roadshows, traveling multiday training workshops provided to staff at a number of locations, and Acquisition Reform Week, designated by DOD as an opportunity for all service organizations to cease their normal operations and focus on acquisition and logistics reform initiatives, are also used to provide best practices training to a wide range of the acquisition workforce. These methods also have limitations in depth and workforce coverage. For example, roadshows typically provide awareness training on the practices and do not provide in-depth information needed for implementation at the workplace. A program official believes that only 10 to 15 percent of the acquisition staff attend the second day of roadshow workshops, where more detailed training is provided. The annual Acquisition Reform Weeks also provide awareness level training. Neither method is tailored to specific program offices or provides assurance that it is delivered at the time most needed by the workforce. DOD Training Organizations DOD does not have organizations that are comparable to those in Not Set Up to Help Design commercial firms and that work with program offices in identifying best practices applicable to a particular program, designing an approach to or Deliver Tailored Training training the program office staff, or delivering the training to the program to Program Offices offices. DOD training organizations—those that deliver DAWIA and acquisition reform training—are not set up or have the resources to deliver best practices quickly and easily to program staff, and then ensure implementation at the working level. Page 47 GAO/NSIAD-99-206 Best Practices Chapter 3 DOD Training Methods for Best Practices Do Not Go as Far as Leading Commercial Firms The DAU consortium integrates best practices topics into DAWIA certification or acquisition reform training but does not develop training specifically for implementing the practices. Although DAU has a substantial full-time faculty, the faculty members’ main priority has been to teach in the classroom. One course director informed us that faculty members are evaluated on the basis of hours of training provided in the classroom, which implicitly discourages work outside the classroom, such as consulting with program offices. Also, little or no consultation occurs between the course designers and the implementers on how to best implement a practice at the program office level. The Director of DOD’s International and Commercial Systems Acquisition unit stated that DOD’s current structure for defining and developing training courses does not have clear accountability to ensure that training on best practices is provided at the program level. Several program office staff informed us that they have not been given the opportunity to provide input to influence the content of the courses. Instructors for the courses, who do have frequent contact with program office staff, also believed that their ability to influence the content of the courses was limited. DOD does use internet site surveys to obtain staff feedback on acquisition reform training, which could help tailor the courses more to the needs of the program offices. However, program office staff believed the survey forms were inadequate for addressing the specific training needs of individual program offices. Organizations with more direct responsibility for providing training on best practices are staffed to the level needed to design distance learning courses, such as web-based training, satellite broadcasts, and instructional videos. For example, the services’ acquisition reform offices each employ from 8 to 20 people, not nearly enough to provide on-site training or in-depth consulting to individual program offices. Some DOD organizations have recognized the value of providing training tailored to specific program offices and are attempting to go beyond traditional training roles. However, these are largely ad hoc efforts that do not go as far as commercial methods. For example, the Navy is creating Total Ownership Cost teams, that are to advise program offices on the use of CAIV. Navy officials stated that due to limited resources, the teams will only reach about 15 of the 300 program offices per year. The Air Force plans to begin using training support teams to address best practices issues from the program office perspective. However, the teams have not yet begun and full operation is not scheduled to begin until fiscal year 2001. Also, an Air Force official stated that another team, created to address acquisition reform training issues, will not involve program managers, which raises the question of whether program office needs will be adequately considered. Page 48 GAO/NSIAD-99-206 Best Practices Chapter 4 DOD and Commercial Training Methods Reflect Different Strategic Approaches to New Practices Chapte4 r The intensive, hands-on training methods leading commercial firms employ on new practices are the result of a systematic, institutionally driven approach to implementation. These firms commit to and concentrate their resources and attention on a few well-defined practices and make a significant front-end investment in providing the training to the primary implementers. Additionally, the firms strive to create a supportive environment to put the implementers of the practice in a good position to succeed. Their objective is to have successful implementation at the work level; training methods are shaped to meet that objective. DOD’s training methods for best practices do not benefit from a similarly strategic approach for deploying new best practices and provide little assurance that the practices are consistently implemented in the workplace. DOD has promulgated numerous initiatives in the past few years without communicating their relative priority to trainers and implementers. Often, the initiatives were not accompanied by clear guidance or by the initial training needed for implementation. While DOD commits significant resources to training, it does not make a uniform front-end investment to ensure the implementers of new practices will succeed. This approach depends more on the individual program offices to recognize the need for and to make this investment. Recent DOD actions reflect its recognition that training improvements are needed. Two plans proposing divergent improvements to the training of the acquisition workforce are now being considered. One plan does not focus on best practices training specifically, offering instead an incremental approach to changing the DAU structure. The other targets implementation of best practices directly, with a broad scope of efforts that would require substantial changes from the current approach to training. The latter plan is much more aligned with the commercial approach to training described in this report. However, if it is adopted, it faces many challenges to providing better training for program offices. Another recent DOD action—creation of a continuous learning policy—may provide more training, especially for more senior members of the acquisition workforce, but it does not ensure that best practices will be included. Page 49 GAO/NSIAD-99-206 Best Practices Chapter 4 DOD and Commercial Training Methods Reflect Different Strategic Approaches to New Practices Leading Firms’ Although the four companies we contacted initiated very different types of practices, they exhibited a similar strategic approach to ensuring that the Strategic Approach key practices were implemented. The elements of this approach can be Better Ensures summarized as Implementation of Key • clear vision, Practices • adoption of few practices at any one time, • assessment and identification of target population for implementation, • well-defined goals, • significant investment in training and other vehicles to aid implementation, and • supportive environment to facilitate implementation. In general, we found that the firms shared a common strategy for implementing key practices that were intended to change company culture. First, the firms’ corporate management committed to and adopted seven or less key practices at any given time. One firm we contacted adopted only one or two key new practices, which enabled the company to concentrate its attention and target resources, including training, to the implementation of the practice, and signaled the importance of the practices to trainers and implementers. Second, for practices that were not companywide, the firms assessed which staff should be included in the implementation. For example, Ford’s training unit assessed each existing program team working on vehicle lines beyond the initial development phase to determine if the team could benefit from adopting portions of FPDS or if some older teams should be excluded. An IBM official stated that while the company currently promotes six practices, not all will apply to every segment of the employee population. Third, once the target population was identified, company leaders made the implementation of the practice mandatory. For example, the Ford Motor Company required that all new vehicle lines built for the year 2000 and beyond use FPDS. Lastly, to further assist in the implementation, the companies developed well-defined goals to better ensure that the target population consistently understood how to apply a new practice to improve product production—the ultimate goal. The firms we contacted ensured that the implementers of the practice received the assistance necessary to succeed. Consequently, these companies made a significant investment, including a comprehensive front-end effort, to support the needs of program offices that would implement the practice. According to a Ford training official, if the initiative is important enough to the organization, then the investment is Page 50 GAO/NSIAD-99-206 Best Practices Chapter 4 DOD and Commercial Training Methods Reflect Different Strategic Approaches to New Practices justified. Other companies agreed with that way of thinking. Boeing’s Learning Program Development Director summarized the corporate training strategy for implementing new practices as one that includes a clearly stated vision or mission statement, well-defined goals, and enablers, such as training and good processes, to support the implementers. This philosophy enabled Boeing to take a year to develop the training program tailored to the 777 program—which was intended to change the corporate culture and encourage employees to rethink how they did their jobs. Both Boeing training and program officials believe that the training investment resulted in the successful implementation of the key 777 program practices. While the company officials acknowledged that training was instrumental in the implementation of the key practices, nearly every official we spoke with stated that training is just one of the necessary components. They noted that creating the right environment is key to the successful implementation of new practices and that the quality of training was dependent on this environment. Company officials stressed that strong leadership is often another key. An IBM official stated that, at inception, top leaders need to provide sufficient funding for training, well-defined expectations, clear direction, oversight, continued interest, and incentives to ensure that the new practices are possible to implement. The manager for the 777 program stated that Boeing’s management works in teams—a key practice. He believed that it was management’s ability to lead by example that helped to prevent a return to the former functional way of operating. These companies believe that other factors, such as an accommodating organizational structure, good internal communication, consistent application, and supportive technology, are needed to foster the implementation of key new practices. DOD’s Training on Best DOD does not have a uniform or defined strategic approach to implementing best practices. Accordingly, training methods are not shaped Practices Does Not by the same drive for implementation found in leading commercial firms. Stem From a Strategic DOD’s approach commonly begins with policy memorandums widely distributed to the entire acquisition workforce. However, the policy Approach provides little specific guidance on how or which program offices should implement the practice. Since 1994, DOD has proposed close to 40 acquisition reform initiatives without an indication of relative priority. The absence of priorities makes it difficult to not only develop training courses but also to determine which courses are most appropriate for the needs of program offices. Page 51 GAO/NSIAD-99-206 Best Practices Chapter 4 DOD and Commercial Training Methods Reflect Different Strategic Approaches to New Practices DOD’s Policy on Best DOD’s policy on implementing the best practices is promulgated without a Practices Is Not Coupled strategy or specificity as to whom within the acquisition workforce is expected to implement the practices or which program offices are in the With a Strategy for best position, in a program life cycle, to implement the practices. Typically, Implementation the policy is not promulgated in conjunction with a detailed plan that outlines the support and training required by the program offices tasked with implementing the practice. Similarly, a 1997 Coopers & Lybrand study on acquisition reform implementation found inconsistencies in DOD’s implementation of new practices. It stated that these inconsistencies may have resulted from a lack of an integrated, cohesive DOD strategic plan to put them in context to each other and to larger strategic goals. While DOD has made some progress in establishing such goals through its Government Performance Reporting Act plans, organizations responsible for training and implementing the practices have not experienced the effects. In addition, the policies themselves are not always clear. For example, although the CAIV initiative was promulgated in 1995, training officials reported they did not have a clear understanding of what CAIV means. AAAV program officials said they developed training for 1997 to clarify the CAIV concept for the program. A November 1998 Air Force workshop on CAIV reported that the concept is still not well understood or widely implemented. Similarly, the IPT policy states that IPTs should be implemented “when it makes sense.” The policy for the use of past performance data states that the practice should be implemented to the “degree possible.” While a fair degree of latitude is good for implementers, some DOD officials believe that this level of generality is insufficient to ensure implementation of the practice. For example, a training official observed that policies are not well defined because of the desire to bring new practices quickly to the workforce and so some are deployed when they are not ready for implementation. According to a DAU official, unclear and general policies can contribute to ineffective training because it makes it difficult to develop a course of instruction that teaches a unified, cohesive way to implement a policy. A program official noted that general policy statements provide little guidance or assistance in the program offices’ implementation of the practice, even though program offices are often the primary implementers of best practices. Page 52 GAO/NSIAD-99-206 Best Practices Chapter 4 DOD and Commercial Training Methods Reflect Different Strategic Approaches to New Practices DOD Attempts to In addition to IPTs, performance specifications, CAIV, and past Implement Many Practices performance, DOD has proposed numerous reform initiatives since 1994. In 1997, the Secretary of Defense proposed several new reform initiatives Without Prioritization under the umbrella of the Defense Reform Initiative. These initiatives have been promulgated without an indication of relative priority, leaving educators and implementers to decide on what is important. The lack of prioritization makes it difficult to determine training needs, especially when the number of initiatives continues to increase. While the prolific offering of new ideas can stimulate innovation, the absence of priorities makes it difficult to focus training on the specific initiatives that are the most important to implement. Moreover, program management offices are not necessarily in a good position to sort through the initiatives to focus on those that are the most important to the job at hand. Several DOD officials expressed concerns regarding the number of initiatives without prioritization. For example, a DAU training official noted that it is impractical to expect to train the entire workforce on all of the initiatives and that the Office of the Secretary of Defense should set the priorities for implementation of initiatives. A DAU consultant echoed the view that there are too many new reform initiatives and that DOD needs to set priorities on which initiatives to address. A service acquisition reform official observed that the large numbers of best practices promoted by DOD overwhelms the services. This official told us that the three services’ acquisition reform offices meet quarterly to share progress on acquisition reform initiatives. He believes these meetings could serve as an opportunity to prioritize the practices. However, the current focus of the meetings is unclear and the offices’ charters are under review. Another acquisition reform office representative stated that the combination of a large number of reform initiatives and unclear priorities puts the office in the position of having to guess at what is the most important. This leads to emphasizing what is perceived to be popular, he added. Other officials explained that due to the frequency of new initiatives being introduced, training courses should be regularly revisited. However, training personnel have not been able to consistently keep up with the pace of change. For example, a service acquisition reform official stated that it requires at least a full year to determine the elements of a new initiative and then the challenge of how to train the initiative still exists. A 1997 DOD study team that reviewed DOD’s acquisition education and training structure and process found that curriculum development often lagged behind the rapidly changing policy requirements. The study also noted that instructors were not up to date on current acquisition policies and that Page 53 GAO/NSIAD-99-206 Best Practices Chapter 4 DOD and Commercial Training Methods Reflect Different Strategic Approaches to New Practices because they were out of touch with DOD policy makers, the instructors found it difficult to develop up-to-date training courses. DOD Proposals to Alter DOD is aware of the need to improve the means by which the acquisition workforce receives and implements new initiatives. Two proposals being Training Reflect considered by the Under Secretary of Defense for Acquisition and Conflicting Technology that are intended to improve workforce training, and a new continuing education policy has been put in place. The first proposal is a Approaches DAU transition plan for improving DAU’s structure. The plan is based on a 1997 study by a DOD team and a 1998 study by the Logistics Management Institute. Both studies confirmed strengths in the DAU approach, particularly relating to functional or skill-based training and the weaknesses in some of the same areas that surfaced in our work. The studies made recommendations for significant organizational and operational changes in DAU to better meet the educational needs of the acquisition workforce. However, a more moderate approach was adopted in preparing the transition plan for DAU. While the proposed structure offers improvements, it does not discernibly address key weaknesses in the training of best practices. The plan has yet to be approved. The second proposal, a report from the DOD Section 912(c) Commercial Business Environment Group directly advocates incorporating best practices and is much broader in recommending changes. It proposes that DOD’s acquisition structure be transformed into a team type of organization with members drawn from across the DOD procurement enterprise. A new learning organization would be created specifically to support accelerated change. DOD has also instituted a continuous learning policy, which should help people obtain current training, even if they are already fully certified. However, the required levels of continuous learning can be met in many ways and do not place any particular emphasis on training in best practices. These DOD efforts are a step in the right direction for seeing that program offices have the information they need, but none focus on the program offices’ needs. Given the pivotal role they play in weapons acquisition, a more direct approach may be required. Page 54 GAO/NSIAD-99-206 Best Practices Chapter 4 DOD and Commercial Training Methods Reflect Different Strategic Approaches to New Practices DAU Transition Plan Is The DOD transition plan for restructuring DAU was based on Largely Silent on recommendations of a 1997 process action team that DOD chartered to create a clear vision and a structure for future workforce training and Improvements to Best education. Several findings of the study related to training for new Practices Training initiatives and practices, including (1) the existing curriculum design process was functionally driven, did not address the needs of people outside the functional career fields, and did not lend itself to successful development of cross-functional curricula; (2) a mechanism was needed to validate and prioritize requirements before they are submitted; (3) faculty members did not incorporate up-to-date case studies; (4) DAU schools used procedures for measuring performance that focused on student satisfaction with a course, not whether the user community’s acquisition education requirements were being met; and (5) DAU’s use of technology- based learning was insufficient. The team recommended that DAU be replaced with a unified Defense Acquisition Institute that would have responsibility for development and delivery of acquisition training, the consortium be streamlined to reduce duplication, and lines of authority between DAU and the members be strengthened. The team envisioned the institute as, among other things, (1) fostering innovation and facilitating reform and continuous improvements, (2) having a clear message of support from DOD acquisition leadership that education is the key to meeting DOD goals, (3) identifying acquisition training needs that are cross-functional areas or that are multifunctional in nature, (4) focusing on the needs of the learner, and (5) employing a process to validate the effectiveness of the training received. However, a dissenting opinion offered by one of the team members to evolve the current DAU organization structure through a series of near-term and long-term actions was accepted by the Under Secretary of Defense. Subsequently, the Logistic Management Institute conducted a review for DOD and developed a structural reorganization for DAU that was to be used as the starting point for the transition plan. The Institute noted that DAU training was generally skill and specialty oriented, with limited overlap among “stove-piped” career fields. It recommended that the new structure be designed along the lines of a corporate university. As such, its curriculum would include cross-functional training and business executive development and act as an agent of change, being more involved with the workforce in determining how changes would be carried out. The DAU president submitted the draft transition plan for restructuring in September 1998. As of June 1999, DOD had not approved the plan. The Page 55 GAO/NSIAD-99-206 Best Practices Chapter 4 DOD and Commercial Training Methods Reflect Different Strategic Approaches to New Practices plan has several features that offer promise for more effective training overall. These include (1) a unified structure in which DAU has full responsibility for ensuring that the overall course structure meets the needs of the acquisition workforce, (2) consolidation of the consortium schools into four main campuses under the direct authority of DAU, (3) an increased emphasis on distance learning techniques, (4) steps to improve the qualifications of faculty, and (5) replacement of functional boards with functional IPTs that have representation from both functional experts and instructors and that will design curricula together. Other features of the plan raise doubts about its ability to provide a strategic approach to shape training that will help implementation of best practices at program offices. The transition team that prepared the plan concluded that DAU was unique and could not be matched with other institutions, like corporate universities. Thus, the corporate university model recommended by the Institute was not adopted. While the curriculum may benefit from the functional IPTs, the plan does not mention the need for cross-functional or best practices training or for DAU to accept the role of a change agent. It is silent on prioritizing initiatives or improving feedback. While it is possible that under the proposed plan DAU could play a more active role in the design and delivery of tailored training to the workplace, the plan does not that suggest that, other than increased distance learning, training will be offered in a manner substantially different from what has been traditionally offered or that the relationship between DAU and the acquisition community will be any closer. Commercial Business The National Defense Authorization Act for Fiscal Year 1998, section 912, Environment Plan Proposes directed the Secretary of Defense to submit to Congress an implementation plan to streamline the acquisition organizations, workforce, and Major Changes to Hasten infrastructure. As part of that mandate, the Deputy Under Secretary of Reform Defense for Acquisition and Technology formed a team to develop a plan for ensuring that new practices would be incorporated by DOD. The draft plan, dated June 1999, concludes that adopting the most effective commercial practices requires a cultural and organizational transformation within DOD. The draft proposes a very broad approach to making this change, which includes a training regime to implement key commercial business practices and a change model to accelerate the implementation of new practices. The model calls for establishing goals and a scorecard to measure whether they are met. To inform everyone of the goals, a tiered approach is envisioned, starting with senior leaders, rolling down to teams with specific reform target goals. The teams are to report on their Page 56 GAO/NSIAD-99-206 Best Practices Chapter 4 DOD and Commercial Training Methods Reflect Different Strategic Approaches to New Practices outcomes within 60 days. A mission support office will support the teams as well as capture results across DOD and make them available to others. This office is to be led by a senior DOD official and staffed by management and facilitation experts, and possibly outside contractors. The plan proposes a team approach for managing acquisitions that embraces best practices and maintains “learning organizations that seek out and adopt best practices that improve individual and organizational performance.” The plan also calls for new organizational roles in adopting of best practices. Among these is DAU. The plan recommends that DAU be broadly recast to adopt the corporate university model and become a change agent. This plan, while not specific about the help that program offices would receive, calls for a strategic approach that would make it more likely that DOD could provide its program offices tailored training— more help—in implementing best practices. Nonetheless, translating this plan into the type of training and other help program offices need to implement best practices is challenging. Key elements, such as the cascading of goals from senior levels on down and the revision of what constitutes teams, are major undertakings and challenge long established patterns of interaction. How the mission support office would reach specific program offices is unclear, as is how it would interface with DAU. Also, whether initiatives are to be prioritized and how DAU would be restructured to provide functional training and become a change agent for new practices remains to be delineated. Continuous Learning Policy The objective of DOD’s recently issued continuous learning policy is to Does Not Ensure Best augment the acquisition training standards for career field certification of acquisition professionals. The policy requires all civilian and military Practices Training members of the acquisition workforce to complete 80 hours of continued education within a 2-year period. As such, it is a good step for reaching more senior people in the acquisition workforce who would normally not receive this kind of training because they are fully certified. The policy also places increased emphasis on distance learning, which may address some of the timeliness and availability issues raised by program office officials in obtaining useful best practice information. One distance learning course that deals directly with a best practice was jointly developed last year by the National Contract Management Association and the National Association of Purchasing Management at the request of DOD. It is an internet-based course to integrate best commercial practices for managing suppliers into DOD acquisitions. It entitles participants to 24 hours of Page 57 GAO/NSIAD-99-206 Best Practices Chapter 4 DOD and Commercial Training Methods Reflect Different Strategic Approaches to New Practices continuous education credit and participants are encouraged to take the course as teams. While the policy does not preclude staff from receiving training on new practices, given the array of options for meeting requirements, it does little to ensure that best practices training will be received. An earlier version of the policy recommended 16 units of acquisition reform training that could have included best practices, but the requirement was deleted. The policy encourages acquisition reform training but allows for requirements to be met in many ways—including courses, professional activities, and on the job experience—which may or may not include best practices information. According to a training official, crediting acquisition workforce members for almost anything dilutes the policy. Officials of one program noted that no one advises them on what courses to take. They can take courses or attend conferences that they believe would benefit them, but they believed this may be inadequate for ensuring that senior officials are aware of new ideas. Another program official said that her program office could meet continuing education requirements by mandatory attendance of Acquisition Reform Week. While this exposure would increase awareness of new practices, such training was characterized as providing just a “flavor” for the initiative—not providing enough information to implement the practice. Page 58 GAO/NSIAD-99-206 Best Practices Chapter 5 Conclusions and Recommendations Chapte5 r Conclusions Leading commercial firms and weapon system program offices have successfully implemented best practices or new initiatives by taking a tailored, on-site training approach. Such an approach is a concentrated, up-front effort that creates an environment for accepting the practice within a program and provides the impetus and know-how to apply the practice at the workplace. It is warranted by the investment and importance that large programs represent. The difference is that within DOD, the approach is allowed to happen, while leading commercial firms ensure it happens. These firms ensure that initiatives are successfully implemented at the program level by customizing the initiatives to fit the programs, working with program office staff to develop training that meets the needs of the program, and providing on-site assistance to the program. This approach would not necessarily be used every time by commercial firms to train their entire workforce or to impart every skill; it is instead an intensive approach they reserve for key initiatives and individual programs. It stems from an overall training strategy that restricts the number of initiatives that are brought to the workforce, ensures that the initiatives have well-defined objectives, and makes training organizations responsible for seeing that policy is turned into practice at the program level. Little regarding implementation is left to chance. Within DOD, this approach can be taken if a program’s management has the insight to recognize the importance of an initiative, the necessary resources needed to implement the initiative at the work site, and support from top management. Even though DOD has drawn its reform initiatives from the practices of leading commercial firms, it has not adopted the accompanying training strategy to ensure a concerted effort is made to implement key initiatives at the program level. As a result, more is left to chance. In the past 4 years, over 40 initiatives have been introduced to the DOD acquisition workforce, without delineating which are the most important for weapon system programs. Several organizations are responsible for training and each service has an office that promotes acquisition reform initiatives. Although these organizations are doing valuable work and are contributing to workforce training, it is unclear how, collectively, they can drive key initiatives to the program office level. DAU’s main tool for training the acquisition workforce—functional training—is aimed at increasing the expertise of people in their career fields, but it does not have the throughput and ability to reach the program office as a work unit, nor the practical depth needed to imbed an initiative at the program office. Other tools, such as satellite broadcasts, distance Page 59 GAO/NSIAD-99-206 Best Practices Chapter 5 Conclusions and Recommendations learning, and Acquisition Reform Week, make program office staff aware of new initiatives, but they do not provide the depth or environment to implement them because DOD’s training organizations and methods have not been designed to provide this kind of support. DOD recognizes the importance of training as a tool that can help infuse best practices into weapon acquisitions and has several draft plans to reorganize how training is delivered. However, it is not evident that the planning and methods called for by the DAU transition plan will change the way such training is conceived and delivered to weapon system program offices. The Commercial Business Environment Plan does call for a strategic approach that would make it more likely that DOD could provide its program offices tailored training—more help—in implementing best practices. The fact that neither plan has been approved indicates that DOD has not decided what role it wants acquisition training to play on best practices. Whether through existing organizations or through new ones, DOD needs to increase its capability to provide tailored training of specific initiatives at the program office level. Developing this additional capability will require a strategy for implementation and collaboration with program offices on the design of training. Without a concerted approach to foster the implementation of best practices by providing for customized training at the program offices, better outcomes in weapon programs will be more difficult to achieve. The recommendations that follow address ways DOD can provide tailored training on selected practices to program offices. The recommendations are made without prejudice toward the functional and other training DOD provides to the acquisition workforce. Recommendations We recommend that the Secretary of Defense develop a strategy for delivering targeted training on selected new practices to program offices to ensure the practices will be implemented. This strategy should accomplish the following. • Identify those initiatives most worthy of a targeted training investment by screening the numerous initiatives to separate those for which a general awareness is sufficient from those that warrant a targeted approach. Those to be targeted should be relatively few in number at any given point in time. The process of setting priorities would be of a continuing nature and would benefit from the input of current program office members. Page 60 GAO/NSIAD-99-206 Best Practices Chapter 5 Conclusions and Recommendations • Decide whether DAU is to play a more traditional role, as called for by the DAU Transition Plan, or a more proactive role, as called for by the Commercial Business Environment Plan and clearly communicate to DAU and other providers of DOD training their responsibilities in supporting a targeted approach to training. • Identify the key organizations and people that are critical to the program offices’ ability to implement best practices, including those not currently defined as part of the acquisition workforce and contractors, as the potential audience of targeted training. We also recommend that the Secretary make the Under Secretary of Defense for Acquisition and Technology responsible for taking steps to institutionalize the methods for tailoring training on key initiatives that have been shown to be successful by leading program offices and commercial firms. This involves having proactive organizations and the tools to inform, prepare, and assist program offices to implement the initiatives most applicable to their programs. This approach should include the following. • Making new program managers aware of the initiatives that could significantly affect the outcomes of their programs and the role the managers play in having their staffs trained to implement those initiatives. This could be done through existing program managers’ courses. • Working with individual program managers to tailor the initiatives to their programs. • Developing an approach, in conjunction with the program managers, to create a culture—and necessary incentives—within the program office to make it receptive for adopting the initiatives. • Helping the program managers determine the best methods for making initial and sustaining resource investments in training the program office staff. This help could consist of providing advice, assisting in the identification of experts in the needed areas, and culling lessons learned from other programs that have implemented the initiative. To introduce and reinforce new practices, we recommend that the Secretary require the Under Secretary of Defense for Acquisition and Technology more effectively use existing training vehicles to • incorporate new practices more quickly, • better reflect the changing role of program managers, Page 61 GAO/NSIAD-99-206 Best Practices Chapter 5 Conclusions and Recommendations • ensure that instructors’ knowledge keeps pace with the latest practices, and • provide more case study material that gives current implementation examples for a variety of situations. Agency Comments and DOD concurred with a draft of this report and all of its recommendations (see app. I). DOD noted that it was taking steps to develop a strategy for Our Evaluation delivering targeted training on selected new practices to program offices, consistent with our report. It cited the Commercial Business Environment study team’s vision for accelerating cultural change within the acquisition community, in conjunction with a strategy of delivering team training, to implement best practices. DOD stated that it anticipates adopting and launching many of the study’s recommendations without delay. DOD also recognized that DAU should augment its capability, consistent with the best practices of the private sector corporate universities, by embracing the role of “change agent” and by designating “performance consultants” who focus on developing tailored training to meet program team needs. DOD noted that it has also begun the restructuring of DAU to provide a more centralized and integrated education and training program that will also be integrated with the initiatives identified by the Commercial Business Environment report. We agree that the DOD study frames a strategic approach that would make it more likely that DOD could provide its weapon system program offices tailored training to help them implement new practices. We are also encouraged by the speed with which DOD is implementing the study’s recommendations. We do note that the study is not specific about the help that program offices would receive and that translating the study into such specifics will be a significant challenge. As DOD proceeds with implementation, we reiterate the importance of screening the numerous initiatives to identify those most worthy of targeted investments and to involve the staff of current program offices in setting these priorities. In agreeing on the need for targeted training, DOD noted that DAU’s communications arm will work to provide for immediate delivery of tailored training on key initiatives to meet the needs of program offices. DAU’s current communications arm—the ARCC—provides training primarily by distributing videos and periodic satellite broadcasts. As such, ARCC may not be well-poised to take on the responsibilities of a proactive organization with the tools needed to inform, prepare, and assist program offices to help them implement the initiatives most applicable to their Page 62 GAO/NSIAD-99-206 Best Practices Chapter 5 Conclusions and Recommendations programs, as we have recommended. Regardless of which organization is assigned the responsibility, it is important that the organization go beyond the traditional approach of making standard training available and instead work with program offices to (1) tailor the initiatives to their programs; (2) develop an approach to help make their staffs receptive to adopting new practices; and (3) help the program managers—on site, if necessary— determine the best methods for making the investments in training the program office staff. Regarding making more effective use of existing training vehicles, DOD stated that during a review of core curriculum requirements, it will be completely reviewing the principal program management training toward implementing a fully integrated strategy. It noted that the strategic plan will address the development of tailored case study materials, practical exercises, and assessment criteria for the adoption of new practices. We believe that DOD’s proposed review has the potential to more effectively promote best practices training. In addition to these actions, as DOD conducts its review, it needs to have existing training vehicles incorporate new practices more quickly, better reflect the changing role of program managers, and ensure that instructors’ knowledge keeps pace with the latest practices. Greater involvement of weapon system program office staff in the design and content of these training vehicles could link the training more closely with the job at hand. While DOD plans to focus its continuous learning activities on key acquisition reforms, currently, members of the acquisition workforce do not necessarily have to take training on acquisition reform to meet continuous learning requirements. Page 63 GAO/NSIAD-99-206 Best Practices Appendix I Comments From the Department of Defense Appenx Idi Page 64 GAO/NSIAD-99-206 Best Practices Appendix I Comments From the Department of Defense Page 65 GAO/NSIAD-99-206 Best Practices Appendix I Comments From the Department of Defense Page 66 GAO/NSIAD-99-206 Best Practices Appendix II Description of Program Offices Visited AppenIx di Following is a description of each of the program offices we visited. Advanced Amphibious The Advanced Amphibious Assault Vehicle (AAAV) is to be used by the Marine Corps to provide high-speed transportation of troops from ships Assault Vehicle located beyond the horizon to the beaches. It will also provide armor protection, land mobility, and fire support during combat operations on the shore. It is a category I1 major acquisition program for the Marine Corps. The program office is staffed by about 75 people. Advanced The Advanced Medium-Range Air-to-Air Missile (AMRAAM) is a new generation radar-guided missile that fighter aircraft are to use against Medium-Range enemy aircraft. It can be launched beyond visual range, day or night and in Air-to-Air Missile all weather. The missile is used on the Air Force F-15 and F-16 and the Navy F-14 and F/A-18. It is a category I acquisition program for the Air Force. The AMRAAM program started over 20 years ago. About 70 people staff the program office. Battlefield Combat The Battlefield Combat Identification System (BCIS) is a millimeter wave electronic, question, and answer combat identification system capable of Identification System identifying friendly ground combat vehicles. The BCIS interrogation is triggered automatically by activation of the shooter, which sends an encrypted query message to the targeted vehicle. If the targeted vehicle is friendly and equipped with BCIS, its transponder answers with an encrypted friend message that is illuminated in the shooter’s sights. It is a category II2 acquisition managed by the Army. The program office is staffed by 16 core people and 30 additional people who are assigned from other government organizations or contractor support. 1 Category I programs are defined as major defense acquisition programs estimated to cost over $355 million for research, development, test, and evaluation, or have procurement costs of more than $2.135 billion. 2 Category II programs are defined as acquisition programs estimated to cost over $75 million for research, development, test and evaluation, or have procurement costs of more than $300 million. Page 67 GAO/NSIAD-99-206 Best Practices Appendix II Description of Program Offices Visited Joint Air-to-Surface The Joint Air-to-Surface Standoff Missile (JASSM) is an autonomous, long-range cruise missile to be capable of launch from outside area Standoff Missile defenses to hit ground targets. It is to be launched from a wide range of bomber, attack, and fighter fixed-wing aircraft. It is a joint Air Force and Navy category I major acquisition program. Joint Strike Fighter The Joint Strike Fighter (JSF) program’s objective is to develop and field an affordable, highly common family of next generation multirole strike fighter aircraft for the Navy, the Air Force, the Marine Corps, and U.S. allies. The focus of the program is affordability. It is a joint service category I major acquisition program with Navy as lead. About 100 people staff the program office. Joint Surveillance and The Joint Surveillance and Target Attack Radar System– Joint Tactical Terminal (JSTARS-JTT) is to provide warfighters with near real-time Target Attack Radar tactical intelligence and targeting information. It is a terminal that supplies System–Joint Tactical the critical data link from various intelligence sources to battle managers across all services. The terminal is integrated into other weapon systems Terminal and is to be mounted in fixed- and rotary-wing aircraft, surface ships, and fixed or mobile ground platforms and vehicles. It is managed by the Army and is a category III 3 program. About 10 people staff the program office. 3 Category III programs are defined as those acquisitions programs that do not meet the criteria for categories I or II. Page 68 GAO/NSIAD-99-206 Best Practices Appendix III GAO Contacts and Staff Acknowledgments AppeInIx di GAO Contacts Louis J. Rodrigues (202) 512-4841 Paul L. Francis (202) 512-2811 Acknowledgments In addition to those named above, Charles Cannon, Monica L. Kelly, Gordon W. Lusby, Elisabeth G. Ryan, Sally Shipman, and Yelena K. Thompson made key contributions to this report. Page 69 GAO/NSIAD-99-206 Best Practices Page 70 GAO/NSIAD-99-206 Best Practices Page 71 GAO/NSIAD-99-206 Best Practices Related GAO Products Best Practices: Commercial Quality Assurance Practices Offer Improvements for DOD (GAO/NSIAD-96-162, Aug. 26, 1996). Major Acquisitions: Significant Changes Underway in DOD’s Earned Value Management Process (GAO/NSIAD-97-108, May 5, 1997). Best Practices: Successful Application to Weapon Acquisitions Requires Changes in DOD’s Environment (GAO/NSIAD-98-56, Feb. 24, 1998). Best Practices: DOD Can Help Suppliers Contribute More to Weapon System Programs (GAO/NSIAD-98-87, Mar. 17, 1998). Defense Acquisition: Improved Program Outcomes Are Possible (GAO/T-NSIAD-98-123, Mar. 18, 1998). Defense Acquisition: Best Commercial Practices Can Improve Program Outcomes (GAO/T-NSIAD-99-116, Mar. 17, 1999). Best Practices: Better Management of Technology Development Can Improve Weapon System Outcomes (GAO/NSIAD-99-162, July 30, 1999). (707343) Leter Page 72 GAO/NSIAD-99-206 Best Practices Ordering Information The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary, VISA and MasterCard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent. Orders by mail: U.S. General Accounting Office P.O. Box 37050 Washington, DC 20013 or visit: Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000 or by using fax number (202) 512-6061, or TDD (202) 512-2537. Each day, GAO issues a list of newly available reports and testimony. To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone. A recorded menu will provide information on how to obtain these lists. For information on how to access GAO reports on the INTERNET, send an e-mail message with “info” in the body to: email@example.com or visit GAO’s World Wide Web Home Page at: http://www.gao.gov United States Bulk Rate General Accounting Office Postage & Fees Paid Washington, D.C. 20548-0001 GAO Permit No. GI00 Official Business Penalty for Private Use $300 Address Correction Requested
Best Practices: DOD Training Can Do More to Help Weapon System Programs Implement Best Practices
Published by the Government Accountability Office on 1999-08-16.
Below is a raw (and likely hideous) rendition of the original report. (PDF)