oversight

Defense Acquisition: Best Commercial Practices Can Improve Program Outcomes

Published by the Government Accountability Office on 1999-03-17.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                             United States General Accounting Office

GAO                          Testimony
                             Before the Subcommittee on Readiness and Management
                             Support, Committee on Armed Services, U.S. Senate




For Release on Delivery on
Wednesday, March 17, 1999    DEFENSE ACQUISITION

                             Best Commercial Practices
                             Can Improve Program
                             Outcomes
                             Statement for the Record by Louis J. Rodrigues, Director,
                             Defense Acquisitions Issues, National Security and
                             International Affairs Division




GAO/T-NSIAD-99-116
                   Mr. Chairman and Members of the Subcommittee:

                   Thank you for the opportunity to submit this statement for the record,
                   which discusses the best practices that can improve the way the
                   Department of Defense (DOD) buys major weapon systems. With DOD’s
                   annual research, development, and production spending for major systems
                   at about $85 billion, the Subcommittee’s oversight of acquisition policy can
                   have a major impact on the value the taxpayer gets for that expenditure.

                   For several years, the leadership in the Office of the Secretary has been
                   committed to instituting reforms to improve the outcomes of the
                   acquisition process. In particular, the Under Secretary of Defense’s
                   (Acquisition and Technology) focus on shorter cycle times is welcome.
                   Shorter cycle times make for a more agile acquisition process, which is
                   critical if DOD is to respond quickly in a national security environment of
                   unknown threats. Shorter cycle times are also important for solving
                   problems associated with readiness by ensuring that the industrial base
                   will be able to support systems once they are fielded.

                   Despite good intentions and some progress, our ongoing reviews of DOD’s
                   major system acquisitions are showing that these efforts at systemic
                   change have not yet been reflected in the management and
                   decision-making on individual programs. The flagship systems, as well as
                   many other top priorities in each of the services, continue to cost
                   significantly more, take longer, and deliver less than was promised. Our
                   work for this Subcommittee shows that lessons for major system
                   acquisitions can be learned from the best commercial practices and applied
                   in the DOD environment. Adopting these practices will require a dramatic
                   change in behavior—a change that must be supported by incentives. In this
                   context, the Office of the Secretary of Defense, the Congress, and the
                   services’ organizations for requirements, research, and acquisition, each
                   play a critical role in getting the better outcomes sought on major weapon
                   system programs.



Results in Brief   On the basis of the work we have done over the past 3 years, we believe the
                   best practices of leading commercial firms can be used to improve the
                   development of technology and weapon systems in DOD. In particular,
                   knowledge standards that are rigorously applied, coupled with the practice
                   of keeping technology development separate from product development,
                   stand out as key factors in the most successful commercial examples.
                   These practices have put managers in the best position to succeed in



                   Page 1                                        GAO/T-NSIAD-99-116 Best Practices
                         developing better products in less time and producing them within
                         estimated costs. DOD programs, with some exceptions, proceed with
                         lower levels of knowledge available about key factors of product
                         development, such as proof of design maturity and production readiness.
                         In addition, DOD allows technology development to take place during
                         product development. These practices put DOD program managers in a
                         much more difficult position to deliver better weapons more quickly and
                         within cost projections.

                         Getting better outcomes on weapon system programs will take more than
                         attempting to graft commercial best practices onto the existing acquisition
                         process. There are underlying reasons and incentives for why such
                         practices are not a natural part of how weapon systems are bought.
                         Environmental factors, such as the intense competition for funding when a
                         program is launched, encourage lower standards of knowledge and the
                         acceptance of higher, but unrecognized, risks. What we offer to help the
                         adoption of best practices is not a cookbook recipe, but a series of actions
                         aimed at fostering an environment in DOD that encourages or rewards such
                         practices. These actions will put managers of DOD programs in a better
                         position to succeed, for we believe they are as informed and capable as
                         their commercial counterparts.

                         Following the discussion of our work on best practices, we present
                         information on the status of other work we are doing that is of interest to
                         the Subcommittee. Most of this work deals with DOD initiatives related to
                         acquisitions. The work includes acquisition workforce training in best
                         practices, best practices for test and evaluation of weapon systems, pricing
                         of sole-source commercial items, spare parts price trends, the Cost
                         Accounting Standards Board, other transactions, and government-wide
                         information technology contracts.



A Best Practices Model   For an acquisition process that meets DOD’s goal of developing and
                         producing militarily superior weapons in a resource-constrained
for Acquisition          environment, we look to answer the basic question of how a capability can
                         best be provided to the customer. The characteristics of best practices, as
                         we have analyzed them, suggest a process for developing new
                         capabilities—whether they are commercial or defense products—that is
                         based on knowledge. It is a process in which technology development and
                         product development are treated differently and managed separately. The
                         process of developing technology culminates in discovery and must, by its
                         nature, allow room for unexpected results and delays. The process of



                         Page 2                                        GAO/T-NSIAD-99-116 Best Practices
developing a product culminates in delivery, and therefore, gives great
weight to design and production. Discipline is inherent because criteria
exist, tools are used, and a program does not go forward unless the strong
business case on which the program was originally justified continues to
hold true.

In the past several years, we have examined best practices used by world
class commercial firms such as Boeing, Chrysler, Hughes, Ford, and 3M,
and individual DOD acquisition programs for weapons such as the F-22, the
C-17, the Comanche, the New Attack Submarine, and the Advanced
Amphibious Assault Vehicle with the objective of finding best practices for
developing and producing major weapon systems. Our completed work
has examined best practices for quality assurance, earned value
management, supplier management, and transitioning products from
development to production. A listing of these reports is included in the
appendix. We are currently reviewing best practices for readying
technology for inclusion in product development programs, best practices
for test and evaluation of weapon systems, and how well DOD training
supports the implementation of best practices.

We have learned that a knowledge-based process is essential to getting
better cost, schedule, and performance outcomes. This means that
decisionmakers must have virtual certainty about critical facets of the
product under development when needed. Such knowledge is the inverse
of risk. The commercial and military programs we reviewed did not all
follow the same processes in their development cycles. However, at some
point, full knowledge was attained about a completed product, regardless
of what development approach was taken. This knowledge can be broken
down into three junctures that we refer to as knowledge points: when a
match is made between the customer’s requirements and the available
technology, when the product’s design is determined to be capable of
meeting performance requirements, and when the product is determined to
be producible within cost, schedule, and quality targets. In addition, we
have identified metrics that indicate the knowledge levels associated with
best practices and can thus help forecast problems as a development
program progresses. An important corollary to having a knowledge-based
process is that technology development should take place separate from an
acquisition program and its related product development process. The
knowledge points and their associated metrics are depicted in figure 1.




Page 3                                       GAO/T-NSIAD-99-116 Best Practices
Figure 1: Levels of Knowledge Attained in Best Practices for Developing Technology and Products


        Technology
                                                  Product Development                              Production
       Development




                         Knowledge                    Knowledge             Knowledge
                          Point 1                       Point 2              Point 3



                        Technology                                          Percent of key
                                                       Percent of
                         readiness                                            production
   Metrics                                             drawings
                           levels                                            processes in
                                                       complete
                                                                                control


                                         The leading commercial firms we visited gained more knowledge about a
                                         product’s technology, design, and producibility much earlier than DOD in
                                         the acquisition programs we reviewed. In fact, product development in
                                         commercial ventures is a clearly defined undertaking for which firms insist
                                         on having the technology in hand that is needed to meet customer
                                         requirements before starting. Once underway, the firms demand—and
                                         receive—specific knowledge about the design capability and producibility
                                         of a new product before production begins. The process of discovery—the
                                         accumulation of knowledge and elimination of unknowns—is completed
                                         well ahead of production. There is a synergy in this process, as the
                                         attainment of each successive knowledge point builds on the preceding
                                         one. In contrast, DOD programs are started earlier and allow technology
                                         development to continue into product development and even into
                                         production. Consequently, the programs proceed with much less
                                         knowledge available—and thus more risk—about required technologies,
                                         design capability, and producibility. Proceeding with lower levels of
                                         knowledge available explains much of the turbulence in DOD program
                                         outcomes. Metrics, such as those associated with knowledge points, show
                                         this to be a predictable consequence.




                                         Page 4                                           GAO/T-NSIAD-99-116 Best Practices
Knowledge Point 1:       Technology development has the ultimate objective of bringing a
Requirements and         technology up to the point that it can be readily integrated into a new
                         product and counted on to meet requirements. As a technology is
Technology Are Matched
                         developed, it moves from a concept to a feasible invention to a component
                         that must fit onto a product and function as expected. In between, there
                         are increasing levels of demonstration that can be measured. In our
                         ongoing review of best practices for including new technology in products,
                         we applied a scale of technology readiness levels—from one to nine—
                         pioneered by the National Aeronautics and Space Administration and
                         adapted by the Air Force Research Laboratories. Without going into the
                         details of each level, let me note that a level four equates to a laboratory
                         demonstration of a technology that is not in a usable form. Imagine, if you
                         will, an advanced radio technology that can be demonstrated with
                         components that take up a table top. A level seven is the demonstration of
                         a technology that approximates its final form and occurs in an environment
                         outside the laboratory. A level eight is a technology that has been proven to
                         work in its final form and in its intended operating conditions. The same
                         radio at this level would have been installed in the instrument panel in the
                         aircraft cockpit, integrated with other aircraft systems, and flown under all
                         expected conditions.

                         The lower the level of the technology at the time it is included in a product
                         development program, the higher the risk that it will cause problems in the
                         product development. According to the people that use the technology
                         readiness levels in DOD, level seven enables a technology to be included on
                         a product development with acceptable risk. In applying these standards
                         to leading commercial firms, we have observed that the firms do not let a
                         new technology onto a product development until it reaches level eight. On
                         weapon systems that experienced cost and schedule problems, we
                         observed that they were started with key technologies at levels three and
                         four. By the time the same programs reached a point DOD considers
                         analogous to beginning product development, key technologies were still at
                         level five or lower.

                         We also observed that three factors contribute to the successful maturation
                         of technology for inclusion on a product development. These are:
                         flexibility in resources and requirements to allow for the uncertainties of
                         technical progress; disciplined paths for technology to take toward
                         inclusion on products, with strong “gatekeepers” to decide when to allow it
                         onto a product development program; and high standards for judging the
                         maturity and readiness of technology. The commercial technologies and a
                         few of the DOD technologies we reviewed exhibited these factors and were



                         Page 5                                         GAO/T-NSIAD-99-116 Best Practices
                         successfully included in product developments. Technologies that caused
                         problems in product development did not exhibit all of these factors. In
                         essence, these technologies were still going through discovery in a
                         delivery-oriented environment. As one might imagine, the most difficult
                         situation occurs when a product development is launched with inflexible
                         requirements that can only be met with a new and immature technology.


Knowledge Point 2: The   The commercial firms we visited had achieved near certainty that their
Design Will Perform as   product designs would meet customer requirements and had gone a long
                         way toward ensuring that the products could be produced by the halfway
Required
                         point of product development. Both DOD and the commercial firms hold a
                         critical design review (CDR) to review engineering drawings, confirm the
                         design is mature, and “freeze” it to minimize changes in the future. The
                         completion of engineering drawings and their release to manufacturing
                         organizations signify that program managers are confident in their
                         knowledge that the design performs acceptably and is mature. The
                         drawings are critical to documenting this knowledge because they are not
                         only precision schematics of the entire product and all of its component
                         parts—they also reflect the results of testing and simulation, and they
                         describe the materials and manufacturing processes to be used to make
                         each component.

                         Both DOD and commercial companies consider the design to be essentially
                         complete when about 90 percent of the engineering drawings are
                         completed. Officials from commercial companies such as Boeing and
                         Hughes told us that they typically had over 90 percent of these drawings
                         available for the CDR. Two DOD programs we reviewed had less than
                         60 percent—one had less than one-third—of the drawings done at the time
                         their CDRs were held. Thus, these programs had significantly less
                         knowledge available about their designs. The programs did not get or were
                         not expected to get to the 90-percent level of completion on the drawings
                         until late in development or in production. Nonetheless, at the time of the
                         CDRs, the risks of proceeding with the rest of development on these
                         programs as planned were deemed acceptable. Both programs
                         encountered significant design problems in testing that occurred after the
                         CDR.




                         Page 6                                       GAO/T-NSIAD-99-116 Best Practices
Knowledge Point 3:            Leading commercial firms reached the point at which they knew that
Production Units Will Meet    manufacturing processes would produce a new product conforming to
                              cost, quality, and schedule targets before they began fabricating production
Cost, Quality, and Schedule
                              articles. Reaching this point meant more than knowing the product could
Objectives                    be manufactured; it meant that all key processes were under control, such
                              that the quality, volume, and cost of their output were proven acceptable.
                              Commercial firms relied on good supplier relationships, known
                              manufacturing processes, and statistical process control to achieve this
                              knowledge early and, in fact, had all their key processes under statistical
                              process control when production began. The ability to establish control for
                              key processes before production began was the culmination of all the
                              practices employed to identify and reduce risk. All of the companies we
                              visited agreed that knowledge about technology and design early in the
                              process makes the control of processes possible.

                              One weapon system program that had been in production for nearly 9 years
                              at the time of our 1998 review still had less than 13 percent of its key
                              manufacturing processes in control. Another program had 40 percent of its
                              key manufacturing processes in control 2 years before production was
                              scheduled to begin, but was not scheduled to have all key processes in
                              control until 4 years into production. Both programs experienced basic
                              producibility problems that were not discovered until late in development
                              or early in production. These risks went unrecognized even though the
                              DOD had established criteria for determining whether risks were
                              acceptable and whether enough knowledge had been gained to enter the
                              next development phase.



Impediments to                The most important factors in the adoption of best practices are the
                              incentives the development or acquisition process offer to managers of
Adopting Best                 technology and product developments. The differences in the practices
Practices                     employed by the leading commercial firms and DOD reflect the different
                              demands imposed on programs by the environment in which they are
                              managed. The way success and failure are defined for commercial and
                              defense product developments differs considerably, which creates a
                              different set of incentives and evokes different behaviors from the people
                              managing the programs. Specific practices take root and are sustained
                              because they help a program succeed in its environment.

                              Leading commercial firms launch a product development only when a solid
                              business case can be made. The business case basically revolves around



                              Page 7                                        GAO/T-NSIAD-99-116 Best Practices
the ability to produce a product that the customer will buy and that will
provide an acceptable return on investment. The point of sale occurs in
production after product development is complete; program success is
determined when the customer buys the finished product. If the firm has
not made a sound business case, or has been unable to deliver on one or
more of the business case factors, it faces a very real prospect of failure—
the customer may walk away. Also, when each program is delivered as
promised, the company does not put at risk resources invested in other
products. Because the match between technologies and product
requirements is made before the product development is launched, the cost
and schedule consequences associated with discovery are minimized.

Production is a dominant concern throughout the commercial product
development process and forces discipline and trade-offs in the design
process. This environment encourages realistic assessments of risks and
costs; doing otherwise would threaten the business case and invite failure.
For the same reasons, the environment places a high value on knowledge
for making decisions. Program managers have good reasons to want risks
identified early, be intolerant of unknowns, and not rely on testing as the
main vehicle for discovering the performance characteristics of the
product. By protecting the business case as the key to success, program
managers in leading commercial firms are conservative in their estimates
and aggressive in risk reduction. Ultimately, preserving the business case
strengthens the ability of managers to say “no” to pressures to accept high
risks or unknowns. Practices, such as maturing technologies to high
readiness levels before inclusion in a program, having 90 percent of
engineering drawings done by the CDR, and achieving statistical process
control before production, are adopted because they help ensure success.

The basic management goal for a weapon system program in DOD is
similar: to develop and deliver a product that meets the customer’s needs.
However, the pressures of successfully competing for the funds to start and
sustain a DOD acquisition program make for a much different business
case. Compared with commercial programs, the DOD environment
encourages launching product developments that embody more technical
unknowns and less knowledge about the performance and production risks
they entail. Although DOD is attempting to create more flexibility in how
technical requirements are set for programs, a new product development is
encouraged to possess performance features that significantly distinguish
it from other systems. Consequently, aspiring DOD programs have
incentives to include performance features and design characteristics that
rely on immature technologies. These unknowns place a much greater



Page 8                                        GAO/T-NSIAD-99-116 Best Practices
reliance on maturing technology during product development than we
found on commercial programs.

Even though less information about a new product development is
available at the time of launch, the competition for funding requires
detailed projections to be made from what information does exist. A
product development deemed worthy cannot be launched unless the
program’s development and production cost, as well as timing, fall within
available funding. Because DOD relies largely on forecasts of cost,
schedule, and performance that are comparatively soft at the time, success
in funding competition encourages the cost and schedule estimates to be
squeezed into profiles of available funding. Additional requirements, such
as high reliability and maintainability, serve to make the fit even tighter. As
competition for funding will continue throughout the program’s
development, success is measured in terms of ability to secure the next
installment.

Untempered by knowledge to the contrary, the risks associated with
developing new technologies together with the product within tight
estimates are deemed acceptable. Production realities, critical to matching
technological capabilities with customer requirements on commercial
programs, are too far away from the DOD launch decision to have the same
curbing effect on technology decisions. Thus, the environment for
managing weapon system programs is a particularly difficult environment
for managing technology development. The ups and downs and resource
changes associated with the technology discovery process do not mesh
well with a program’s need to meet cost, schedule, and performance goals.
Problems with developing technologies, which are to be expected, can
actually threaten the support for a program if they become known.

These pressures and incentives explain why the behavior of weapon
system managers differs from that of managers of commercial product
developments. Problems or indications that the estimates are decaying do
not help sustain funding support for the program in subsequent years, and
thus, their admission is implicitly discouraged. An optimistic cost estimate
makes it easier to launch a product development and sustain annual
approval; admission that costs are likely to be higher could invite failure.
Rewards for discovering and recognizing potential problems early in a DOD
product development are few. Less available knowledge makes it harder
for program managers to say “no.” In contrast with leading commercial
firms, not having attained knowledge—such as on the full performance of a
key technology or the true risks facing manufacture—can be perceived as



Page 9                                          GAO/T-NSIAD-99-116 Best Practices
better than knowing that problems exist. For these reasons, the practices
associated with managing to knowledge points—such as applying
knowledge standards for technology, design, and production maturity—are
not readily adopted in DOD.

Our ongoing work on technology inclusion shows that in addition to the
incentives provided by the acquisition environment, some structural and
budgetary impediments exist that make it difficult for technology to be
matured to a high level of readiness before being included in a DOD
weapon system development. First, it is not clear that organizations
exist—other than program offices—that have the role to take technology
from a readiness level of three or four, where new technologies are often
put into a program, to the level seven needed to be readily included in a
product development. We found that for the DOD cases in which high
technology readiness levels were attained before program launch,
organizations stepped in and played atypical roles in bridging the gap from
science and technology to product development. Second, budget
realities—the fact that programs attract much higher levels of funding than
science and technology projects—make programs a favored venue for
funding technology development through the higher and more expensive
readiness levels. Third, if science and technology organizations became
responsible for managing technology development to the higher levels,
they most likely would need additional funding.

These observations about the differences between the commercial and
DOD environments should not be interpreted to mean that commercial
managers are somehow more skilled or knowledgeable than their DOD
counterparts. Nor do DOD program managers act irrationally. They see
the acquisition of the weapons under their purview as aligned with national
interests, and they do what they believe is right, given the pressures they
face. All of the numerous participants in the acquisition process play a part
in creating these pressures. In fact, the weapon systems acquisition
process asks much more of DOD program managers than commercial firms
do. Perhaps they are asked to do too much: develop advanced technology,
manage product design and production, and champion the program’s cause
through budget and other decisions over many years. In fact, one
commercial executive observed that it is unreasonable to expect people to
focus on a goal (such as production start-up) that is 4 or more years away.
Commercial program managers are put in a better position to succeed; they
have to worry only about product design and production within the cost,
schedule, and performance demands of the business case.




Page 10                                        GAO/T-NSIAD-99-116 Best Practices
Charting a Course for     Commercial practices for gaining knowledge and assessing risks can help
                          produce better outcomes on weapon systems. Collectively, such individual
Adopting Best             outcomes will help attain DOD’s modernization goals and improve funding
Practices to Get Better   stability for programs. For such practices to work, however, the
                          knowledge they produce must help a DOD program succeed in its
Outcomes                  environment. Thus, the DOD environment must become conducive to such
                          practices. At least two factors are critical to fostering such an
                          environment. First, program launch decisions must be relieved of the need
                          to overpromise on performance and resource estimates. The pressure to
                          amass broad support at launch creates incentives for new programs to
                          embrace far more technology development than commercial programs do.
                          Separating technology development so that it does not have to be managed
                          within the confines of a weapon systems program would go a long way to
                          relieving this pressure. The objectives of technology development, as well
                          as what is demanded of knowledge and estimates, differ from those of
                          product development. Clearly, DOD has to develop technology, particularly
                          the technology that is unique to military applications. However, by
                          separating technology development from weapon programs, DOD could
                          insist on higher standards for knowledge on its programs and get better
                          results when those programs transition to production.

                          Second, once a program is underway, the participants in the acquisition
                          process must make it acceptable for managers to identify unknowns as
                          high risks so that they can be aggressively worked on earlier in
                          development. Commercial firms insist on knowledge measured against a
                          criterion for assessing risk. Firms then make decisions to preserve the
                          business case by eliminating risks. The result is discipline provided from
                          within. We believe that if the Congress and DOD weighed program launch
                          decisions and subsequent progress on weapon systems by applying a
                          common set of knowledge standards, like those gleaned from leading
                          commercial firms, they could create a better “business” case for weapon
                          systems. By developing technology separately to high readiness levels
                          before including it in a program and by adhering to knowledge standards in
                          product development, DOD program managers can be put in a better
                          position to succeed in the timely design and production of weapon
                          systems.

                          The real test of the participants’ resolve to get better outcomes by applying
                          best practices will be the decisions made on individual weapon systems,
                          such as for launch and funding. These decisions define what success
                          means in DOD and what practices contribute to success. Decisions made



                          Page 11                                        GAO/T-NSIAD-99-116 Best Practices
by DOD or the Congress to advance or fund programs that do not have
enough knowledge to meet agreed-upon standards send signals to
managers that not having the necessary level of knowledge is acceptable.
On the other hand, decisions to not start new programs that need
technology advances to meet unforgiving requirements or to recognize
early that a change in a program is necessary to attain desired knowledge
levels merit support.

Managing technology development differently and applying knowledge
standards to both technology development and product development will
have implications for organizational roles and budgeting. For example, if
DOD were to attempt to develop technologies to higher readiness levels
before including them in a product development, then organizations other
than weapon system program offices will have to be made responsible for
bridging the gap from the traditional science and technology role to the
redefined program manager’s role. Likewise, the research and
development funding attendant to those bridging responsibilities may have
to be budgeted and accounted for differently. This does not necessarily
mean more research and development money is needed in the aggregate.
Rather, taking the foregoing actions could actually lower costs in the long
run, thereby freeing funding for other needs.

The best practices we have described and the changes needed to adopt
them are not concepts that are foreign to DOD. One of the
recommendations from a 1996 Defense Science Board study called for
DOD to aggressively pursue high-risk technology before inclusion in a
weapon research and development program.1 We found several
instances—such as the photonics mast for the Navy’s New Attack
Submarine, propulsion and related technologies for the Marine Corps
Advanced Amphibious Assault Vehicle, and the Air Force’s Integrated High
Performance Turbine Engine Technology Program—where DOD has put
the organizations and funding in place to bring technologies to a high level
of readiness before they were included in programs. Another program of
interest is the Army’s Future Scout and Cavalry System, which is being
managed as an Advanced Technology Demonstration, a DOD initiative to
demonstrate immature technologies so they can be more easily
incorporated into a product development. Several other DOD initiatives,



1
  A Streamlined Approach to Weapon Systems Research, Development, and Acquisition: The
Application of Commercial Practices, a Report from the Defense Science Board Task Force on Defense
Acquisition Reform, May 1996.




Page 12                                                    GAO/T-NSIAD-99-116 Best Practices
                              like cost as an independent variable and integrated product teams, are
                              attempting to draw lessons from commercial practices.



Ongoing Work of               The Subcommittee also asked for information on several assignments we
                              have underway. Most of this work deals with DOD initiatives related to
Interest to the               acquisitions. Summaries follow.
Subcommittee

Acquisition Workforce         DOD has a number of reform initiatives that are related to the best
Training in Best Practices    practices that we have reported are possible to adopt in the DOD
                              environment. At the request of this Subcommittee, we are evaluating the
                              role of DOD’s training in getting best acquisition practices applied to
                              weapon system programs and whether training could be improved.

                              We have focused our work on weapon system program offices because
                              they are where the practices are applied. We selected six program offices
                              cited as having been successful in applying one or more of the following
                              practices: cost as an independent variable, past performance, performance
                              specifications, integrated product teams, and supplier relations. We are
                              assessing how DOD’s training supported their use of the practices. For
                              comparative purposes, we are meeting with four leading commercial firms
                              noted for their excellence in workforce training. We expect to issue a final
                              report in August 1999.


Best Practices for Test and   At the request of this Subcommittee, we have recently begun reviewing
Evaluation of Weapon          best practices as they can be applied to the test and evaluation of weapon
                              systems. The objectives of this work are to determine whether (1) the
Systems
                              testing practices of leading commercial firms offer improvements to DOD’s
                              testing practices on weapon acquisition programs, (2) a particular area of
                              best testing practices stands out as a leverage point that could offer a
                              significant improvement for weapon acquisitions, (3) the role or purpose of
                              testing in best commercial product developments shape actual testing
                              practices and to what extent testing plays a different role in major weapon
                              acquisitions, and (4) obstacles hinder the implementation of best practices
                              on DOD weapon acquisition programs. We plan to complete the design
                              phase of this assignment in June 1999.




                              Page 13                                       GAO/T-NSIAD-99-116 Best Practices
Pricing of Sole-Source   DOD, with the support of the Congress, is increasing its use of
Commercial Items         commercially available products and services. While the current level of
                         commercial purchasing is relatively small and sole-source purchases even
                         smaller, we expect commercial purchases to increase in the future. We are
                         currently reviewing the price analysis tools DOD contracting personnel use
                         to arrive at fair and reasonable prices for commercial sole-source items and
                         the guidance and training available to assist them in determining price
                         reasonableness. In March 1998, we reported our preliminary observations
                         on the pricing of commercial sole-source spare parts during testimony
                         before the former Subcommittee on Acquisition and Technology.2
                         Currently, we are nearing completion of that work and plan to issue our
                         report to you in June. Our tentative findings are summarized below.

                         The current contracting environment for sole-source commercial items
                         presents negotiating challenges for DOD contracting personnel. While the
                         Federal Acquisition Regulation grants DOD contracting officers wide
                         latitude on the type and extent of price analysis techniques they can use,
                         they are, nevertheless, required to perform sufficient price analyses to
                         determine whether offered prices are fair and reasonable. Our review of
                         the price analyses they perform found a number of weaknesses. For
                         example, some contracting personnel accepted initially offered prices
                         because they had a misperception that if the offered prices were the same
                         as the catalog or list price, the offered prices were fair and reasonable.
                         Some contracting personnel did not seem to use pertinent contract file
                         information on historical pricing in their price analysis. Others did not
                         appear to understand the makeup of catalog prices and paid prices that
                         were based on rapid delivery service when it was not needed because the
                         purchases were for restocking inventories.

                         DOD continues to provide training and guidance to assist contracting
                         personnel in understanding the requirements of a sound price analysis in a
                         commercial contracting environment. In addition, recent legislation
                         requires increased guidance for contracting personnel on price analysis
                         tools, the appropriate use of information other than certified cost or pricing
                         data, and the role of support agencies. In time, as more contracting
                         personnel are trained and additional guidance is provided, we expect to see
                         improvement in the quality of price analyses for sole-source commercial
                         items.


                         2
                             Defense Acquisition: Improved Program Outcomes Are Possible (GAO/T-NSIAD-98-123, Mar. 18, 1998).




                         Page 14                                                       GAO/T-NSIAD-99-116 Best Practices
Spare Parts Price Trends    Recently, the services have been concerned that the prices they are paying
                            for critical spare parts have been increasing above the rate of inflation. As
                            a result, we have begun a review of the prices DOD end-users are paying for
                            consumable spare parts and repairable items and the change in these prices
                            over time relative to inflation. Where there are significant increases over
                            and above the rate of inflation, we will attempt to ascertain the factors that
                            most contribute to those increases.


Cost Accounting Standards   The Congress asked us to establish a panel of experts to review and make
Board                       recommendations regarding the Cost Accounting Standards (CAS) Board
                            and the CAS system in light of recent procurement reforms. More than
                            25 years ago, the Congress established the CAS Board to protect the
                            government from certain risks inherent in cost-based contracts and to
                            improve communications between the government and contractors with
                            regard to those contracts. Cost-based contracts continue to represent the
                            majority of all federal contracting dollars. The CAS Board Review Panel is
                            expected to recommend that there is a continuing need for the CAS Board.

                            The Review Panel is also expected to recommend reforms to encourage the
                            participation of new commercial companies in government procurement
                            and to reduce the burden of government unique accounting system
                            requirements on smaller companies. The Review Panel’s report is expected
                            to be released within the next few weeks with its recommendations aimed
                            at significantly reducing the burdens and costs of the CAS system without
                            diminishing its benefits.


Other Transactions          In 1993, the Congress passed section 845 of the National Defense
                            Authorization Act for fiscal year 1994, which authorized the Defense
                            Advanced Research Projects Agency to conduct prototype projects of
                            weapons or weapon systems under the authority of 10 U.S.C. 2371. This is
                            commonly referred to as “other transaction” authority. An other
                            transaction is distinct from other instruments, such as contracts, grants,
                            and cooperative agreements, and is generally not subject to the laws and
                            regulations that apply to these instruments. Subsequently, section 804 of
                            the fiscal year 1997 National Defense Authorization Act extended the
                            authority to enter into other transaction agreements to the military
                            departments and other DOD components. Other transactions can be used
                            even if a contract would be feasible or appropriate. Further, contractors




                            Page 15                                        GAO/T-NSIAD-99-116 Best Practices
                         are not required to share in the costs of the agreements. DOD’s authority
                         under section 845 is to expire on September 30, 2001.

                         In response to a Subcommittee request, we are determining (1) the extent
                         to which DOD has used its section 845 authority, (2) why other transactions
                         were selected as the procurement instrument, and (3) the extent to which
                         other transactions address key areas of risk normally covered by Federal
                         Acquisition Regulations. The scope of our review includes all other
                         transactions that DOD entered into under the authority of section 845, as
                         amended, between fiscal years 1994 and 1998 and two other transactions
                         that were awarded in October 1998 for the Evolved Expendable Launch
                         Vehicle. Our concerns on the potential risks of using other transactions for
                         that program were discussed in a prior report.3 We will issue our report
                         this summer.


Governmentwide           We were asked to examine selected governmentwide agency contracts,
Information Technology   typically for various information technology resources, to determine
                         whether competition requirements were being met. In 1994, the Congress
Contracts
                         directed that agencies consider awarding these task and delivery order
                         contracts to multiple firms—rather than a single firm—to provide for
                         competition in ordering.4 Federal agencies are to provide each of the
                         multiple contractors a fair opportunity to be considered for orders placed
                         under the contracts.

                         In September 1998, we reported that efforts to provide a fair opportunity—
                         and thereby promote competition—varied among the six organizations
                         reviewed.5 Four agencies had experienced difficulty obtaining
                         competition, while two had achieved consistent competition. Because
                         multiple award task and delivery order contracts are a relatively new
                         contracting mechanism, and because of the large size of some orders that
                         had been awarded pursuant to the new authority, we were requested to
                         continue a review of the implementation of these provisions. We have


                         3
                          Evolved Expendable Launch Vehicle: DOD Guidance Needed to Protect Government’s Interest
                         (GAO/NSIAD-98-151, June 11, 1998).

                         4A task- or delivery-order contract provides for an indefinite quantity, within stated limits, of supplies or
                         services to be furnished during a fixed period, with deliveries or performance to be scheduled by
                         placing orders with the contractor.

                         5
                           Acquisition Reform: Multiple-Award Contracting at Six Federal Organizations (GAO/NSIAD-98-215,
                         Sept. 30, 1998)




                         Page 16                                                           GAO/T-NSIAD-99-116 Best Practices
selected four organizations that have awarded multiple award contracts for
information technology products and services that are available for
governmentwide use. Our work will provide a perspective on how agencies
ensure contractors a fair opportunity to be considered for interagency
orders.

Three of the four organizations were included in our prior work and have
attempted to strengthen their processes for providing contractors a fair
opportunity. The Department of Transportation—which sometimes
accepted unconvincing rationales for placing sole-source orders—now
provides customers placing interagency orders specific guidance about
acceptable rationales to justify sole-source orders. The National Institutes
of Health—which had identified preferred contractors when announcing
plans to place orders—has eliminated any reference to a preferred
contractor from such announcements. In addition, a proposed change to
the governmentwide procurement regulations would prohibit designating
preferred contractors. Finally, the Defense Information Systems Agency—
which had not required that all multiple award contractors be notified of
planned competitive orders—modified its procedures to require that all
contractors be notified of orders.

Although our current review is now in its initial phases, our preliminary
work shows that competition for the largest orders under multiple award
contracts has not been routinely achieved. During our work, we will
identify factors that might have deterred contractors from competing for
these orders and assess how changing agencies’ fair opportunity processes
might broaden competition for orders.

This concludes our statement. We appreciate the opportunity to have it
placed in the record.




Page 17                                       GAO/T-NSIAD-99-116 Best Practices
Appendix I

GAO Products on Best Practices Applicable to
Weapon Acquisitions                                                                              AppenIx
                                                                                                       di




                     Best Practices: Commercial Quality Assurance Practices Offer
                     Improvements for DOD (GAO/NSIAD-96-162, Aug. 26, 1996).

                     Major Acquisitions: Significant Changes Underway in DOD’s Earned Value
                     Management Process (GAO/NSIAD-97-108, May 5, 1997).

                     Best Practices: Successful Application to Weapon Acquisitions Requires
                     Changes in DOD’s Environment (GAO/NSIAD-98-56, Feb. 24, 1998).

                     Best Practices: DOD Can Help Suppliers Contribute More to Weapon
                     System Programs (GAO/NSIAD-98-87, Mar. 17, 1998).

                     Defense Acquisition: Improved Program Outcomes Are Possible
                     (GAO/T-NSIAD-98-123, Mar. 18, 1998).




(707408)     Letrt   Page 18                                     GAO/T-NSIAD-99-116 Best Practices
Ordering Information

The first copy of each GAO report and testimony is free.
Additional copies are $2 each. Orders should be sent to the
following address, accompanied by a check or money order made
out to the Superintendent of Documents, when necessary, VISA and
MasterCard credit cards are accepted, also.

Orders for 100 or more copies to be mailed to a single address are
discounted 25 percent.

Orders by mail:

U.S. General Accounting Office
P.O. Box 37050
Washington, DC 20013

or visit:

Room 1100
700 4th St. NW (corner of 4th and G Sts. NW)
U.S. General Accounting Office
Washington, DC

Orders may also be placed by calling (202) 512-6000
or by using fax number (202) 512-6061, or TDD (202) 512-2537.

Each day, GAO issues a list of newly available reports and
testimony. To receive facsimile copies of the daily list or any list
from the past 30 days, please call (202) 512-6000 using a touchtone
phone. A recorded menu will provide information on how to obtain
these lists.

For information on how to access GAO reports on the INTERNET,
send an e-mail message with “info” in the body to:

info@www.gao.gov

or visit GAO’s World Wide Web Home Page at:

http://www.gao.gov
United States                       Bulk Rate
General Accounting Office      Postage & Fees Paid
Washington, D.C. 20548-0001           GAO
                                 Permit No. GI00
Official Business
Penalty for Private Use $300

Address Correction Requested