oversight

Information Technology: DOD Needs to Leverage Lessons Learned from Its Outsourcing Projects

Published by the Government Accountability Office on 2003-04-25.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

             United States General Accounting Office

GAO          Report to the Subcommittee on
             Readiness and Management Support,
             Committee on Armed Services,
             U.S. Senate

April 2003
             INFORMATION
             TECHNOLOGY
             DOD Needs to
             Leverage Lessons
             Learned from Its
             Outsourcing Projects




GAO-03-371
             a
                                                 April 2003


                                                 INFORMATION TECHNOLOGY

                                                 DOD Needs to Leverage Lessons Learned
Highlights of GAO-03-371, a report to the        from Its Outsourcing Projects
Subcommittee on Readiness and
Management Support, Committee on
Armed Services, U.S. Senate




 Given the magnitude of its reported             The projects in GAO’s review substantially used leading commercial
 spending on information                         practices as specified in GAO’s framework for outsourcing IT services.
 technology (IT) services—more                   Specifically, the agencies fully implemented 88 percent of the practices (not
 than $6.2 billion in fiscal year                including practices not applicable to a particular project). This framework
 2001—it is critical that the                    consists of practices organized into seven phases: (I) determine sourcing
 Department of Defense (DOD)                     strategy, (II) define operational model, (III) develop the contract, (IV) select
 adopt effective practices for                   the provider(s), (V) transition to provider(s), (VI) manage the performance
 acquiring IT services.                          of the provider(s), and (VII) ensure services are provided. The figure below
 GAO researched leading                          shows the percentage of practices that were implemented in each phase.
 commercial practices for the
 outsourcing of IT services, and, in             Although DOD has acted on gathering and disseminating lessons learned and
 November 2001, published a                      commercial leading practices related to general acquisition issues, its
 framework consisting of seven                   actions have generally not been focused on outsourcing or on sharing the
 phases that span the full range of              lessons learned from IT services outsourcing across the department. By not
 activities that are performed during            systematically capturing and disseminating such information across the
 the outsourcing of those services               department, DOD is losing the opportunity to leverage the knowledge gained
 (this is an acquisition in which a              on IT services projects like those in GAO’s review. Lessons learned that are
 client organization transfers
                                                 pragmatic and easily accessible would give DOD managers a more informed
 responsibility for performing
                                                 understanding of important issues to be addressed when making outsourcing
 services to an external provider).
                                                 decisions, as well as the factors to be considered to help ensure the success
 GAO was asked to determine                      of these endeavors.
 (1) the extent to which selected
 DOD projects for outsourcing IT                 Percentage of Practices Implemented, by Phase
 services use leading commercial
 practices as specified in GAO’s
 framework and (2) whether DOD is
 sharing lessons learned from its IT
 outsourcing projects across the
 department.



GAO is making recommendations to
the Secretary of Defense aimed at
leveraging lessons learned across the
department from its components’ IT
outsourcing experiences.

DOD agreed that capturing lessons
learned related to IT outsourcing
                                                 Note: Not applicable—The practice was not relevant to the project’s particular circumstances.
initiatives is important and stated that         No—The agency did not implement the practice. Limited—The agency fully implemented some but
it intends to explore a variety of               not all aspects of the practice and did not take alternative actions that fully satisfied the practice.
mechanisms to do so. DOD’s plans are             Yes—The agency fully implemented the practice or took an alternative action that fully satisfied its
consistent with our recommendations.             intent.
www.gao.gov/cgi-bin/getrpt?GAO-03-371.
To view the full report, including the scope
and methodology, click on the link above.
For more information, contact Randolph C. Hite
at (202) 512-3439 or hiter@gao.gov.
Contents



Letter                                                                                                     1
                              Results in Brief                                                             2
                              Background                                                                   5
                              Projects Substantially Used Leading Commercial Practices                    14
                              Leveraging Lessons Learned DOD-wide
                                Could Assist Other DOD Projects                                           23
                              Conclusions                                                                 27
                              Recommendations                                                             27
                              Agency Comments and Our Evaluation                                          28


Appendixes
               Appendix I:    Objectives, Scope, and Methodology                                          30
               Appendix II:   Projects’ Implementation of Phase I: Determine Sourcing
                              Strategy                                                                    32
              Appendix III:   Project’s Implementation of Phase II: Define Operational
                              Model                                                                       38
              Appendix IV:    Project’s Implementation of Phase III: Develop the
                              Contract                                                                    48
               Appendix V:    Projects’ Implementation of Phase IV: Select the
                              Provider(s)                                                                 59
              Appendix VI:    Projects’ Implementation of Phase V: Transition to
                              Provider(s)                                                                 66
             Appendix VII:    Projects’ Implementation of Phase VI: Manage Provider(s)
                              Performance                                                                 77
             Appendix VIII:   Projects’ Implementation of Phase VII: Ensure Services are
                              Provided                                                                    86
              Appendix IX:    Comments from the Department of Defense                                     90
                              GAO Comments                                                                93
               Appendix X:    GAO Contact and Staff Acknowledgments                                       94
                              GAO Contact                                                                 94
                              Staff Acknowledgments                                                       94


Tables                        Table 1:   Definition of Phases for IT Outsourcing                           8
                              Table 2:   Profile of Outsourcing Projects Reviewed                         13
                              Table 3:   Percentage of Practices Implemented, by Project                  17
                              Table 4:   Summary of Projects’ Use of Phase I Practices                    34



                              Page i                                   GAO-03-371 DOD IT Services Outsourcing
          Contents




          Table 5: Summary of Projects’ Use of Phase II Practices                    41
          Table 6: Summary of Projects’ Use of Phase III Practices                   51
          Table 7: Summary of Projects’ Use of Phase IV Practices                    61
          Table 8: Summary of Projects’ Use of Phase V Practices                     69
          Table 9: Summary of Projects’ Use of Phase VI Practices                    80
          Table 10: Summary of Projects’ Use of Phase VII Practices                  88


Figures   Figure 1: Percentage of Practices Implemented in Each Phase                 3
          Figure 2: Roles of the Client and Provider in an Outsourcing
                     Relationship                                                     6
          Figure 3: GAO’s Framework for Outsourcing IT Services                       7
          Figure 4: Use of Leading Commercial Practices for Outsourcing of
                     IT Services, by Project                                         15
          Figure 5: Percentage of Practices Implemented in Each Phase                16
          Figure 6: Project Implementation of Phase I: Determine Sourcing
                     Strategy                                                        33
          Figure 7: Project Implementation of Phase II: Define Operational
                     Model                                                           40
          Figure 8: Project Implementation of Phase III: Develop the
                     Contract                                                        50
          Figure 9: Project Implementation of Phase IV: Select the
                     Provider(s)                                                     60
          Figure 10: Project Implementation of Phase V: Transition to
                     Provider(s)                                                     68
          Figure 11: Project Implementation of Phase VI: Managing Provider
                     Performance                                                     79
          Figure 12: Project Implementation of Phase VII: Ensure Services Are
                     Provided                                                        87




          Page ii                                 GAO-03-371 DOD IT Services Outsourcing
Contents




Abbreviations

C3I          command, control, communications, and intelligence
C4           command, control, communications, and computer
CIO          chief information officer
DOD          Department of Defense
IT           information technology
IT/IS        Information Technology/Information Services
MHS/ITO      Military Health System/Information Technology Organization
NETCOM       Network Enterprise Technology Command
NIMA         National Imagery and Mapping Agency
NMCI         Navy and Marine Corps Intranet
OMB          Office of Management and Budget
RFP          request for proposals
SLA          service-level agreement
TAC-SWA      Total Army Communications—Southwest Asia




 This is a work of the U.S. Government and is not subject to copyright protection in the
 United States. It may be reproduced and distributed in its entirety without further
 permission from GAO. It may contain copyrighted graphics, images or other materials.
 Permission from the copyright holder may be necessary should you wish to reproduce
 copyrighted materials separately from GAO’s product.




Page iii                                         GAO-03-371 DOD IT Services Outsourcing
A
United States General Accounting Office
Washington, D.C. 20548



                                    April 25, 2003                                                                             Leter




                                    The Honorable John Ensign
                                    Chairman
                                    The Honorable Daniel K. Akaka
                                    Ranking Minority Member
                                    Subcommittee on Readiness and Management Support
                                    Committee on Armed Services
                                    United States Senate

                                    The Department of Defense (DOD) is the government’s largest purchaser of
                                    information technology (IT) services, such as desktop support, network
                                    operations, and software development services. In fiscal year 2001, DOD
                                    reportedly obligated more than $6.2 billion on IT services,1 and this amount
                                    is expected to grow substantially. Given the magnitude of DOD’s spending
                                    on such services, it is critical that the department adopt effective practices
                                    for acquiring IT services.

                                    Since 1996, we have conducted a series of studies for the Senate
                                    Committee on Armed Services concerning how DOD can improve its
                                    acquisition processes by adopting proven practices of leading commercial
                                    organizations. In this vein, in November 2001, we issued a guide that
                                    organized leading commercial practices for the outsourcing2 of IT services
                                    into a framework of seven phases that span the full range of activities that
                                    are performed during IT services outsourcing.3

                                    This report responds to your request that we determine (1) the extent to
                                    which selected DOD IT services outsourcing projects use leading
                                    commercial practices as specified in our framework and
                                    (2) whether DOD is sharing lessons learned from its IT outsourcing
                                    projects across the department. To address the first objective, we selected
                                    five projects from a group of projects to outsource IT services that were



                                    1
                                     This figure is from the Federal Procurement Data System, which contains detailed
                                    information on contract actions over $25,000.
                                    2
                                     IT services outsourcing is a type of acquisition in which a client organization transfers
                                    responsibility for performance of one or more IT services to one or more external providers.
                                    3
                                     U.S. General Accounting Office, Information Technology: Leading Commercial Practices
                                    for Outsourcing of Services, GAO-02-214 (Washington, D.C.: Nov. 30, 2001).




                                    Page 1                                            GAO-03-371 DOD IT Services Outsourcing
                   identified by the military services and other DOD components.4 We then
                   asked the component responsible for each project to perform a self-
                   assessment against selected practices in our framework for outsourcing IT
                   services.5 Next, we obtained and reviewed agencies’ supporting
                   documentation and interviewed the appropriate agency and provider
                   project officials to independently determine whether a practice was met.
                   To address the second objective, we reviewed applicable DOD approaches
                   for capturing and disseminating lessons learned from IT services
                   outsourcing projects and interviewed the applicable acquisition and IT
                   officials. Details of our objectives, scope, and methodology are discussed
                   in appendix I.



Results in Brief   The projects in our review substantially used leading commercial practices
                   as specified in our framework on outsourcing IT services. Specifically, the
                   agencies fully implemented 88 percent of the practices.6 This framework
                   consists of practices organized into seven phases that span the full range of
                   activities that are performed during IT outsourcing: (I) determine sourcing
                   strategy, (II) define operational model, (III) develop the contract, (IV)
                   select the provider(s), (V) transition to provider(s), (VI) manage
                   provider(s) performance, and (VII) ensure services are provided. Figure 1
                   illustrates the percentage of practices that were followed in each phase.
                   Collectively, the projects fully implemented from 70 to 97 percent of the
                   practices in each phase.




                   4
                    We asked the military services and other DOD components to identify candidate projects
                   because DOD does not maintain a central list of IT services outsourcing projects. We chose
                   each project on the basis of the following criteria: (1) no more than one project from each
                   military service and two agencies, (2) illustrative example of DOD IT outsourcing, (3) dollar
                   value greater than $10 million, and (4) enough time elapsed for services to have been
                   delivered and performance measured.
                   5
                    We identified 70 practices in our November 2001 report on leading commercial practices
                   that (1) are typically applied at the project level and (2) were verifiable through
                   documentation and interviews.
                   6
                   This calculation does not include practices that were not applicable to a particular project.




                   Page 2                                             GAO-03-371 DOD IT Services Outsourcing
Figure 1: Percentage of Practices Implemented in Each Phase




Note: Not applicable—The practice was not relevant to the project’s particular circumstances.
No—The agency did not implement the practice. Limited—The agency fully implemented some but not
all aspects of the practice and did not take alternative actions that fully satisfied the practice. Yes—The
agency fully implemented the practice or took an alternative action that fully satisfied its intent.




Page 3                                                    GAO-03-371 DOD IT Services Outsourcing
Although implementing the leading commercial practices in our framework
does not guarantee the success of an outsourcing project, the consensus
view of the leading commercial activities that we studied is that these
practices are the most critical to success when acquiring IT services.7 In
addition, not implementing or only partially implementing particular
practices can produce negative consequences or add risk to a project. For
example, the Department of the Navy project’s baseline of its existing
environment (a phase I practice) was limited because it did not include an
assessment of its legacy applications. Instead, project officials decided to
rely on a preexisting inventory developed to address the Year 2000
challenge. The Navy subsequently found that it had substantially
underestimated the number of legacy applications, which, according to
program officials, later contributed to the transition period slipping from
2-½ years to 3-½ years.

As DOD gathers more experience in implementing projects for outsourcing
IT services, it can benefit from leveraging the lessons derived from these
initiatives. For example, the projects in our review have identified lessons
learned in such areas as transitioning to the provider and partnering with
the provider. Although DOD has taken action to gather and disseminate
lessons learned and best practices on general acquisition issues, these
efforts generally do not focus on outsourcing or include sharing the lessons
learned from IT outsourcing projects across the department. By not
capturing and disseminating such information in a systematic manner
across the department, DOD is losing the opportunity to leverage the
knowledge gained on IT services projects like those in our review.

We are making recommendations to the Secretary of Defense aimed at
leveraging lessons learned across the department from its components’ IT
services outsourcing experiences.

In written comments on a draft of this report signed by the DOD Principal
Director, Deputy Assistant Secretary of Defense (Deputy Chief Information
Officer), DOD agreed that capturing lessons learned in the development
and implementation of its IT outsourcing initiatives is important to
continually improving its outsourcing methods and results achieved. The

7
 This consensus view was based on interviews with managers in leading commercial
organizations, discussions with academic and professional authorities, and extensive
research on IT acquisition practices.




Page 4                                           GAO-03-371 DOD IT Services Outsourcing
             department also stated that it intends to explore a variety of mechanisms
             for best exploiting lessons learned from its IT outsourcing initiatives. We
             agree that it is prudent to consider alternative means to leveraging these
             lessons learned, and we believe that this is consistent with our
             recommendations.



Background   To protect the security of the United States, DOD relies on a complex array
             of computer-dependent and mutually supportive organizational
             components, including the military services, Commanders in Chief, and
             Defense agencies. As such, it invests tens of billions of dollars each year in
             a broad array of computer systems, which include weapon systems,
             command and control systems, satellite systems, inventory management
             systems, transportation management systems, health systems, financial
             systems, personnel systems, and payment systems. In addition, DOD
             spends billions of dollars annually on IT services, which include database
             management, help-desk operations, software maintenance, and network
             services. In fiscal year 2001, DOD reportedly obligated more than $6.2
             billion on IT services alone.8

             Decisions regarding the purchasing of services are critical to ensuring the
             effectiveness of DOD’s operations as well as those of the government as a
             whole. Our November 2001 report recognizes the importance of such
             sourcing decisions and provides a framework that spans the full range of
             activities that are performed during IT services outsourcing.9 At the same
             time, governmentwide policies, initiatives, and challenges exist that
             significantly influence the government’s sourcing decisions.




             8
              This figure is from the Federal Procurement Data System, which contains detailed
             information on contract actions over $25,000.
             9
             GAO-02-214.




             Page 5                                           GAO-03-371 DOD IT Services Outsourcing
GAO’s Framework for       Outsourcing of IT services has become increasingly popular in both the
Outsourcing IT Services   public and private sector. For example, according to the Giga Information
                          Group, Inc., a leading research firm, such outsourcing is expected to grow
                          an average of 5 to 6 percent in 2003.10 The federal sector’s outsourcing is
                          predicted to rise at an even greater rate. For example, INPUT, an IT market
                          research firm, forecasts that defense IT outsourcing will increase about 143
                          percent between fiscal years 2002 and 2007.11

                          IT outsourcing involves the activities associated with acquiring services
                          from one or more external providers. During outsourcing, a client
                          organization transfers responsibilities for performing one or more IT
                          services to one or more external providers. This responsibility is executed
                          through control and management of the processes, people, and technology
                          associated with these services.

                          Figure 2 depicts the roles of the client and provider organizations in an
                          outsourcing relationship.



                          Figure 2: Roles of the Client and Provider in an Outsourcing Relationship




                          10
                            Giga Information Group, Inc., IT Trends 2003: IT Services (Dec. 19, 2002). We did not
                          independently verify these data.
                          11
                            INPUT, The Federal IT Outsourcing Market View (December 2002). We did not
                          independently verify these data.




                          Page 6                                            GAO-03-371 DOD IT Services Outsourcing
Our November 2001 guide on leading commercial practices for outsourcing
IT services provides a generic framework of practices from leading
commercial organizations that can improve purchasing decisions and
manage the resulting government/provider relationship.12 The framework is
represented in figure 3 as a hierarchy of phases, practices, and critical
success factors.



Figure 3: GAO’s Framework for Outsourcing IT Services




Note: The arrow from phase VII to phase I represents the need to reflect on lessons learned from
previous phases. The arrows between phase III and IV represent the iterative nature of developing the
contract and selecting the provider. Although there is a logical order to the sequence of the common
phases, the order of the practices within each phase does not imply any priority or sequence.


Table 1 provides a definition of each phase of the framework. Each of the
phases has specific practices associated with it. Implementing these
practices does not guarantee the success of an outsourcing project.
However, our November 2001 study reflected a consensus view that these
practices were the most critical to success when IT services are being
acquired.13 Restated, application of these practices increases the
probability of a successful outsourcing project.




12
     GAO-02-214.
13
 This consensus view was based on interviews with managers in leading commercial
organizations, discussions with academic and professional authorities, and extensive
research on IT acquisition practices.




Page 7                                                 GAO-03-371 DOD IT Services Outsourcing
Table 1: Definition of Phases for IT Outsourcing

Phase
number         Title                                     Definition
I              Determine sourcing strategy               Client organizations determine whether internal capability or external expertise
                                                         can more effectively meet IT needs.
II             Define operational model                  Client organizations formalize executive leadership, team composition, client
                                                         responsibilities, and operating relationships between client and provider
                                                         organizations.
III            Develop the contract                      Client organizations establish the legal terms for the IT outsourcing relationship.
IV             Select the provider(s)                    Client organizations find one or more providers who can help them reach their
                                                         IT outsourcing goals.
V              Transition to provider(s)                 Client organizations transfer responsibility of IT functions to one or more
                                                         providers.
VI             Manage provider(s) performance            Client organizations make sure each provider is meeting performance
                                                         requirements.
VII            Ensure services are provided              Client organizations make sure that services are provided and end-user needs
                                                         are met.
Source: GAO.


                                                The organizations that we studied also identified certain capabilities
                                                (identified as critical success factors) that were essential for implementing
                                                the practices identified in our framework. First, executive leadership
                                                strengthens the interaction between executive management and the
                                                employees of the client organization. Second, partner alignment
                                                strengthens the interaction between the client and provider organization at
                                                the executive level, which ensures that the goals and objectives of these
                                                organizations support each other. Third, relationship management
                                                strengthens the interaction between the client and provider organization at
                                                the operational level.



Influences on Government                        The federal government is one of the world’s largest users of services.
Sourcing Decisions                              Because of the large dollar value and the number of private- and public-
                                                sector jobs involved, deciding whether the public or private sector would
                                                be the most appropriate provider of the services the government needs (IT
                                                or otherwise) is an important, and often highly charged, question. Among
                                                the factors that agencies must consider as they determine how best to meet
                                                their missions is whether the public or private sector would be the most
                                                appropriate provider of the services the government needs. Phase I of our
                                                framework, determine the sourcing strategy, addresses the client’s




                                                Page 8                                            GAO-03-371 DOD IT Services Outsourcing
assessment of whether expertise from within or outside of the organization
can more effectively meet the client’s needs.

The National Defense Authorization Act for Fiscal Year 2001 required the
Comptroller General of the United States to convene a panel of experts to
study the current process used by the government to make sourcing
decisions. The resulting Commercial Activities Panel conducted a year-long
study and heard repeatedly about the importance of competition and its
central role in fostering economy, efficiency, and continuous performance
improvement. In particular, the panel reviewed the government’s
implementation of the Office of Management and Budget’s (OMB) Circular
A-76, which sets forth federal policy for determining whether federal
employees or private contractors will perform commercial activities for the
government.14 Circular A-76 (1) outlines conditions under which agencies
are permitted to perform a commercial activity with government
employees or by contract and (2) provides guidance for whether, and if so,
how, agencies should conduct a cost comparison when they are
considering transferring the performance of commercial activities from the
public to the private sector (or vice versa). The panel reported that there
were positive elements to Circular A-76 but that both federal employees
and private firms complained that it does not meet the standard of a clear,
transparent, and consistently applied process. For example, both federal
employees and private firms criticized the Circular A-76 process as unequal
and therefore unfair.

The Commercial Activities Panel strongly supported continued emphasis
on competition and concluded that whenever the government is
considering converting work from one sector to another, public/private
competitions should be the norm. In addition, the panel made four
recommendations, including that all sourcing decisions be consistent with
the principles adopted unanimously by the panel, such as the principle that
federal policy provide for accountability in connection with all sourcing
decisions.15



14
   Office of Management and Budget, Circular A-76, Performance of Commercial Activities
(Washington, D.C.: June 14, 1999). In November 2002, OMB issued proposed revisions that
would substantially change this circular. As of April 8, 2003, these revisions have not yet
been made final.
15
 Final report of the Commercial Activities Panel, Improving the Sourcing Decisions of the
Government (Washington, D.C.: Apr. 30, 2002).




Page 9                                            GAO-03-371 DOD IT Services Outsourcing
As part of the administration’s efforts to implement the recommendations
of the Commercial Activities Panel, OMB has published proposed changes
to Circular A-76. Key highlights of the proposed changes include presuming
that all functions are commercial in nature unless they are justified as
inherently governmental;16 limiting the length of time for competitions; and
emphasizing awarding contracts on the basis of best value, not just lowest
cost. Best value allows the contracting official to consider technical
superiority, quality, innovation, and past performance as well as price.

However, we reported that there are several areas in which the proposed
revisions to the circular are not consistent with the principles or
recommendations of the Commercial Activities Panel.17 Specifically, the
proposed revision does not include a link between sourcing policy and
agency missions, has unnecessarily complicated source selection
procedures, contains certain unrealistic time frames, and includes
insufficient guidance on calculating savings.

Beyond the Commercial Activities Panel, other bodies have identified
challenges that the federal government faces in reaching and executing
effective sourcing decisions. For example, members of the Coalition for
Government Procurement, the Professional Services Council, and the
Information Technology Association of America told us that organizational
culture is one of the biggest differences between the commercial sector
and the federal government and one of the greatest barriers to the
government’s use of commercial practices. Also, as we have previously
reported, moving to outsourcing solutions can involve a cultural change for
government organizations because it may require a change to an agency’s
operating model, such as using a contractor to provide IT services
previously performed by government staff or using a performance-based
contract.18 This view was echoed by a 2001 study of DOD competitive




16
 Section 5 of P.L. 105-270, codified at 31 U.S.C. 501 note (1998), defines an inherently
governmental function as a “function that is so intimately related to the public interest as to
require performance by Federal Government employees.”
17
 U.S. General Accounting Office, Proposed Revisions to OMB Circular A-76, GAO-03-391R
(Washington, D.C.: Jan. 16, 2003).
18
 U.S. General Accounting Office, Desktop Outsourcing: Positive Results Reported, but
Analyses Could Be Strengthened, GAO-02-329 (Washington, D.C.: Mar. 29, 2002).




Page 10                                             GAO-03-371 DOD IT Services Outsourcing
sourcing that found cultural, process, execution, and training barriers.19
The study stated that these barriers need to be understood and mitigated
before the benefits of outsourcing can be fully realized. Barriers such as
these can be overcome by strong executive leadership, which is a critical
success factor in our framework.

Another challenge is creating a productive agency/provider relationship—
another critical success factor in our framework. According to a report
sponsored by the PricewaterhouseCoopers Endowment for the Business of
Government, such public/private partnerships are based on trust,
commitment to problem or conflict resolution, and the recognition that
flexibility is necessary and that the relationship will evolve and change over
time.20 If deadlines are not met, or public agency goals change with
differing political climates, the partners need to discuss the basis of the
partnership and construct a different relationship. Our prior report on
desktop outsourcing found that developing a productive agency/contractor
relationship is not always easy.21 Both sides must recognize and understand
each other’s underlying motives and strive to achieve established
expectations.

Finally, human capital issues are another challenge facing federal agencies
that affect their ability to implement outsourcing. Our framework
recognizes the importance of having the right skills in place to support the
outsourcing relationship. However, as we have previously reported,
procurement reforms and technological changes have placed
unprecedented demands on the acquisition workforce.22 Contracting
personnel are now expected to have a much greater knowledge of market
conditions, industry trends, and the technical details of the commodities
and services they procure. The Commercial Activities Panel report stated
that developing and maintaining a skilled acquisition workforce is the
critical first step in managing this more complex procurement
environment. The panel also reported that DOD bore the brunt of a 22


19
   The Industrial College of the Armed Forces, Case Study: Complex Business Management
for Competitive Sourcing (2001).
20
 PricewaterhouseCoopers Endowment for the Business of Government, Contracting for
the 21st Century: A Partnership Model (January 2002).
21
     GAO-02-329.
22
   U.S. General Accounting Office, Contract Management: Taking a Strategic Approach to
Improving Service Acquisitions, GAO-02-499T (Washington, D.C.: Mar. 7, 2002).




Page 11                                         GAO-03-371 DOD IT Services Outsourcing
                               percent downsizing of the federal acquisition workforce in the last decade,
                               going from 96,000 staff in 1991 to about 68,000 in fiscal year 2001.

                               Addressing human capital issues is not just a matter of the size of the
                               workforce; it is also a knowledge and skills issue. According to the
                               Commercial Activities Panel, it is critically important that federal agencies
                               adequately address human capital needs in meeting the current and
                               emerging needs of government and its citizens in the most effective,
                               efficient, and economical manner possible. This will require increased
                               emphasis on training and development, particularly in the area of
                               technology.



Description of Five Projects   The five projects in our study varied in how they approached outsourcing
Reviewed                       IT services, such as in using various solicitation methods, including holding
                               a public/private competition under the policies outlined in OMB Circular A-
                               76 or carrying out a negotiated competitive procurement. In addition, the
                               types of services being outsourced differed: services ranged from the
                               narrowly focused (e.g., help-desk services) to the very broad (e.g.,
                               enterprisewide end-to-end information services); contract terms ranged
                               from 5 to 15 years (assuming all option years are exercised); and estimated
                               contract values ranged from $23 million to $8.8 billion. Table 2 provides
                               information on the variety of IT services and outsourcing approaches taken
                               by the projects.




                               Page 12                                   GAO-03-371 DOD IT Services Outsourcing
Table 2: Profile of Outsourcing Projects Reviewed

                                          Date of
                         Solicitation     contract        Contract        Contract         Estimated total
Agency/Project           method           award           term            type             contract valuea       Project description
Air Force/Kirtland Air   Competitive,     April 2000      1 year, with    Firm, fixed-     $23 million           Management, operations, and
Force Base’s             under OMB                        4 option        price                                  maintenance of command,
Command, Control,        Circular A-76                    years                                                  control, communications, and
Communications, and      and small                                                                               computer systems, multimedia
Computer (C4)            business set-                                                                           services, and information
Services                 aside rules                                                                             management for the Kirtland Air
                                                                                                                 Force Base.
Army/Network          Negotiated          March 2001      1 year, with    Firm, fixed-     $204 million          Operation and maintenance
Enterprise Technology competitive                         4 option        price                                  services, including repair,
Command’s             solicitation                        years                                                  installation, and supply, for
(NETCOM) Total Army                                                                                              communications equipment in
Communications—                                                                                                  Southwest Asia.
Southwest Asia (TAC-
SWA)
Military Health          Negotiated       June 2001       1 year, with    Firm, fixed-     $71 million           Call and help-desk services for
System/Information       competitive                      7 option        price with                             all MHS software applications.
Technology               solicitation                     years           incentive
Organization                                                              awards
(MHS/ITO) Help Desk
Department of the     Negotiated          October         7 years,        Firm, fixed-     $8.8 billion          Department of the Navy-wide
Navy/Navy and Marine competitive          2000            with an         price with                             end-to-end information services
Corps Intranet (NMCI) solicitation                        option for 3    incentive                              through a common computing
                                                          additional      awards                                 and communication
                                                          years                                                  environment.
National Imagery and     Sole-source,     December        1 year, with    Cost plus        $2.1 billion          NIMA-wide IT/IS support
Mapping Agency’s         using a          2001            14 option       award fees                             services for printing, digital
(NIMA) Information       statutory                        years                                                  replication, networks, distributed
Technology/              preferential                                                                            and centralized systems and
Information Services     provider                                                                                services, video and voice
(IT/IS)                  (Alaska native                                                                          communications, information
                         corporation)b                                                                           research, and help-desk
                                                                                                                 operations.c
Source: DOD.
                                              a
                                               Estimated value if all option years are exercised.
                                              b
                                               NIMA performed an OMB Circular A-76 analysis and, on the basis of this analysis, implemented a
                                              direct conversion to a preferential provider rather than holding a public/private competition or obtaining
                                              an agency cost-comparison waiver.
                                              c
                                                As of February 24, 2003, NIMA had transitioned four of these functions to the provider: printing, digital
                                              replication, video and voice communications, and help-desk operations.




                                              Page 13                                                     GAO-03-371 DOD IT Services Outsourcing
Projects Substantially   As illustrated in figure 4, the five IT services projects substantially used
                         leading commercial practices. Specifically, each project used at least 76
Used Leading             percent of the practices.23 Reasons for projects implementing different
Commercial Practices     percentages of the practices include differences in their individual
                         circumstances and objectives. For example, the Army’s Total Army
                         Communications—Southwest Asia (TAC-SWA) project, which used the
                         fewest number of practices and had the largest number of practices that
                         were not applicable, was largely a continuation of an existing approach
                         that already relied on the private sector but with fewer providers. In
                         contrast, the National Imagery and Mapping Agency’s (NIMA) Information
                         Technology/ Information Services (IT/IS) project, which fully or partially
                         implemented all of the applicable practices, involved a significant
                         operational shift (e.g., functions previously performed by NIMA staff are
                         now performed by a contractor) and was intended to result in substantial
                         process improvements. In addition, the three projects that implemented the
                         largest percentage of practices also used third-party assistance—including
                         employing a contractor with sourcing expertise—to help formulate their
                         sourcing strategy, which could account for the extent of their compliance.




                         23
                              This calculation does not include practices that were not applicable to a particular project.




                         Page 14                                                GAO-03-371 DOD IT Services Outsourcing
Figure 4: Use of Leading Commercial Practices for Outsourcing of IT Services, by
Project




Note: Not applicable—The practice was not relevant to the project’s particular circumstances.
No—The agency did not implement the practice. Limited—The agency fully implemented some but not
all aspects of the practice and did not take alternative actions that fully satisfied the practice. Yes—The
agency fully implemented the practice or took an alternative action that fully satisfied its intent.


In addition, figure 5 illustrates that project compliance extended to each of
the phases of our framework. Collectively, the projects fully implemented
from 70 to 97 percent of the practices in each phase. Phase I, determine
sourcing strategy, had the lowest percentage of practices implemented by
the projects (70 percent). This result is not inconsistent with a recent Giga
Information Group, Inc., survey, which found that only half of the




Page 15                                                   GAO-03-371 DOD IT Services Outsourcing
respondent organizations had documented an IT sourcing strategy.24 This
approach carries risk since phase I sets the tone for the outsourcing
initiative within the client organization.



Figure 5: Percentage of Practices Implemented in Each Phase




Note: Not applicable—The practice was not relevant to the project’s particular circumstances.
No—The agency did not implement the practice. Limited—The agency fully implemented some but not
all aspects of the practice and did not take alternative actions that fully satisfied the practice. Yes—The
agency fully implemented the practice or took an alternative action that fully satisfied its intent.


24
 Giga Information Group, Inc., Optimizing IT Sourcing Strategy: Key Stages and Phases
of the IT Sourcing Process (Jan. 31, 2003).




Page 16                                                   GAO-03-371 DOD IT Services Outsourcing
                                          In addition, collectively the projects fully implemented 88 percent of the
                                          practices (see table 3).25



Table 3: Percentage of Practices Implemented, by Project

                                                                 Percentage of practices implementeda
                                       Air Force              Army
                                     C4 Services           TAC-SWA MHS/ITO Help                Navy NMCI          NIMA IT/IS
Phase                                    project             project Desk project                 project            project               Overall
Phase I: Determine sourcing
strategy (6 practices).                          50                33                83                  83                100                 70
Phase II: Define operational
model (13 practices).                            92                83              100                   77                100                 90
Phase III: Develop the contract
(16 practices).                                  64                83              100                   94                100                 89
Phase IV: Select provider(s) (7
practices).                                      86                86              100                  100                 71                 89
Phase V: Transition to provider(s)
(11 practices).                                  82                88              100                   82                100                 90
Phase VI: Manage provider(s)
performance (11 practices).                      73                73              100                   91                 82                 84
Phase VII: Ensure services are
provided (6 practices).                          83              100               100                  100                100                 97
Overall                                          76                79                99                  89                 94                 88
Source: GAO.
                                          a
                                           These calculations do not include practices that were not applicable to a particular project.


                                          The following provides additional information on the projects’
                                          implementation of each phase of our framework.




                                          25
                                               This calculation does not include practices that were not applicable to a particular project.




                                          Page 17                                                  GAO-03-371 DOD IT Services Outsourcing
• Phase I: Determine sourcing strategy. In the first phase of our
  outsourcing framework, the client organization determines whether
  internal capability or external expertise can more effectively meet its IT
  needs. The purpose of a sourcing strategy is to achieve the optimal
  balance between internal and external capabilities, activities, processes,
  and services to ensure the achievement of strategic business objectives
  at the lowest risk.26 The five projects’ implementation of this phase was
  uneven. In particular, two of the six practices in this phase were fully
  implemented by all five projects, but the other four practices were not.
  Among the practices that were implemented by all of the projects was
  determining the business reasons for outsourcing. In addition, the three
  projects that implemented the largest percentage of practices in our
  framework—the Military Health System/Information Technology
  Organization (MHS/ITO) Help Desk, the Department of the Navy’s Navy
  and Marine Corps Intranet (NMCI), and NIMA IT/IS projects—used the
  third-party assistance practice in this phase to help formulate their
  sourcing strategy, which could account for the extent of their
  compliance. For example, the MHS/ITO Help Desk project, which
  implemented the largest percentage of practices, worked with the
  Department of the Interior’s GovWorks Program,27 the Defense
  Acquisition University staff, and a private-sector contractor to obtain
  expertise on sourcing strategies.

     The practice in this phase that was the most unevenly implemented was
     the benchmarking28 and baselining of existing internal services. Of the
     five projects in our review, (1) one fully benchmarked and baselined the
     productivity of the activity being outsourced before making the final
     sourcing decision, (2) two partially baselined their existing activities,
     and (3) two did not perform benchmark and baseline analyses at all.
     The agencies’ reasons for not fully implementing this practice included
     that an executive decision had been made to conduct a public/private
     competition following the OMB Circular A-76 policy, so such an
     analysis would not have affected the sourcing decision, or that

26
   Gartner, Inc., Strategic Analysis Report, How to Build a Sourcing Strategy, Research
Note R-18-1099 (Sept. 23, 2002).
27
 The Department of the Interior’s GovWorks Program is a federal fee-for-service acquisition
center that helps other federal agencies acquire supplies and services for their programs on
a project-by-project basis.
28
 Gartner, Inc., defines benchmarking as a method to compare the cost or price of an IT
environment to peer groups with the same workload characteristics.




Page 18                                           GAO-03-371 DOD IT Services Outsourcing
        available documentation to perform such an analysis was limited.
        Leading research firms suggest benchmarking and baselining the
        entity’s current processes before outsourcing because only then would
        it be able to determine whether the arrangement has been successful.29
        In addition, the risk of not fully baselining the existing environment is
        illustrated by the NMCI project. Specifically, the NMCI project’s
        baseline of its existing environment was limited because it did not
        include an assessment of its legacy applications since project officials
        decided to rely on a preexisting inventory developed to address the
        Year 2000 challenge. The Navy subsequently found that it had
        substantially underestimated its number of legacy applications, that,
        according to program officials, later contributed to the transition
        period slipping from 2-½ years to 3-½ years. Appendix II provides
        additional information on projects’ implementation of the practices
        comprising this phase.

• Phase II: Define operational model. The operational model is an
  important mechanism for an organization to compare its plans with the
  expectations that were set when the decision to outsource was made
  and to ascertain whether these plans will enable the organization to
  meet expectations. The five projects had largely implemented the 13
  practices contained in this phase. Specifically, about 90 percent30 of the
  practices were implemented. For example, all projects implemented the
  practice that executive leadership be established to facilitate the
  outsourcing effort. NIMA, for instance, formed a strategic sourcing
  office to oversee the IT/IS project. Another practice—training the
  provider on the organization’s business environment and goals—was
  fully implemented by one project (in two cases, the practice was not
  applicable). One project that did not fully implement this practice was
  the Air Force’s Kirtland Air Force Base’s Command, Control,
  Communications, and Computer (C4) Services project. Although some
  training was provided (e.g., Kirtland held an orientation session for
  potential bidders), provider officials stated that they did not receive
  adequate training, which made the transition period more difficult.
  Appendix III provides additional information on projects’
  implementation of the practices comprising this phase.

29
 Giga Information Group, Inc., Payment and Incentives for Outsourcing Management
(July 27, 2000) and Gartner, Inc., Benchmarking Helps Outsourcing Deals Stay
Competitive, Research Note COM-16-8055 (June 14, 2002).
30
     This calculation does not include practices that were not applicable to a particular project.




Page 19                                                GAO-03-371 DOD IT Services Outsourcing
• Phase III: Develop the contract. A well-written contract is necessary for
  the outsourcing organization to meet its requirements while allowing
  the service provider to make a fair profit. It sets the expectations for
  service levels, delivery of essential services, and continuous
  improvement and should protect the interests of all parties. The five
  projects largely implemented the practices in this phase. Specifically,
  about 89 percent31 of the practices were implemented in this phase, and
  two projects (the MHS/ITO Help Desk and NIMA IT/IS projects)
  implemented all of the practices. Several practices in this phase address
  performance requirements. For example, all five projects implemented
  the practices that called for basing performance requirements on
  business requirements and reviewing and updating them periodically.
  One practice that was not fully implemented by two projects was
  including performance measures that address both technical and end-
  user satisfaction aspects of performance. For example, the Army TAC-
  SWA project included technical performance measures in its contract
  but not measures related to end-user satisfaction, even though the
  contract included help-desk services. According to the project official
  that developed the performance work statement in the contract, the
  command did not include customer satisfaction measures because it did
  not think that it was necessary to have a performance standard for the
  help-desk service. However, without such measures, the agency does
  not have a contractual standard with which to judge the provider’s
  performance. Appendix IV provides additional information on projects’
  implementation of the practices comprising this phase.

• Phase IV: Select the provider(s). Critical to the success of any
  outsourcing project for IT services is identifying potential providers and
  ultimately selecting a provider(s) that will best meet the needs of the
  agency. The five projects had largely implemented the seven practices
  contained in this phase. About 89 percent of the practices were
  implemented in this phase and two projects (the MHS/ITO Help Desk
  and Navy NMCI projects) used each applicable practice. For example,
  the five projects implemented the practice related to conducting due
  diligence activities to verify provider capabilities before signing the
  contract. In the case of the Army TAC-SWA project, the Network
  Enterprise Technology Command (NETCOM) evaluated the provider’s
  financial and past performance information. The Department of the
  Navy’s NMCI project also evaluated bidders’ past performance and


31
     This calculation does not include practices that were not applicable to a particular project.




Page 20                                                GAO-03-371 DOD IT Services Outsourcing
       performed reference checks. Moreover, as part of its due diligence
       activities, the Navy required all bidders to demonstrate that they had
       experience in implementing large seat management contracts.32 A
       practice that was not implemented by two projects was using third-party
       assistance when selecting the provider. Projects that did not implement
       this practice believed that they had adequate in-house expertise with
       outsourcing, making third-party assistance unnecessary. However,
       because third-party assistance provides an independent resource that
       can suggest options or processes that the client organization may not be
       aware of, these projects may have missed an opportunity to implement
       their outsourcing projects more effectively. Appendix V provides
       additional information on projects’ implementation of the practices
       comprising this phase.

• Phase V: Transition to provider(s). This phase focuses on the client
  organization’s transfer of the IT function to one or more providers. As
  part of this transition, the clear definition of responsibilities and the
  careful consideration of employees’ needs matched against the client
  organization’s needs enable both the client and provider to focus on
  execution and give staff confidence in their future employment. The five
  projects largely implemented the 11 practices associated with this
  phase. Specifically, about 90 percent33 of the practices were
  implemented in this phase, and two projects implemented all of the
  practices (the MHS/ITO Help Desk and NIMA IT/IS projects). Several
  practices in this phase address dealing with employees affected by the
  outsourcing projects. For example, in the four projects in which federal
  employees were affected, the projects provided assistance to those who
  did not want to transfer to the provider, including helping to place them
  in other positions and helping with résumé writing. A related practice is
  to clearly communicate to all employees what is going to happen and
  when it is going to happen. Two projects did not fully implement this
  practice. For example, the Navy used its normal chain of command to
  communicate transition information, but found that implementation of
  this practice was uneven. As a result, some staff did not know current
  information about how NMCI would affect them until the provider was
  ready to contact them regarding their possible transition to the

32
 Seat management generally refers to service provision arrangements in which contractor-
owned desktop and other computing hardware, software, and related services are bundled
and provided to a client organization at a fixed price per unit (or seat).
33
     This calculation does not include practices that were not applicable to a particular project.




Page 21                                                GAO-03-371 DOD IT Services Outsourcing
     contractor. However, according to the NMCI’s Director’s office, this
     problem was somewhat mitigated by the provider’s Web site that
     provides transition information to all NMCI customers/users. Appendix
     VI provides additional information on projects’ implementation of the
     practices comprising this phase.

• Phase VI: Manage provider(s) performance. The effectiveness with
  which the provider(s) performance is managed is critical to the
  successful implementation of an outsourcing project. Indeed, according
  to Gartner, Inc., a leading research firm, an outsourcing project can be
  thwarted by poorly designed, funded, and delivered processes for
  managing the delivery of services.34 The five projects generally
  implemented the 11 practices contained in this phase, with about 84
  percent of the practices being implemented in this phase. For example,
  the practices related to obtaining feedback on provider performance
  were largely implemented. This is important because different levels of
  an agency can have different perceptions about the value of the
  outsourcing project. For example, an outsourcing project may be
  considered successful by the agency’s executive management if it is
  focused on controlling costs, but be considered inadequate by business
  managers and users who may be expecting higher levels of service. Each
  of these viewpoints is valid and should be taken into account when the
  provider’s performance is evaluated. Two other practices—including
  incentives and penalties in contracts—were fully implemented by two
  projects. Incentives and penalties are important because they can help
  motivate the provider to exceed or meet performance requirements.
  Nonetheless, two projects did not include monetary incentives and two
  projects did not include monetary penalties in their contracts. For
  example, the Army TAC-SWA project did not include monetary
  incentives, although the contracting officer stated that incentives might
  have been useful to motivate the provider to exceed performance
  requirements. Incentives can also help control risks. According to a
  guide on performance-based services acquisition, if the incentives in the
  contract are right and if the provider and agency share the same goals,
  risk is largely controlled and effective performance is “almost the




34
 Gartner, Inc., Retain Enough Resources to Manage Outsourcing Deals, Research Note
COM-16-8425 (June 17, 2002).




Page 22                                       GAO-03-371 DOD IT Services Outsourcing
                          inevitable outcome.”35 Appendix VII provides additional information on
                          projects’ implementation of the practices comprising this phase.

                     • Phase VII: Ensure services are provided. Although outsourcing
                       transfers responsibility for performing the service to the provider(s), the
                       client organization is ultimately responsible for ensuring that services
                       are provided and that end-user needs are met. Accordingly, it is critical
                       that the agency ensure that services are provided. The projects had
                       implemented 97 percent of these practices, and four projects
                       implemented all of them. For example, every project monitored the
                       providers’ work. In the case of the Air Force’s C4 Services project,
                       quality assurance evaluators monitored the provider’s work to identify
                       problems or trends in accordance with the project’s quality assurance
                       surveillance plan. The results were reported to the contracting officer
                       and to the functional area chief for resolution. Another practice, using
                       customer satisfaction surveys, was fully implemented by four of the five
                       projects. However, the Air Force project did not conduct, or require its
                       contractor to conduct, customer satisfaction surveys. Although the
                       provider surveys staff annually, the Air Force is nevertheless relying on
                       the provider to voluntarily implement an important practice for
                       determining how customers view the services being delivered and
                       whether changes need to be made. Appendix VIII provides additional
                       information on projects’ implementation of the practices comprising
                       this phase.



Leveraging Lessons   We have previously reported on the importance of collecting and
                     disseminating lessons learned.36 For example, a critical activity in IT
Learned DOD-wide     investment management is establishing a process for developing and
Could Assist Other   capturing lessons learned in a written product or knowledge base and
                     disseminating them to decision-makers.37 In addition, one of the practices
DOD Projects         in our framework for outsourcing IT services addresses incorporating


                     35
                      An Interagency-Industry Partnership in Performance, Seven Steps to Performance-Based
                     Services Acquisition, Benchmark Version (January 2002).
                     36
                      U.S. General Accounting Office, NASA: Better Mechanisms Needed for Sharing Lessons
                     Learned, GAO-02-195 (Washington, D.C.: Jan. 30, 2002).
                     37
                      U.S. General Accounting Office, Information Technology Investment Management: A
                     Framework for Assessing and Improving Process Maturity, GAO/AIMD-10-1.23, Exposure
                     Draft (Washington, D.C.: May 2000).




                     Page 23                                        GAO-03-371 DOD IT Services Outsourcing
lessons learned from peers who have engaged in similar sourcing
decisions. Use of lessons learned is a principal component of an
organizational culture committed to continuous improvement. Sharing
such information serves to communicate acquired knowledge more
effectively and to ensure that beneficial information is factored into
planning, work processes, and activities. Lessons learned can be based on
positive experiences or on negative experiences that result in undesirable
outcomes.

Although DOD has taken action to gather and disseminate lessons learned
and best practices on general acquisition issues, these efforts generally do
not focus on outsourcing or include sharing the lessons learned from IT
outsourcing projects across the department. Specifically, a number of DOD
Web sites provide guidance, lessons learned, and best practices related to
general acquisition issues. However, using these sites to locate specific
information on IT outsourcing best practices and lessons learned can be
time-consuming and difficult because so many topics and information
sources are provided. Specifically, MHS/ITO Help Desk project officials
said that searching numerous Web sites to get relevant information to
address questions and concerns about outsourcing IT services can
consume hours. For example, when we entered the keywords “IT
outsourcing” and “best practices” into the search feature on the Office of
the Undersecretary for Defense of Acquisition, Technology, and Logistics
site, ACQWeb (www.acq.osd.mil) in early March, it provided us with links
to 1,251 documents. Likewise, lessons learned covers so many topics that it
is difficult to search for an applicable IT lesson. For example, when we
used the phrase “lessons learned in IT outsourcing” no documents were
identified on the ACQWeb, but when we inserted “lessons learned” and “IT
outsourcing” links to more than 1,700 documents were produced.

One DOD Web site, Share A-76!, was established to address one of our
previous recommendations,38 that is, to establish a framework for
identifying and analyzing best practices and lessons learned from
competitive sourcing studies and disseminating them DOD-wide. Share A-
76! promotes the sharing of best practices and lessons learned related to
one form of outsourcing that was conducted under the OMB Circular A-76
competitive sourcing process. Among other things, the site contains
guidance, links to other relevant sites, sample documents, and a best


38
 U.S. General Accounting Office, DOD Competitive Sourcing: Lessons Learned System
Could Enhance A-76 Study Process, GAO/NSIAD-99-152 (Washington, D.C.: July 21, 1999).




Page 24                                        GAO-03-371 DOD IT Services Outsourcing
practices library that communicates field staff experiences and advice
about the Circular A-76 process. A NIMA project official said that NIMA’s
staff routinely accesses the Web site because it contains a wealth of
information on policies, procedures, lessons learned, and links to other
outsourcing sites. The DOD analyst responsible for Share A-76! estimated
that the site receives about 12,000 visits per month and said that on the
basis of E-mails and anecdotes, user satisfaction is favorable. However, this
Web site is specific to the OMB Circular A-76 process, which may not apply
to other types of outsourcing. For example, Circular A-76’s policy pertains
to public/private competitions and requires that the final evaluation
between the government and the private sector be based exclusively on
cost.

DOD acquisition and IT officials acknowledged that there is no mechanism
in DOD to easily share and leverage lessons learned relating to outsourcing
IT services. However, these officials agreed that a departmentwide effort to
identify, capture, and disseminate lessons learned and leading practices of
projects with experience in carrying out IT outsourcing could offer
valuable insights and new ideas that would benefit others. Moreover,
officials from three of the projects in our review told us that there is value
in collecting and disseminating the knowledge acquired from IT
outsourcing projects in a systematic manner across the department.

Each of the projects in our review identified knowledge and experience
gained from their approaches to outsourcing IT services that could offer
insights and practices for other ongoing and future projects to consider.
For example:

• MHS developed specific guidance and lessons learned for implementing
  a performance-based incentive contract for help-desk operations.

• The Department of the Navy’s NMCI project has developed a series of
  lessons learned related to transitioning to the provider that is being
  shared within the NMCI community; one example was that all personnel
  should be available during scheduled testing and deployment.

• NIMA has experience in contracting techniques emphasizing a
  partnering approach with providers to refine requirements and establish
  a common understanding of costs.

In addition, a departmental IT outsourcing knowledge-sharing approach
could include links to information about other government agencies’ IT



Page 25                                   GAO-03-371 DOD IT Services Outsourcing
outsourcing projects. For example, our 2002 report on desktop outsourcing
includes an extensive discussion of lessons learned by agencies that have
implemented this type of IT services outsourcing.39

Developing an effective lessons learned activity is not easy. For example,
NMCI officials said that for a lessons learned initiative to be effective, a
process must exist that is clearly understood by everyone and allows
capturing and sharing of knowledge to occur with minimum effort. Other
challenges in developing an effective lessons learned process were outlined
by the Share A-76! analyst. The analyst stated that only a small number of
site users have contributed lessons learned to the Share A-76! Web site,
which she attributed, in part, to the amount of time and effort needed to
document and obtain agreement by all levels of the organization on the
lessons learned. In addition, the analyst stated that there is reluctance to
share negative lessons, and often the review and approval process sanitizes
best practice information so that it becomes too general to be most helpful
to users. Such challenges can be overcome by executive-level support.
Indeed, DOD acquisition and IT officials stated that for lessons learning
activities to be effective, senior management must devote support and
resources to the effort. This is consistent with our prior work, which
showed that knowledge can be effectively shared only when employees are
given adequate time as well as established places where they can actually
transfer knowledge.40

Last year, we outlined a generic lessons learned process that could be used
to guide the development of such a process for outsourcing IT services.41
Although the mechanism or processes used to collect, share, and
disseminate lessons learned may vary, in general such a process comprises
four main elements: collection, verification, storage, and dissemination.
The collection process involves the capture of information through
structured and unstructured processes. Verification serves to verify the
correctness and applicability of lessons submitted. The storage aspect of
lessons learned usually involves incorporating the lessons into an
electronic database for the dissemination and sharing of information,
including the ability to conduct information searches. The final element,
and the most important, is the dissemination of lessons learned, since

39
     GAO-02-329.
40
     GAO-02-195.
41
     GAO-02-195.




Page 26                                  GAO-03-371 DOD IT Services Outsourcing
                  lessons are of little benefit unless they are distributed and used by people
                  who will benefit from them. Lessons can be “pushed,” or automatically
                  delivered to a user, or “pulled” in situations where a user must manually
                  search for them. Lessons can also be disseminated with an assigned
                  priority descriptor, which denotes the risk, immediacy, and urgency of the
                  lessons learned content.



Conclusions       The projects in our review substantially implemented leading commercial
                  practices for outsourcing IT services, which has increased each project’s
                  probability of success. Capturing how these projects operationalized
                  leading commercial practices could help other IT services outsourcing
                  projects succeed. Although currently there is no such DOD-wide
                  mechanism, such as an electronic tool, to easily share and leverage lessons
                  learned, DOD IT and acquisition officials agreed that a departmentwide
                  effort to identify, capture, and disseminate lessons learned could offer
                  valuable insights and new ideas that would benefit others. Lessons learned
                  that are pragmatic and easily accessible could give DOD managers a more
                  informed understanding of the important issues to be addressed when
                  making outsourcing decisions, as well as the factors to be considered to
                  help ensure the success of these endeavors. DOD managers can also
                  benefit from lessons learned on the basis of negative experiences. The
                  projects in our review were well into implementation, and therefore, at this
                  late stage, we see little advantage for them to revisit practices that were not
                  implemented. Nevertheless, an electronic tool for capturing and
                  disseminating lessons learned would allow the rest of DOD to benefit from
                  the negative consequences and increased risks associated with those
                  practices that the projects did not implement.

                  Developing a lessons learned mechanism is not easy; thus, senior
                  management support and resources are keys to success. Without such
                  support driving the capture and dissemination of lessons learned, DOD is
                  losing an opportunity for wider application of leading practices and thus
                  better ensuring that its IT outsourcing efforts are successful.



Recommendations   To assist DOD organizations in planning and implementing outsourcing
                  projects for IT services, we recommend that the Secretary of Defense
                  direct the Undersecretary of Defense for Acquisition, Technology, and
                  Logistics, working in conjunction with the Assistant Secretary of Defense
                  for Command, Control, Communications, and Intelligence (C3I), to provide



                  Page 27                                    GAO-03-371 DOD IT Services Outsourcing
                      senior management support and adequate resources to develop and
                      implement an electronic tool to capture and disseminate examples and
                      lessons learned from actual IT outsourcing projects. These examples and
                      lessons learned, at a minimum, should include the results of our review of
                      the five projects discussed in this report.

                      We also recommend that the Secretary of Defense direct the
                      Undersecretary of Defense for Acquisition, Technology, and Logistics,
                      working in conjunction with the Assistant Secretary of Defense for
                      Command, Control, Communications, and Intelligence (C3I), to ensure that
                      the method used to gather information for this electronic tool incorporate
                      the main elements of a lessons learned process—namely, collection,
                      verification, storage, and dissemination.



Agency Comments and   In written comments on a draft of our report, signed by DOD’s Principal
                      Director, Deputy Assistant Secretary of Defense (Deputy Chief Information
Our Evaluation        Officer), the department partially concurred with our recommendations.
                      Specifically, DOD agreed that capturing lessons learned in the development
                      and implementation of its IT outsourcing initiatives is important to
                      continually improve its outsourcing methods and results achieved. The
                      department also stated that before deciding on a specific method to
                      achieve this aim, it intends to explore a variety of mechanisms that could
                      be used. In particular, the department stated that it currently has several
                      processes and communities of interest that collect and disseminate lessons
                      learned in other areas, which are logical starting points for determining the
                      best path forward. DOD’s written comments are reproduced in appendix
                      IX.

                      We agree that it is prudent to explore various alternatives to leveraging
                      lessons learned from DOD’s IT services outsourcing experiences. Our
                      recommendations are not prescriptive as to the electronic method to be
                      used to capture and disseminate lessons learned. Therefore, the
                      department’s plan to explore various alternatives is consistent with our
                      recommendations.


                      We are sending copies of this report to the appropriate congressional
                      committees, the Secretary of Defense, and the Director of the Office of
                      Management and Budget. We will also provide copies to others upon




                      Page 28                                   GAO-03-371 DOD IT Services Outsourcing
request. In addition, the report will be available at no charge on the GAO
Web site at http://www.gao.gov.

If you have any questions on matters discussed in this report, please
contact me at (202) 512-3439 or Linda J. Lambert, Assistant Director, at
(202) 512-9556. We can also be reached by E-mail at hiter@gao.gov and
lambertl@gao.gov, respectively. Other contacts and key contributors to this
report are listed in appendix X.




Randolph C. Hite
Director, Information Technology Architecture
 and Systems Issues




Page 29                                  GAO-03-371 DOD IT Services Outsourcing
Appendix I

Objectives, Scope, and Methodology                                                                    AA
                                                                                                       ppp
                                                                                                         ep
                                                                                                          ned
                                                                                                            n
                                                                                                            x
                                                                                                            id
                                                                                                             e
                                                                                                             x
                                                                                                             Iis




              Our objectives were to determine (1) the extent to which selected
              Department of Defense (DOD) information technology (IT) services
              outsourcing projects use leading commercial practices as specified in our
              framework and (2) whether DOD is sharing lessons learned from its IT
              outsourcing projects across the department.

              To determine the extent to which selected DOD outsourcing projects for IT
              services use leading commercial practices, we identified the practices in
              our November 2001 report on leading commercial practices1 that (1) are
              typically applied at the project level and (2) were verifiable through
              documentation and interviews. Because DOD did not centrally maintain a
              list of outsourcing projects for IT services, we asked the department to
              identify candidate projects for our evaluation. From this list, we selected
              the following five projects for our review: (1) Air Force Kirtland Air Force
              Base’s Command, Control, Communications, and Computer (C4) Services
              project; (2) Army Network Enterprise Technology Command’s (NETCOM)
              Total Army Communications – Southwest Asia (TAC-SWA) project; (3)
              Military Health System/Information Technology Organization (MHS/ITO)
              Help Desk project; (4) Department of the Navy’s Navy and Marine Corps
              Intranet (NMCI) project; and (5) National Imagery and Mapping Agency
              (NIMA) Information Technology/Information Services (IT/IS) project. We
              chose each project on the basis of the following criteria: (1) no more than
              one project from each military service and two agencies, (2) illustrative
              example of DOD IT outsourcing, (3) dollar value greater than $10 million,
              and (4) enough time elapsed for services to have been delivered and
              performance measured.

              At our request, each project completed a self-assessment on whether and
              how it implemented leading commercial practices. We reviewed the agency
              self-assessments and accompanying documentation and interviewed the
              appropriate agency project officials to verify whether the practices were
              followed. In addition, we interviewed representatives from each of the
              providers associated with these projects.

              We also researched additional information on commercial practices in our
              November 2001 guide2 and the challenges the federal government faces in


              1
               U.S. General Accounting Office, Information Technology: Leading Commercial Practices
              for Outsourcing of Services, GAO-02-214 (Washington, D.C.: Nov. 30, 2001).
              2
              GAO-02-214.




              Page 30                                        GAO-03-371 DOD IT Services Outsourcing
Appendix I
Objectives, Scope, and Methodology




implementing them. Specifically, we performed a literature search, which
included reviewing reports issued by leading research firms, such as
Gartner, Inc., and Giga Information Group, Inc. In addition, we interviewed
representatives from industry organizations that have an interest in
outsourcing IT services, including the Coalition for Government
Procurement; the Information Technology Association of America; the
Professional Services Council; and Acquisition Solutions, Inc.

To determine whether DOD is sharing lessons learned from its IT
outsourcing projects across the department, we identified and reviewed
various approaches that DOD currently uses to capture and disseminate
such information. This included identifying and reviewing various Web
sites and performing key word searches on these sites to identify lessons
learned for outsourcing IT services. We also interviewed applicable DOD
acquisition and IT officials. Finally, we identified and reviewed a generic
lessons learned process contained in our January 2002 report on the
National Aeronautics and Space Administration’s lessons learned
mechanisms.3

We performed our work at the Army’s NETCOM in Ft. Huachuca, Arizona;
the MHS/ITO Help Desk project office in Falls Church, Virginia; the NMCI
Director’s office in Crystal City, Virginia; and NIMA’s headquarters in
Bethesda, Maryland. We conducted our review between May 2002 and early
March 2003 in accordance with generally accepted government auditing
standards.




3
 U.S. General Accounting Office, NASA: Better Mechanisms Needed for Sharing Lessons
Learned, GAO-02-195 (Washington, D.C.: Jan. 30, 2002).




Page 31                                       GAO-03-371 DOD IT Services Outsourcing
Appendix II

Projects’ Implementation of Phase I:
Determine Sourcing Strategy                                                                           Appendx
                                                                                                            Ii




               In the first phase of our outsourcing framework, the client organization
               determines whether internal capability or external expertise can more
               effectively meet its IT needs. The purpose of this sourcing strategy is to
               achieve the optimal balance between internal and external capabilities,
               activities, processes, and services to ensure that strategic business
               objectives are achieved at the lowest risk. Among the factors that an
               organization should evaluate in crafting this strategy are technology,
               business, financial, and personnel requirements and whether it has skilled
               business and IT managers. In addition, according to Gartner, Inc., sound
               sourcing decisions depend on whether IT organizations (1) know and
               understand their business priorities, (2) are prepared to invest in some
               skills and divest others, and (3) identify and assess trade-offs.1

               The six practices in this phase that we used to evaluate the five projects are
               as follows:

               • Use third-party assistance with experience in a variety of sourcing
                 arrangements when formulating a sourcing strategy.

               • Incorporate lessons learned from peers who have engaged in similar
                 sourcing decisions.

               • Estimate impact of sourcing decision on internal organization.

               • Benchmark and baseline productivity of internal services before making
                 the final sourcing decision.


               1
                Gartner, Inc., Five Tough Questions About Skill Sourcing, Research Note SPA-13-2537
               (Mar. 28, 2001).




               Page 32                                         GAO-03-371 DOD IT Services Outsourcing
Appendix II
Projects’ Implementation of Phase I:
Determine Sourcing Strategy




• Determine the business reasons for outsourcing IT.

• Determine reasons for outsourcing IT that can improve the
  organization’s ability to use and manage technology.

Figure 6 shows that the implementation of the practices by the five projects
in our review was uneven.



Figure 6: Project Implementation of Phase I: Determine Sourcing Strategy




Note: Not applicable—The practice was not relevant to the project’s particular circumstances.
No—The agency did not implement the practice. Limited—The agency fully implemented some but not
all aspects of the practice and did not take alternative actions that fully satisfied the practice. Yes—The
agency fully implemented the practice or took an alternative action that fully satisfied its intent.




Page 33                                                   GAO-03-371 DOD IT Services Outsourcing
                                               Appendix II
                                               Projects’ Implementation of Phase I:
                                               Determine Sourcing Strategy




                                               Table 4 provides detailed information on whether and how each project
                                               implemented each of the six practices in this phase.



Table 4: Summary of Projects’ Use of Phase I Practices

                                                              Did the project use the practice?
                      Air Force C4
Practice              Services                Army TAC-SWA           MHS/ITO Help Desk Navy NMCI                  NIMA IT/IS
Use third-party       No—According to Air     No—According to a      Yes—For example,       Yes—The Navy          Yes—NIMA
assistance with       Force project           TAC-SWA project        MHS worked with the    employed a            employed a
experience in a       officials, they         official, the Army     Department of the      contractor with       contractor with
variety of sourcing   decided that they       decided that it had    Interior’s GovWorks    sourcing expertise    sourcing expertise.
arrangements when     had sufficient in-      sufficient expertise   Program and the        and contacted other
formulating a         house expertise, and    in-house since the     Defense Acquisition    government entities
sourcing strategy.    the project had no      TAC-SWA contract       University staff to    about their
                      funding available to    was a consolidation    obtain expertise on    experiences.
                      employ a contractor.    of three existing      sourcing strategies.
                      However, the former     contracts.             Also, MHS employed
                      functional area chief                          a contractor with
                      told us that using                             sourcing expertise.
                      third-party
                      assistance would
                      have been beneficial
                      because the
                      requirements would
                      have been better
                      written. According to
                      this official, the
                      requirements had to
                      be substantially
                      rewritten 2 years
                      after contract award.




                                               Page 34                                          GAO-03-371 DOD IT Services Outsourcing
                                                  Appendix II
                                                  Projects’ Implementation of Phase I:
                                                  Determine Sourcing Strategy




(Continued From Previous Page)
                                                                  Did the project use the practice?
                        Air Force C4
Practice                Services                 Army TAC-SWA              MHS/ITO Help Desk Navy NMCI                      NIMA IT/IS
Incorporate lessons     Yes—Primarily from       Yes—According to a        Yes—Primarily from        Yes—Navy officials     Yes—A NIMA
learned from peers      other Department of      TAC-SWA project           industry peers and        stated that there      contractor provided
who have engaged in     Defense (DOD)            official, it used         MHS’s prior help-         were no peers that     the agency with a
similar sourcing        projects that            lessons learned from      desk function.            had engaged in         report on industry
decisions.              implemented Office       its prior contracts for                             similar sourcing       best practices. In
                        of Management and        these services and                                  decisions because      addition, NIMA held
                        Budget (OMB)             another similar Army                                no other outsourcing   discussions with the
                        Circular A-76            contract.                                           project was of as      National Security
                        policies.                                                                    large a scale as       Agency on its
                                                                                                     NMCI. However, the     outsourcing effort.
                                                                                                     Navy did talk to
                                                                                                     members of private
                                                                                                     industry and the
                                                                                                     National Aeronautics
                                                                                                     and Space
                                                                                                     Administration on
                                                                                                     their more limited
                                                                                                     efforts.
Estimate impact of     Yes—The Air Force         No—The Army did           Yes—MHS assessed          Yes—The Navy           Yes—NIMA
sourcing decision on estimated that there        not analyze the           the staff and financial   performed an           performed an
internal organization. would be substantial      impact on its internal    impact of its sourcing    assessment of          assessment of
                       internal impact, such     organization              decision.                 staffing and other     staffing and other
                       as to its staff, due to   because, according                                  impacts, such as       impacts, such as
                       its decision to           to the TAC-SWA                                      cost, related to its   cost, related to its
                       outsource.                program manager,                                    sourcing decision.     sourcing decision.
                                                 NETCOM was
                                                 outsourcing a
                                                 function that was
                                                 already contracted
                                                 out.a




                                                  Page 35                                                 GAO-03-371 DOD IT Services Outsourcing
                                                   Appendix II
                                                   Projects’ Implementation of Phase I:
                                                   Determine Sourcing Strategy




(Continued From Previous Page)
                                                                  Did the project use the practice?
                        Air Force C4
Practice                Services                   Army TAC-SWA          MHS/ITO Help Desk Navy NMCI                        NIMA IT/IS
Benchmark and           No—The Air Force           No—The Army did       Limited—MHS              Limited—The Navy          Yes—A NIMA
baseline productivity   did not implement          not perform a         attempted to             had a private-sector      contractor baselined
of internal services    this practice              benchmark or          establish a baseline     firm benchmark its        the existing NIMA
before making the       because, according         baseline analysis     of its prior             environment against       environment and
final sourcing          to project officials, an   because, according    environment but,         seven large public        benchmarked it to
decision.               executive decision         to the TAC-SWA        according to project     and private               peers.
                        was made to                program manager,      officials, available     organizations. In
                        outsource the C4           NETCOM was            documentation was        addition, the Navy
                        services following the     outsourcing a         limited to historical    and the Marine
                        policies contained in      function that was     trouble ticket           Corps performed an
                        OMB Circular A-76.         already contracted    workload data;           analysis at a sample
                        Accordingly, Air           out.                  therefore, this          of representative
                        Force officials stated                           baseline was a best      locations to obtain a
                        that such analyses                               estimate. In addition,   baseline. However,
                        would not have                                   MHS project officials    this baseline did not
                        affected the final                               stated that a            include an
                        sourcing decision.                               contractor performed     assessment of the
                                                                         a benchmark              Department of the
                                                                         analysis, but they did   Navy’s legacy
                                                                         not provide              applications since
                                                                         supporting               project officials
                                                                         documentation.           decided to rely upon
                                                                                                  an inventory
                                                                                                  developed in
                                                                                                  addressing the Year
                                                                                                  2000 challenge. The
                                                                                                  Navy subsequently
                                                                                                  found that it had
                                                                                                  substantially
                                                                                                  underestimated its
                                                                                                  number of legacy
                                                                                                  applications.
                                                                                                  According to NMCI
                                                                                                  program officials, this
                                                                                                  underestimation
                                                                                                  contributed to the
                                                                                                  transition period
                                                                                                  slipping from 2-½
                                                                                                  years to 3-½ years.




                                                   Page 36                                             GAO-03-371 DOD IT Services Outsourcing
                                                    Appendix II
                                                    Projects’ Implementation of Phase I:
                                                    Determine Sourcing Strategy




(Continued From Previous Page)
                                                                     Did the project use the practice?
                         Air Force C4
Practice                 Services                  Army TAC-SWA              MHS/ITO Help Desk Navy NMCI                          NIMA IT/IS
Determine the        Yes—To achieve cost           Yes—To address            Yes—To achieve            Yes—To have private        Yes—To obtain
business reasons for savings and to shift          shortages in military     increased                 industry capitalize        improved customer
outsourcing IT.      military personnel to         personnel to perform      productivity and          infrastructure             services and
                     other work.                   its mission.              customer satisfaction     improvements that          decreased costs.
                                                                             and to decrease           were needed to             Also, this function
                                                                             costs.                    quickly and securely       was determined to
                                                                                                       share knowledge            be a commercial
                                                                                                       around the globe.          function under the
                                                                                                                                  Federal Activities
                                                                                                                                  Inventory Reform
                                                                                                                                  Act.b
Determine reasons        No—According to           No—According to          Yes—To improve IT          Yes—To improve, for        Yes—To achieve
for outsourcing IT       Kirtland Air Force        TAC-SWA project          expertise and              example, system            better IT
that can improve the     Base officials, the Air   officials, improving its knowledge.                 security,                  management
organization’s ability   Force decided to          ability to use and                                  interoperability,          performance.
to use and manage        hold a public/private     manage technology                                   reliability, and
technology.              competition following     was not a factor in                                 network response.
                         OMB Circular A-76         determining its
                         policies; therefore,      outsourcing strategy.
                         improving its ability
                         to use and manage
                         technology was not a
                         factor in determining
                         its outsourcing
                         strategy.
Source: GAO.
                                                    a
                                                     Having had the activity previously performed by a contractor does not obviate the need to estimate the
                                                    impact of a sourcing decision on the internal organization—there are still risks involved, such as the
                                                    potential disruption of services during the transfer to the new contractor. In fact, the staff from the
                                                    incumbent contractor did not transition to the TAC-SWA provider, and the provider had difficulty filling
                                                    these slots within the schedule outlined in the contract.
                                                    b
                                                     The Federal Activities Inventory Reform Act of 1998 requires federal agencies to prepare and submit
                                                    to OMB, by June 30 of each year, inventories of the commercial activities performed by federal
                                                    employees.




                                                    Page 37                                                  GAO-03-371 DOD IT Services Outsourcing
Appendix III

Project’s Implementation of Phase II: Define
Operational Model                                                                              Appendx
                                                                                                     iI




               Critical to the successful outsourcing relationship is an operational model
               for guiding the structure of the contract and the plans for transition. In
               defining the operational model, client organizations formalize executive
               leadership, team composition, client responsibilities, and operating
               relationships between the client and provider. The operational model helps
               the organization to compare its plans with the expectations that were set as
               the initial decision to outsource was made and to ascertain whether these
               plans will enable the organization to meet those objectives. An important
               aspect of the operational model is an explicit understanding of how the
               client organization plans to communicate its needs and provide feedback to
               the provider. In addition, communication between the business and IT
               offices within the client organization is always critical. This is particularly
               true in the case of outsourcing because the IT service provider is outside
               the client organization and disconnects are more likely to occur. Therefore,
               organizational processes to facilitate good communication are critical.

               The 13 practices in this phase that we used to evaluate the five projects are
               as follows:

               • Establish executive leadership for IT to facilitate the outsourcing
                 initiative.

               • Continually communicate/clarify outsourcing objectives, while
                 correcting misinformation that affects the organization.

               • Establish a core group of people who will be involved in all phases of
                 outsourcing.

               • Select a person involved in the negotiation of the contract to manage the
                 outsourcing relationship.



               Page 38                                    GAO-03-371 DOD IT Services Outsourcing
Appendix III
Project’s Implementation of Phase II: Define
Operational Model




• Create and define a contract management structure with operational
  points of contact and managers.

• Define the role of internal IT managers and business leaders.

• Ensure that the right skills are in place to support the outsourcing
  relationship.

• Establish a point of contact high in the provider management structure
  for elevating provider performance concerns.

• Have provider establish an on-site support team to serve as liaison
  between client and provider.

• Train provider on client business environment and goals.

• Select or develop standard tools for managing the relationship.

• Use third-party assistance to take advantage of expertise from a variety
  of outsourcing arrangements in defining the operational model.

• Ensure that the provider management team has prior experience in the
  client’s field of business.

Figure 7 illustrates that the five projects in our review largely implemented
the practices.




Page 39                                        GAO-03-371 DOD IT Services Outsourcing
Appendix III
Project’s Implementation of Phase II: Define
Operational Model




Figure 7: Project Implementation of Phase II: Define Operational Model




Note: Not applicable—The practice was not relevant to the project’s particular circumstances.
No—The agency did not implement the practice. Limited—The agency fully implemented some but not
all aspects of the practice and did not take alternative actions that fully satisfied the practice. Yes—The
agency fully implemented the practice or took an alternative action that fully satisfied its intent.


Table 5 provides detailed information on whether and how each project
implemented each of the 13 practices in this phase.




Page 40                                                   GAO-03-371 DOD IT Services Outsourcing
                                                     Appendix III
                                                     Project’s Implementation of Phase II: Define
                                                     Operational Model




Table 5: Summary of Projects’ Use of Phase II Practices

                                                                    Did the project use the practice?
                          Air Force C4
Practice                  Services                  Army TAC-SWA            MHS/ITO Help Desk Navy NMCI                       NIMA IT/IS
Establish executive       Yes—The initiative        Yes—NETCOM used         Yes—MHS used its        Yes—In 1999, the          Yes—The NIMA
leadership for IT to      was planned by a          the existing            existing leadership     Navy established a        enterprise
facilitate the            steering group made       leadership in its       structure and           program executive         transformation
outsourcing initiative.   up of representatives     offices of operations   processes. This         office for IT primarily   directorate formed a
                          from various major        and logistics to        structure includes a    to support the NMCI       strategic sourcing
                          offices at Kirtland Air   provide executive       program executive       outsourcing effort.       office to provide
                          Force Base.               leadership.             office, steering        The Congress later        executive oversight
                          Subsequent to                                     committee, and          directed the Navy to      of the effort.
                          contract award, the                               program review          identify a single
                          Air Force established                             board.                  individual whose sole
                          a functional area                                                         responsibility would
                          chief to manage the                                                       be to oversee and
                          initiative, and a                                                         direct the NMCI
                          Lieutenant Colonel                                                        program. As a result,
                          was appointed to be                                                       in February 2002, the
                          responsible for this                                                      Navy established the
                          project.                                                                  NMCI Director’s
                                                                                                    Office to take over
                                                                                                    responsibility for
                                                                                                    NMCI.




                                                     Page 41                                             GAO-03-371 DOD IT Services Outsourcing
                                                    Appendix III
                                                    Project’s Implementation of Phase II: Define
                                                    Operational Model




(Continued From Previous Page)
                                                                    Did the project use the practice?
                          Air Force C4
Practice                  Services                 Army TAC-SWA             MHS/ITO Help Desk Navy NMCI                         NIMA IT/IS
Continually               Yes—The Air Force        Yes—According to         Yes—MHS provided          Yes—The Navy              Yes—NIMA created
communicate/clarify       provided                 TAC-SWA project          updates to the Chief      established an action     an internal Web site
outsourcing               communication            officials, the           Information Officer       collaboration team        to post information
objectives, while         through, for example,    applicable NETCOM        (CIO) and program         structure to involve      on the outsourcing
correcting                briefings and            officials were           review boards,            the Navy community        project, distributed
misinformation that       meetings with            informed about the       provided briefings to     in the NMCI               periodic global
affects organization.     employees.               initiative and           the deputy surgeon        communications            electronic mails, and
                                                   provided comments        general, and              process. Also, the        held town hall
                                                   on the draft             published questions       NMCI Information          meetings.
                                                   performance work         and answers.              Bureau initiated
                                                   statement. TAC-SWA                                 press conferences,
                                                   project officials also                             briefings, site visits,
                                                   brief new                                          informational
                                                   commanders on the                                  pamphlets, and
                                                   contract before they                               promotional material.
                                                   are transferred to the                             In addition, the NMCI
                                                   military theater.                                  Director and other
                                                                                                      staff provide
                                                                                                      numerous briefings
                                                                                                      and presentations to
                                                                                                      commands, industry
                                                                                                      gatherings, and
                                                                                                      government officials.
                                                                                                      Finally, Web sites
                                                                                                      and Web-based
                                                                                                      collaboration sites
                                                                                                      were established to
                                                                                                      facilitate
                                                                                                      communications.
Establish a core          Yes—The core group       Yes—According to         Yes—According to          Yes—The program           Yes—NIMA
group of people who       included the             TAC-SWA project          program officials, the    executive office for IT   established the
will be involved in all   contracting officer,     officials, the core      core group included       and the NMCI              Strategic Sourcing
phases of                 functional area chief,   group included           representatives from      Director’s                Office, which was
outsourcing.              and manpower             representatives from     the MHS program           organization              dedicated to
                          specialist.              logistics, operations,   executive office,         comprise the core         managing and
                                                   and resource             MHS IT program            group responsible for     facilitating the
                                                   management as well       offices, military         managing the              outsourcing initiative.
                                                   as the contracting       department chief          outsourcing initiative.
                                                   officer.                 information officers’
                                                                            offices, and Interior’s
                                                                            GovWorks
                                                                            organization, which
                                                                            provided contracting
                                                                            services.




                                                    Page 42                                                GAO-03-371 DOD IT Services Outsourcing
                                                   Appendix III
                                                   Project’s Implementation of Phase II: Define
                                                   Operational Model




(Continued From Previous Page)
                                                                   Did the project use the practice?
                        Air Force C4
Practice                Services                  Army TAC-SWA             MHS/ITO Help Desk Navy NMCI                      NIMA IT/IS
Select a person         Yes—The contracting Yes—The contracting Yes—According to        Yes—The primary                     Yes—A lead
involved in the         officer.a           officer.            MHS program             contracting officer.                contracting officer.
negotiation of the                                              officials, the chair of
contract to manage                                              the contract
the outsourcing                                                 evaluation committee
relationship.                                                   and the individual in
                                                                charge of transition
                                                                are responsible for
                                                                managing the
                                                                outsourcing
                                                                relationship.
Create and define       Yes—The structure         Yes—The overall          Yes—Responsibility      Yes—The primary          Yes—NIMA has
contract                was defined by the        responsibility for the   for contract            contracting officer      appointed an
management              roles and                 contract rests with      management is           establishes              operational point of
structure with          responsibilities of the   the contracting          jointly held by the     procedures and           contact for each of
operational points of   contracting officer       officer. The             contracting officer,    controls necessary       the seven functional
contact and             and her staff.            contracting officer’s    who is part of          for effective            areas being
managers.                                         representative acts      Interior’s GovWorks     contractual oversight    outsourced, which
                                                  as a liaison between     organization, and the   of the NMCI initiative   are documented in a
                                                  the government and       MHS program office,     and has a matrix         “rules of
                                                  the contractor.          which provides day-     relationship with the    engagement”
                                                                           to-day oversight of     NMCI Director. The       agreement between
                                                                           the provider.           primary contracting      the government and
                                                                                                   officer certifies        the contractor.
                                                                                                   contracting officer
                                                                                                   representatives for
                                                                                                   the NMCI contract to
                                                                                                   provide technical
                                                                                                   coordination efforts.




                                                   Page 43                                              GAO-03-371 DOD IT Services Outsourcing
                                                   Appendix III
                                                   Project’s Implementation of Phase II: Define
                                                   Operational Model




(Continued From Previous Page)
                                                                  Did the project use the practice?
                       Air Force C4
Practice               Services                   Army TAC-SWA             MHS/ITO Help Desk Navy NMCI                      NIMA IT/IS
Define the role of     Yes—Kirtland Air           Limited—TAC-SWA          Yes—MHS’s help-        Yes—The NMCI              Yes—NIMA
internal IT managers   Force Base did not         project officials        desk performance       execution plan            transformation teams
and business           initially define the       explained the roles of   assessment plan        describes the roles of    for the activities
leaders.               roles of its internal IT   the principal staff      defines the roles of   internal and external     being outsourced
                       and business               involved with this       various groups and     organizations that        define the roles of
                       managers. However,         project but generally    individuals            directly or indirectly    their managers and
                       subsequent to              did not have             associated with the    affect the                leaders.
                       contract award,            supporting               program.               management of
                       Kirtland Air Force         documentation                                   NMCI and explains in
                       Base established a         defining these roles.                           detail the duties and
                       CIO committee and                                                          responsibilities of the
                       configuration control                                                      program executive
                       board comprising the                                                       office for IT and the
                       functional area chief                                                      NMCI program
                       (Kirtland’s CIO) and                                                       management offices.
                       representatives from
                       various business
                       areas. Although not
                       exclusively devoted
                       to the C4 Services
                       project, these groups
                       help provide direction
                       to the project.




                                                   Page 44                                             GAO-03-371 DOD IT Services Outsourcing
                                                    Appendix III
                                                    Project’s Implementation of Phase II: Define
                                                    Operational Model




(Continued From Previous Page)
                                                                   Did the project use the practice?
                         Air Force C4
Practice                 Services                  Army TAC-SWA            MHS/ITO Help Desk Navy NMCI                       NIMA IT/IS
Ensure that the right    Yes—According to          Yes—According to        Yes—According to         Limited—The              Yes—According to
skills are in place to   Air Force officials,      the TAC-SWA project     MHS program              Director, NMCI, has      NIMA, its
support the              the right skills are in   officials, the Army     officials, the Tri-      staff that report        transformation teams
outsourcing              place.                    has created several     Service Management       directly or are          provide the skills
relationship,                                      positions to ensure     Program Office,          matrixed with him        necessary to support
including those                                    that the right skills   GovWorks (the            that are responsible     the outsourcing
dealing with                                       are in place to         Interior organization    for performing all but   relationship.
                                                   manage the              that provides            two of these
• contract                                         outsourcing             contracting              functions.
  management,                                      relationship,           assistance to MHS),      Specifically, at this
• financial                                        including a             and a contractor         time, NMCI does not
  management,                                      contracting officer,    collectively employ      have staff assigned
• IT management,                                   legal advisor,          the skill sets needed    to support teaming
• negotiation                                      operations and          to support the           and interpersonal
  strategies,                                      logistics personnel     outsourcing              relationships and
• teaming and                                      as well as a            relationship.            relationship
  interpersonal                                    NETCOM unit                                      management. The
  relationships,                                   commander.                                       Navy recognizes the
• project                                          According to Army                                need for these skills
  management, and                                  TAC-SWA officials,                               and is taking, or
• relationship                                     each position                                    plans to take, various
  management.                                      employs an                                       actions to obtain
                                                   individual with the                              these skills.
                                                   necessary skills to
                                                   support the
                                                   outsourcing
                                                   relationship.
Establish a point of     Yes—According to          Yes—The provider        Yes—The provider         Yes—The NMCI             Yes—The
contact high in the      the contracting           established a           established an           Director interacts       transformation team
provider                 officer, concerns can     program manager as      executive program        directly with the        charters define
management               be addressed to the       the point of contact    manager as the point     provider’s program       procedures for
structure for            senior vice president     for elevating           of contact for           executive on a           addressing issues,
elevating provider       at the provider           concerns.               elevating concerns.      regularly scheduled      including possibly
performance              headquarters.                                                              basis; such meetings     elevating concerns to
concerns.                                                                                           would include any        the provider’s
                                                                                                    performance issues.      general manager.
Have provider            Yes—The provider          Yes—Site managers       Yes—The provider’s       Yes—The provider         Yes—The rules of
establish an on-site     has established an        have been               transition plan          establishes site leads   engagement define
support team to          on-site support team,     established in          identifies the on-site   that remain on-site      the on-site support
serve as a liaison       which is led by the       accordance with the     support team.            as the location goes     team for each major
between client and       on-site manager.          contract.                                        through the cutover      NIMA location.
provider.                                                                                           to NMCI.




                                                    Page 45                                              GAO-03-371 DOD IT Services Outsourcing
                                                Appendix III
                                                Project’s Implementation of Phase II: Define
                                                Operational Model




(Continued From Previous Page)
                                                               Did the project use the practice?
                      Air Force C4
Practice              Services                 Army TAC-SWA             MHS/ITO Help Desk Navy NMCI                        NIMA IT/IS
Train provider on     Limited—Prior to         Not applicable—The       Yes—MHS officials         Limited—The Navy         Not applicable—
client business       contract award,          Army contract was a      stated that they          did not provide          According to the
environment and       Kirtland Air Force       consolidation of prior   provided some             training on its          project’s contracting
goals.                Base provided an         contracts for this       training to the           business                 officer, such training
                      orientation session in   activity, and the        provider on their         environment and          was not necessary
                      which they discussed     winning bidder was       business                  goals; instead it        because the senior
                      their operations,        one of the incumbent     environment. In           relied on the NMCI       officers of the
                      business                 contractors.             particular, MHS           design reference         provider are all
                      environment, and                                  trained and certified     mission document,        recent NIMA
                      goals. Subsequent to                              the provider’s staff on   which was included       management
                      contract award,                                   the agency’s              with the request for     employees and were
                      specific Kirtland Air                             application systems.      proposals (RFP).         already
                      Force Base                                        In addition, the          This document            knowledgeable of
                      government                                        provider had              defined the NMCI         these areas. For
                      personnel were                                    previous MHS              operational              example, among
                      responsible for aiding                            experience.               environment.             those who
                      the provider during                                                         However, a provider      transitioned to the
                      the transition period.                                                      official stated that     provider was a
                      However, provider                                                           although the             former director of the
                      officials stated that                                                       document was             hydrographic
                      they did not receive                                                        useful, it did not       topographic center of
                      adequate training,                                                          identify Navy            the Defense Mapping
                      which made the                                                              enterprisewide           Agency, a
                      transition period                                                           operations.              predecessor agency
                      more difficult.                                                             According to NMCI        to NIMA.
                                                                                                  program officials, the
                                                                                                  provider somewhat
                                                                                                  mitigated this
                                                                                                  problem by hiring
                                                                                                  several highly
                                                                                                  knowledgeable staff
                                                                                                  from the Department
                                                                                                  of the Navy shortly
                                                                                                  after contract award.
Select or develop     Yes—For example,         Yes—NETCOM uses          Yes—According to          Yes—The Navy is          Yes—NIMA used
standard tools for    the revised contract     a monthly report         MHS officials, the        using a balanced         various tools,
managing the          outlines the use of      from the contracting     program office and        scorecard process to     including twice-a-day
relationship (e.g.,   specific software to     officer’s                the provider have         provide Navy and         performance reports
performance           help manage the          representative to        selected, for             Marine Corps             for the operational
scorecards,           provider’s               evaluate the provider.   example, an               leadership with          help-desk function
enterprise resource   performance.                                      enterprise resource       information to judge     and monthly
management                                                              management                how well NMCI is         performance reports.
system).                                                                system.                   supporting the
                                                                                                  missions and
                                                                                                  strategies of the
                                                                                                  department.




                                                Page 46                                                GAO-03-371 DOD IT Services Outsourcing
                                                     Appendix III
                                                     Project’s Implementation of Phase II: Define
                                                     Operational Model




(Continued From Previous Page)
                                                                      Did the project use the practice?
                           Air Force C4
Practice                   Services                 Army TAC-SWA              MHS/ITO Help Desk Navy NMCI                          NIMA IT/IS
Use third-party            Yes—Although third-      No—The Army               Yes—For example,          No—The Navy                Yes—NIMA
assistance to take         party assistance was     believed that it had      MHS contacted the         believed it had            employed
advantage of               not used at the onset    sufficient expertise      Interior’s GovWorks       sufficient internal        contractors to
expertise from a           of the C4 Services       in-house.                 Program, Defense          experience and             develop contractual
variety of outsourcing     project, a person with                             Acquisition               expertise.                 roles and assist in
arrangements in            prior experience in                                University, and                                      evaluating the
defining the               managing IT                                        others. Also, MHS                                    contract.
operational model          services contracts for                             employed a
(i.e., defining roles      the Air Force was                                  contractor with
and responsibilities).     brought in to provide                              sourcing expertise.
                           advice. He was
                           subsequently put in
                           charge of the project.
Ensure that the            Yes—Provider             Yes—Vendor past           Yes—The request for       Yes—According to           Yes—It was expected
provider                   management               performance was           quote laid out MHS’s      NMCI program               that most of the
management team            capabilities and         part of the evaluation    evaluation criteria for   officials, the Navy        provider’s workforce
has prior experience       experience in            criteria during source    selecting a provider,     required bidders to        would be composed
in the client’s field of   providing IT services    selection and the         which included prior      provide information        of former NIMA
business.                  were evaluation          provider was an           experience in health      on their prior             employees. Among
                           factors in the           incumbent contractor      care.                     experience in related      those who
                           technical evaluation     with NETCOM.                                        business fields. For       transitioned to the
                           process.                                                                     example, the               provider was a
                                                                                                        solicitation required      former director of the
                                                                                                        bidders to                 hydrographic
                                                                                                        demonstrate                topographic center of
                                                                                                        experience                 the Defense Mapping
                                                                                                        managing a similar         Agency.
                                                                                                        effort of 100,000 or
                                                                                                        more seats at one
                                                                                                        time.b
Source: GAO.
                                                     a
                                                      The Air Force’s initial contract was not negotiated because Kirtland Air Force Base used an OMB
                                                     Circular A-76 2-step, sealed bid process. However, the contracting officer was involved in the bid
                                                     process as well as in managing the winning contractor.
                                                     b
                                                      This threshold could be met through multiple efforts, but at least one had to include at least 20,000
                                                     seats.




                                                     Page 47                                                  GAO-03-371 DOD IT Services Outsourcing
Appendix IV

Project’s Implementation of Phase III: Develop
the Contract                                                                                  Appendx
                                                                                                    iIV




               Phase III focuses on the development of the contract, which defines the
               legal terms of the relationship between client and provider. While other
               phases in the outsourcing process describe the need for mutual trust and a
               close, flexible working relationship, this phase focuses on the development
               of the contract, which is the foundation on which a working relationship
               will be built. A well-written contract helps the outsourcing organization
               meet its requirements while allowing the service provider to make a fair
               profit. It sets the expectations for service levels, delivery of essential
               services, and continuous improvement and is intended to protect the
               interests of all parties.

               The 16 practices in this phase that we used to evaluate the five projects are
               as follows:

               • Base performance requirements on business outcomes.

               • Include measures that reflect end-user satisfaction as well as technical
                 IT performance.

               • Review and update performance requirements periodically.

               • Require the provider to meet the minimum performance in each
                 category of service.

               • Require the provider to achieve escalating performance standards at
                 agreed-upon intervals.

               • Incorporate sufficient flexibility so that minimum acceptable
                 performance can be adjusted as conditions change, as the provider




               Page 48                                   GAO-03-371 DOD IT Services Outsourcing
Appendix IV
Project’s Implementation of Phase III:
Develop the Contract




   becomes more adept at satisfying customer demands, and as
   improvement goals are achieved.

• Use service-level agreements (SLA)1 to clearly articulate all aspects of
  performance, including management, processes, and requirements.

• Client and provider work together to define the appropriate number of
  SLAs and appropriate structure for each.

• Specify circumstances under which the provider is excused from
  performance levels mandated by master service agreements.

• Client and provider work together to identify SLAs for which
  compensation is based, while additional SLAs may be defined to manage
  performance.

• The contract should include clauses for (1) determining pricing
  structures; (2) performing customer satisfaction surveys and using the
  results to redefine performance levels; (3) terminating the contract; (4)
  resolving disputes in a timely manner; (5) taking work away, without
  penalty, from provider for nonperformance; (6) declaring a significant
  event that can lead to a change in the contract; (7) defining performance
  requirements; and (8) conducting regularly scheduled meetings.

• Consider setting up a master services agreement under which all
  arrangements between client and provider operate.

• Include the appropriate representation from each major organizational
  unit on the contract negotiation team.

• Specify the use of volume purchases to obtain optimal discounts.

• Use third-party assistance in negotiating and developing the contract.

• Sign the contract after contract negotiations and final vendor selections.

As figure 8 shows, the five projects largely implemented the practices.



1
 SLAs define the agency’s expectations and are used to track and measure a contractor’s
performance.




Page 49                                          GAO-03-371 DOD IT Services Outsourcing
Appendix IV
Project’s Implementation of Phase III:
Develop the Contract




Figure 8: Project Implementation of Phase III: Develop the Contract




Note: Not applicable—The practice was not relevant to the project’s particular circumstances.
No—The agency did not implement the practice. Limited—The agency fully implemented some but not
all aspects of the practice and did not take alternative actions that fully satisfied the practice. Yes—The
agency fully implemented the practice or took an alternative action that fully satisfied its intent.


Table 6 provides detailed information on whether and how each project
implemented each of the 16 practices in this phase.




Page 50                                                   GAO-03-371 DOD IT Services Outsourcing
                                                  Appendix IV
                                                  Project’s Implementation of Phase III:
                                                  Develop the Contract




Table 6: Summary of Projects’ Use of Phase III Practices

                                                                  Did the project use the practice?
                        Air Force C4
Practice                Services                 Army TAC-SWA              MHS/ITO Help Desk Navy NMCI                        NIMA IT/IS
Base performance        Yes—The                  Yes—The                   Yes—Industry             Yes—The                   Yes—Performance-
requirements on         performance              performance               standards and            performance               based requirements
business outcomes.      requirements are         requirements are          metrics were used to     requirements are          are in the contract,
                        contained in the         outcome based and         base performance         included in the           including those
                        contract and are         stated in terms of the    requirements on          contract and are          related to customer
                        based on business        level of operation and    business outcomes.       based on business         satisfaction, process
                        outcomes. For            maintenance                                        outcomes. SLAs            improvement, quality
                        example, the             services required.                                 establish the             control, and timely
                        requirements define      According to TAC-                                  performance               and accurate
                        the percentage of        SWA officials,                                     standards and             completion of
                        system availability      multiple trips to                                  service quality for all   requirements.
                        needed to support        overseas locations                                 types of NMCI seats.
                        users.                   were made to identify
                                                 the performance
                                                 requirements.
Include measures        Limited—The Air          Limited—The Army’s        Yes—MHS measures         Yes—The SLAs              Yes—The SLAs
that reflect end-user   Force measures           contract includes         the service provider’s   include measures for      contain technical IT
satisfaction as well    include technical IT     technical IT              technical IT             technical IT              performance
as technical IT         performance.             performance               performance and          performance and           measures. In
performance.            However, end-user        measures but not          uses an electronic       customer                  addition, after a
                        satisfaction             end-user satisfaction     customer satisfaction    satisfaction. In          function is
                        measures are limited     performance               survey to assess the     addition, the contract    transitioned, the
                        to the percentage of     measures. For             quality of help-desk     requires the service      contractor must
                        complaints received      example, the contract     services. Satisfaction   provider to measure       provide a plan that
                        and do not measure       includes a                rates and number of      and report on overall     includes measures
                        overall customer         requirement for help-     survey responses         customer satisfaction     for customer
                        satisfaction. Project    desk services, but        are tracked and used     with NMCI services.       satisfaction, which
                        officials did not know   does not include          for both evaluation                                the provider is
                        why overall end-user     measures related to       and incentive                                      contractually
                        satisfaction             customer satisfaction     payments (positive                                 obligated to meet.
                        measures were not        associated with such      and negative).
                        established.             services. According
                                                 to the project official
                                                 that developed the
                                                 performance work
                                                 statement in the
                                                 contract, NETCOM
                                                 did not include such
                                                 customer satisfaction
                                                 measures because it
                                                 did not think that it
                                                 was necessary to
                                                 have a performance
                                                 standard for that
                                                 service.




                                                  Page 51                                                GAO-03-371 DOD IT Services Outsourcing
                                                 Appendix IV
                                                 Project’s Implementation of Phase III:
                                                 Develop the Contract




(Continued From Previous Page)
                                                                 Did the project use the practice?
                        Air Force C4
Practice                Services                Army TAC-SWA              MHS/ITO Help Desk Navy NMCI                      NIMA IT/IS
Review and update       Yes—The                 Yes—The contract is       Yes—According to         Yes—Performance         Yes—According to
performance             performance             reviewed and              MHS program              reviews and             NIMA, requirements
requirements            requirements were       modified as needed        officials, performance   adjustments are         are updated as
periodically.           revised about 2 years   on the basis of           requirements are         ongoing. For            needed or once a
                        after contract award.   additions or changes      updated as required      example, the Navy is    year when the
                                                to such                   and reviewed on an       in the process of       government is
                                                requirements.             annual basis before      refining its SLAs to    assessing whether to
                                                According to TAC-         the decision is made     ensure that they        exercise its annual
                                                SWA and provider          to execute the option    adequately reflect      option.
                                                officials, changes are    year on the contract.    technical
                                                coordinated between                                performance and
                                                the government and                                 customer satisfaction
                                                the contractor before                              needs.
                                                they are finalized.
Require the provider    Yes—The minimum         Yes—The contract          Yes—The minimal          Yes—The SLAs            Yes—The provider
to meet the minimum     performance is          defines minimum           acceptable               provide the basic       must meet the
performance in each     defined in the          performance               performance criteria     level of service the    performance
category of service.    contract.               requirements.             the contractor must      contractor must         standards as
                                                                          meet are based upon      deliver for every       specified in the
                                                                          commercial industry      NMCI seat.              contract, SLAs, and
                                                                          standards and are                                NIMA’s performance
                                                                          defined in the                                   assessment plan.
                                                                          contract.
Require the provider    No—According to         Not applicable—           Yes—The contract         No—According to         Yes—The contract
to achieve escalating   Kirtland Air Force      According to a TAC-       defines negative,        NMCI program            allows for the
performance             Base officials, they    SWA project official,     acceptable, and          officials, the Navy     redefinition of service
standards at agreed-    did not include         this practice was not     positive incentive       expected vendors to     levels. In addition,
upon intervals.         escalating              applicable because        ranges associated        develop their pricing   according to NIMA
                        performance             NETCOM believed           with escalating          proposal assuming 8     program officials, the
                        standards because       that the performance      performance              yearsb of providing     need for escalating
                        the focus of this       standards in the          standards.               service in              performance
                        contract was on         contract were                                      accordance with the     standards will be
                        meeting                 already high.                                      SLAs included in the    evaluated on an as-
                        requirements at the     Provider officials also                            RFP.                    needed basis as well
                        least cost.a            stated that the                                                            as during the
                                                performance                                                                semiannual award
                                                requirements in the                                                        fee analysis period
                                                contract were high.                                                        and during the
                                                The Army expected,                                                         annual review on
                                                and the contractor                                                         whether to exercise
                                                agreed, to meet                                                            the contract option.
                                                these standards                                                            However, such
                                                immediately.                                                               escalating standards
                                                                                                                           have not yet been
                                                                                                                           identified.




                                                 Page 52                                                GAO-03-371 DOD IT Services Outsourcing
                                                  Appendix IV
                                                  Project’s Implementation of Phase III:
                                                  Develop the Contract




(Continued From Previous Page)
                                                                 Did the project use the practice?
                         Air Force C4
Practice                 Services                Army TAC-SWA            MHS/ITO Help Desk Navy NMCI                          NIMA IT/IS
Incorporate sufficient   No—The contracting      Not applicable—         Yes—According to            Yes—The contract         Yes—According to
flexibility so that      officer said that the   According to a TAC-     MHS program                 includes a provision     NIMA, the
minimum acceptable       requirements should     SWA project official,   officials, the contract     requiring the provider   performance
performance can be       be built into the       NETCOM believed         incorporates                to submit an annual      measures are
adjusted as              contract, which can     that the performance    sufficient flexibility to   plan for technology      designed to first
conditions change,       be modified if          standards in the        adjust minimal              refreshment and          stabilize NIMA’s IT
as the provider          necessary.              contract were           acceptable                  deployment. This         environment and
becomes more adept                               already high, and       performance as              plan is to include       then to be adjusted
at satisfying                                    provider officials      conditions change.          proposed revisions to    to bring minimum
customer demands,                                agreed. According to    For example, the            the contract and an      performance into
and as improvement                               the contracting         contract includes           estimate of changes      alignment with
goals are achieved.                              officer, the military   incentives for the          in performance that      industry best
                                                 theater where           service provider to         would result.            practices. Also, the
                                                 performance is          introduce new                                        contract indicates
                                                 delivered is not an     capabilities and new                                 that the SLAs may be
                                                 area where anything     services within the                                  further defined
                                                 less than meeting the   scope of the                                         throughout the
                                                 defined standards is    performance work                                     contract life.
                                                 acceptable.             statement without
                                                                         further competition.
Use SLAs to clearly      Yes—The SLA             Yes—The TAC-SWA         Yes—The contract            Yes—According to         Yes—The contract
articulate all aspects   defines the             project did not use     defines the                 NMCI officials, SLAs     incorporates SLAs,
of performance,          requirements,           SLAs, but the           requirements,               cover all aspects of     which articulate
including                processes, and who      contract addresses      service performance         provider                 performance
management,              is responsible for      the management,         standards,                  performance. For         requirements and
processes, and           meeting the             processes, and          thresholds,                 example, the E-mail      take effect when the
requirements.            requirements.           requirements            objectives, and             services SLA             function is
                                                 associated with the     metrics as well as the      includes location and    transitioned to the
                                                 project.                requirements for            frequency of the         provider.
                                                                         quarterly                   service, performance
                                                                         management reviews          categories, the
                                                                         and quality                 performance
                                                                         assurance plans.            measures, and
                                                                                                     methods of
                                                                                                     measurement.




                                                  Page 53                                                 GAO-03-371 DOD IT Services Outsourcing
                                                  Appendix IV
                                                  Project’s Implementation of Phase III:
                                                  Develop the Contract




(Continued From Previous Page)
                                                                 Did the project use the practice?
                        Air Force C4
Practice                Services                 Army TAC-SWA            MHS/ITO Help Desk Navy NMCI                       NIMA IT/IS
Client and provider     Yes—Although the         Yes—SLAs are not        Yes—MHS did not          Yes—The SLAs were        Yes—According to
work together to        original SLA was         used but, according     work with the service    developed by the         NIMA and service
define the              prepared by the Air      to TAC-SWA and          provider to determine    Navy with the help of    provider officials,
appropriate number      Force, the revised       provider officials,     the number and           a third-party            they used a
of SLAs and             SLA included             they work together      structure of these       contractor. According    partnering
appropriate structure   contractor input and     on any contract         requirements.            to the NMCI Deputy       contracting approach
for each.               concurrence.             modifications dealing   However, according       Director for             to jointly define,
                                                 with requirements       to MHS and provider      Enterprise               develop, and
                                                 and performance         officials, they have     Operations, it would     structure the SLAs.
                                                 levels. In addition,    worked together on       have been
                                                 the partnering clause   revisions to these       inappropriate to work
                                                 in the contract         requirements.            with individual
                                                 emphasizes a mutual                              competing
                                                 commitment                                       contractors before
                                                 between government                               the contract was
                                                 and industry to work                             awarded. However,
                                                 as a team.                                       NMCI staff and the
                                                                                                  service provider are
                                                                                                  now working together
                                                                                                  on SLA revisions.
Specify                 Yes—The contract         Yes—The contract        Yes—For example,         Yes—Contract terms       Yes—The contractor
circumstances under     includes a clause        includes clauses that   the provider’s           and conditions           can be excused from
which the provider is   that the contractor is   the provider is         proposal, which is       specify the situations   meeting performance
excused from            excused in the event     excused from            incorporated as part     where the service        levels with the
performance levels      of government delay      performance levels if   of the contract,         provider does not        permission of the
mandated by master      of work.                 there are               states that during       have to meet the         contracting officer if
service agreements.                              government delays       transition periods,      SLAs. For example,       circumstances occur
                                                 or factors beyond its   metrics will not be      during transition, the   beyond the service
                                                 control.                reported for incentive   service provider         provider’s control.
                                                                         and penalty              does not have to
                                                                         purposes. As a           meet the
                                                                         result, according to     performance levels
                                                                         program officials, the   set by the SLAs.
                                                                         provider would be
                                                                         excused from
                                                                         required
                                                                         performance levels
                                                                         when baseline
                                                                         requirements are
                                                                         being established for
                                                                         new applications.




                                                  Page 54                                              GAO-03-371 DOD IT Services Outsourcing
                                             Appendix IV
                                             Project’s Implementation of Phase III:
                                             Develop the Contract




(Continued From Previous Page)
                                                            Did the project use the practice?
                      Air Force C4
Practice              Services               Army TAC-SWA          MHS/ITO Help Desk Navy NMCI                         NIMA IT/IS
Client and provider   Yes—According to       Not applicable—       Yes—Before the            Yes—Meetings were         Yes—NIMA and the
work together to      Kirtland Air Force     SLAs are not used.    contract was              held with potential       service provider work
identify SLAs for     Base officials, the                          awarded, MHS did          bidders to obtain         together in
which compensation    service provider had                         not work with the         their input in defining   transformation teams
is based, while       input on the revised                         contractor to identify    the SLAs, including       to develop SLAs,
additional SLAs may   SLAs, including                              SLAs for which            discussions of            including those
be defined to         those for                                    compensation is           compensation. In          affecting
manage                compensation.                                based. However,           addition, Navy and        compensation.
performance.                                                       according to MHS          the provider are now
                                                                   and provider officials,   refining the SLAs,
                                                                   after award, there        including those
                                                                   have been instances       affecting
                                                                   in which they have        compensation.
                                                                   worked together on
                                                                   refining the SLAs,
                                                                   including those
                                                                   affecting
                                                                   compensation.




                                             Page 55                                              GAO-03-371 DOD IT Services Outsourcing
                                              Appendix IV
                                              Project’s Implementation of Phase III:
                                              Develop the Contract




(Continued From Previous Page)
                                                               Did the project use the practice?
                      Air Force C4
Practice              Services               Army TAC-SWA              MHS/ITO Help Desk Navy NMCI                         NIMA IT/IS
The contract should    Limited—All clauses Yes—All contract        Yes—All contract      Yes—All contract      Yes—All contract
include clauses for    are included in the   clauses are included. clauses are included. clauses are included. clauses are included.
                       contract, except a
• determining pricing clause pertaining to
  structures;          customer satisfaction
• performing           surveys.
  customer
  satisfaction surveys
  and using the
  results to redefine
  performance levels;
• terminating the
  contract, including
  early terminations;
• resolving contract
  disputes in a timely
  manner;
• taking work away,
  without penalty,
  from the provider
  for
  nonperformance;
• declaring a
  significant event
  that can lead to a
  change in the
  contract;
• defining
  performance
  requirements; and
• conducting
  regularly scheduled
  meetings.
Consider setting up   Yes—According to       Yes—According to          Yes—According to          Yes—According to          Yes—According to
master services       Kirtland Air Force     TAC-SWA project           MHS program               NMCI program              NIMA, the contract is
agreement under       Base officials, the    officials, the contract   officials, the contract   officials, the contract   considered the
which all             contract, which        is considered the         is considered the         is considered to be a     master services
arrangements          incorporates the       master services           master services           master services           agreement.
between client and    contractor proposal,   agreement.                agreement.                agreement.
provider operate.     governs all
                      arrangements
                      between the
                      government and the
                      contractor and is
                      considered the
                      master services
                      agreement.




                                              Page 56                                                 GAO-03-371 DOD IT Services Outsourcing
                                                   Appendix IV
                                                   Project’s Implementation of Phase III:
                                                   Develop the Contract




(Continued From Previous Page)
                                                                   Did the project use the practice?
                         Air Force C4
Practice                 Services                 Army TAC-SWA              MHS/ITO Help Desk Navy NMCI                    NIMA IT/IS
Include appropriate      Not applicable—The       Yes—                      Yes—Staff from         Yes—The contracting     Yes—The NIMA
representation from      C4 Services contract     Representatives from      affected program       team was built with     contract negotiating
each major               was awarded under a      the field and             management offices     experts from each of    team included
organizational unit on   sealed bid process       headquarters              and the chief          the major systems       representatives from
contract negotiation     and was not              commands and the          information offices    commands. Also, the     each major
team.                    negotiated.              contracting office        were on the            source selection        organizational unit.
                                                  participated in           negotiation team.      evaluation board
                                                  contract negotiation.                            consisted of more
                                                                                                   than 50 people from
                                                                                                   various commands.
Specify the use of       Not applicable—          Not applicable—           Yes—Call bandsc are    Yes—The contract        Yes—NIMA has
volume purchases to      According to Kirtland    According to TAC-         used to obtain         includes volume         authorized the
obtain optimal           Air Force Base           SWA project officials,    optimal discounts on   discounts. For          provider to use
discounts.               officials, such volume   such volume               the number of calls    example, seat           government sources,
                         purchases were not       purchases were not        being fielded to the   moves, adds, and        such as the Federal
                         relevant to this         relevant to this          help desk.             changes are             Supply Schedule, in
                         contract.                contract.                                        purchased in bulk       procuring products,
                                                                                                   only, because prices    services, and
                                                                                                   for these actions are   supplies related to
                                                                                                   lower when              this contract.
                                                                                                   purchased in groups     According to the
                                                                                                   of 250.                 contracting officer,
                                                                                                                           this authorization
                                                                                                                           was made so that the
                                                                                                                           provider could take
                                                                                                                           advantage of
                                                                                                                           discounts available to
                                                                                                                           the government.
Use third-party          No—This was not          No—NETCOM                 Yes—Private-sector     Yes—The Navy used       Yes—NIMA obtained
assistance in            done because the Air     officials believed that   and other              private-sector firms    help from (1) private
negotiating and          Force staff believed     the Army’s                government             to assist in            contractors to help
developing the           that it had adequate     designated                organizations          developing the          develop the
contract.                expertise available      contracting agency        provided assistance.   overall NMCI concept    performance work
                         in-house.                had adequate                                     and negotiation         statement, SLAs,
                                                  expertise and                                    strategies as well as   and award fee plan;
                                                  experience.                                      in drafting the         (2) another agency
                                                                                                   contract documents.     on a particular
                                                                                                                           contract technique;
                                                                                                                           and (3) a private-
                                                                                                                           sector firm to
                                                                                                                           compare the
                                                                                                                           vendor’s proposal
                                                                                                                           with industry best
                                                                                                                           practices and trends
                                                                                                                           and to attend some
                                                                                                                           negotiation meetings
                                                                                                                           to answer questions.




                                                   Page 57                                             GAO-03-371 DOD IT Services Outsourcing
                                                  Appendix IV
                                                  Project’s Implementation of Phase III:
                                                  Develop the Contract




(Continued From Previous Page)
                                                                     Did the project use the practice?
                        Air Force C4
Practice                Services                 Army TAC-SWA                MHS/ITO Help Desk Navy NMCI                        NIMA IT/IS
Sign contract after     Yes—The contract         Yes—The contract            Yes—The contract         Yes—The contract          Yes—The contract
contract negotiations   was signed after         was signed after            was signed after         was signed after          was signed after
and final vendor        negotiations and final   negotiations and final      negotiations and final   negotiations and final    negotiations with the
selections.             vendor selection.        vendor selection.           vendor selection.        vendor selection.         selected vendor.
Source: GAO.
                                                  a
                                                   Requiring a provider to meet escalating requirements and focusing on achieving results at the least
                                                  cost are not mutually exclusive goals. As the provider becomes more familiar with the client
                                                  organization, it may be able to exceed the original performance requirements at the same, or possibly
                                                  lower, cost.
                                                  b
                                                      The NMCI contract was subsequently extended to 7 years, with an option for an additional 3 years.
                                                  c
                                                   Call bands are call volume ranges used to determine contractor pricing.




                                                  Page 58                                                   GAO-03-371 DOD IT Services Outsourcing
Appendix V

Projects’ Implementation of Phase IV: Select
the Provider(s)                                                                                        Append
                                                                                                            x
                                                                                                            i
                                                                                                            V




               Critical to the success of any project to outsource IT services is the
               identification of potential providers and the ultimate selection of a
               provider(s) that will best meet the needs of the organization. Developing a
               strategy that will lead to the selection of the “right contractor” is especially
               important in a performance-based acquisition.1 The overall success of the
               outsourcing project requires the contractor to understand the
               performance-based approach, know or develop an understanding of the
               organization’s requirement, have a history of performing exceptionally in
               the field, and have the processes and resources in place to support the
               mission.

               The seven practices in this phase that we used to evaluate the five projects
               are as follows:

               • Conduct research on the state of the market, vendors, and technology
                 before defining vendor selection criteria.

               • Identify and evaluate various sourcing solutions (e.g., single vendor,
                 multivendor, and alliance).

               • Define a process for selecting vendors to be providers.

               • Define vendor selection and evaluation (acceptance) criteria at the
                 outset.




               1
                An Interagency-Industry Partnership in Performance, Seven Steps to Performance-Based
               Services Acquisition, Benchmark Version (January 2002).




               Page 59                                        GAO-03-371 DOD IT Services Outsourcing
Appendix V
Projects’ Implementation of Phase IV: Select
the Provider(s)




• When issuing an RFP, identify services with expected performance
  levels and define client and provider roles and responsibilities.

• Use third-party assistance with expertise in a variety of outsourcing
  arrangements when selecting provider(s), including developing the RFP.

• Conduct due diligence activities to verify vendor capabilities before
  signing the contract.

As figure 9 illustrates, all five projects largely implemented the practices.



Figure 9: Project Implementation of Phase IV: Select the Provider(s)




Note: Not applicable—The practice was not relevant to the project’s particular circumstances.
No—The agency did not implement the practice. Limited—The agency fully implemented some but not




Page 60                                             GAO-03-371 DOD IT Services Outsourcing
                                                 Appendix V
                                                 Projects’ Implementation of Phase IV: Select
                                                 the Provider(s)




                                                 all aspects of the practice and did not take alternative actions that fully satisfied the practice. Yes—The
                                                 agency fully implemented the practice or took an alternative action that fully satisfied its intent.


                                                 Table 7 provides detailed information on whether and how the projects in
                                                 our review implemented each of the seven practices in this phase.



Table 7: Summary of Projects’ Use of Phase IV Practices

                                                                   Did the project use the practice?
                       Air Force C4
Practice               Services                 Army TAC-SWA                MHS/ITO Help Desk Navy NMCI                          NIMA IT/IS
Conduct research on    Yes—The Air Force        Yes—A Commerce              Yes—MHS had a             Yes—The Navy and a         Limited—NIMA staff
state of the market,   advertised in the        Business Daily notice       third-party contractor    private-sector firm        and a contractor
vendors, and           Commerce Business        was published               perform market            performed market           researched
technology before      Daily and interested     identifying TAC-SWA         research.                 research to, for           preferential providers
defining vendor        parties submitted        requirements to                                       example, help define       before defining
selection criteria.    statements of            interested parties. In                                the market conditions      vendor selection
                       capabilities. This was   addition, a draft RFP                                 and vendor selection       criteria. NIMA
                       done before the Air      was issued before                                     criteria. The Navy         program officials
                       Force identified the     the final solicitation to                             also held meetings         noted that they
                       vendor selection         solicit comments                                      with leading               limited their analysis
                       criteria.                from industry that                                    companies that had         to such providers
                                                might affect the                                      outsourced IT              because they had
                                                requirements.                                         services on an             previously decided
                                                According to the                                      enterprisewide level.      on a strategy to
                                                contracting officer,                                  Finally, more than         directly convert their
                                                comments received                                     200 companies              IT/IS activities to a
                                                from industry were                                    participated in the        preferential provider.
                                                incorporated into the                                 July 1999 NMCI
                                                final solicitation,                                   Industry Day
                                                where applicable.                                     Conference, which
                                                Additionally, a                                       informed companies
                                                preproposal                                           about the NMCI
                                                conference was                                        vision, requirements,
                                                conducted in the                                      and procurement
                                                overseas operations                                   strategy.
                                                location to inform
                                                industry
                                                representatives
                                                about issues and the
                                                procurement
                                                strategy.




                                                 Page 61                                                   GAO-03-371 DOD IT Services Outsourcing
                                                Appendix V
                                                Projects’ Implementation of Phase IV: Select
                                                the Provider(s)




(Continued From Previous Page)
                                                               Did the project use the practice?
                        Air Force C4
Practice                Services               Army TAC-SWA            MHS/ITO Help Desk Navy NMCI                      NIMA IT/IS
Identify and evaluate   Yes—As outlined in     Yes—A multivendor       Yes—According to an     Yes—The Navy             Yes—NIMA
various sourcing        OMB Circular A-76,     approach has been       MHS program             evaluated single-        considered the
solutions (e.g.,        the Air Force          used in the past,       official, MHS           vendor and               sourcing solutions
single-vendor,          evaluated private-     which the Army          evaluated continuing    multivendor              allowed under OMB
multivendor, and        sector and internal    opted not to            with the status quo,    approaches. A joint      Circular A-76,
alliance).              government staff       continue. Instead,      using another federal   Navy and Marine          including the direct
                        proposals to satisfy   NETCOM chose to         agency’s help-desk      Corps team               conversion to a
                        the C4 services        consolidate its         services as part of a   determined that          preferential provider,
                        requirements.          requirements and        cross-services          single-vendor point of   holding a
                                               contractor oversight    agreement, and          contact for              public/private
                                               by choosing a single-   contracting with a      accountability and       competition, or
                                               vendor solution.        commercial firm.        responsibility was       obtaining an agency
                                                                       According to the        critical to mission      cost comparison
                                                                       MHS program office,     success.                 waiver. NIMA
                                                                       MHS chose a single-                              decided that directly
                                                                       vendor solution to                               converting the
                                                                       consolidate                                      selected IT functions
                                                                       requirements and                                 to a preferential
                                                                       performance                                      procurement sourcea
                                                                       accountability.                                  was the lowest risk to
                                                                                                                        its mission and NIMA
                                                                                                                        employees.




                                                Page 62                                            GAO-03-371 DOD IT Services Outsourcing
                                                 Appendix V
                                                 Projects’ Implementation of Phase IV: Select
                                                 the Provider(s)




(Continued From Previous Page)
                                                                 Did the project use the practice?
                       Air Force C4
Practice               Services                 Army TAC-SWA           MHS/ITO Help Desk Navy NMCI                    NIMA IT/IS
Define a process for   Yes—As explained in      Yes—The contract       Yes—MHS used a           Yes—The NMCI RFP      Limited—NIMA’s
selecting vendors to   the invitation to bid,   was awarded on         selection process        used a negotiated     decision to use a
be providers (e.g.,    the Air Force            “best value”           that included the        commercial items      phased direct
issuing an RFP and     followed the two-step    considerations of      prequalification of 13   evaluation process.   conversion of its
prequalifying          process outlined in      technical and          industry leaders in      The vendor            IT/IS functions to a
vendors).              OMB Circular A-76        management             both medical and         evaluation criteria   statutory, preferential
                       for public/private       capabilities, past     commercial help-         included technical    procurement vendor
                       competitions.            performance, and       desk operations.         approach,             limited its vendor
                                                price, as defined in   MHS also issued a        management plan,      selection process to
                                                the solicitation.      request for comment      small business        the identification of
                                                                       and a request for        utilization, past     potential Alaska
                                                                       quote, which defined     performance, and      Native Corporation
                                                                       vendor evaluation        price.                companies capable
                                                                       and selection criteria                         of performing the
                                                                       on the basis of                                requirements. NIMA
                                                                       technical approach,                            decided to take this
                                                                       past performance,                              approach because it
                                                                       key personnel                                  believed that it would
                                                                       qualifications,                                avoid schedule
                                                                       organizational                                 delays and mission
                                                                       experience, and                                risk that it thought
                                                                       price.                                         would result from
                                                                                                                      taking other
                                                                                                                      approaches. In
                                                                                                                      addition, NIMA
                                                                                                                      program officials
                                                                                                                      believed that the
                                                                                                                      preferential provider
                                                                                                                      approach would
                                                                                                                      maximize the
                                                                                                                      retention of
                                                                                                                      institutional
                                                                                                                      knowledge whether
                                                                                                                      employees stayed at
                                                                                                                      NIMA or transitioned
                                                                                                                      to the provider.




                                                 Page 63                                            GAO-03-371 DOD IT Services Outsourcing
                                                 Appendix V
                                                 Projects’ Implementation of Phase IV: Select
                                                 the Provider(s)




(Continued From Previous Page)
                                                                 Did the project use the practice?
                         Air Force C4
Practice                 Services               Army TAC-SWA              MHS/ITO Help Desk Navy NMCI                        NIMA IT/IS
Define vendor            Yes—The criteria       Yes—The solicitation      Yes—Vendor               Yes—The final RFP         Yes—NIMA
selection and            were included in the   included vendor           evaluation and           required bidders to       evaluated the past
evaluation               invitation to bid.     evaluation and            selection were based     have implemented          performance and
(acceptance) criteria                           selection criteria. The   upon industry-           and provided service      financial capabilities
at the outset.                                  source selection          defined help-desk        to at least 100,000       of Alaska Native
                                                evaluation plan           criteria. MHS used a     seats, of which           Corporation
                                                describes the             multistep process        20,000 were to be on      companies.
                                                evaluation process.       that included an         the same contract. In     According to NIMA,
                                                                          assessment of            addition, bidders         only two companies
                                                                          minimum                  were evaluated on         had demonstrated
                                                                          qualifications,          their technical           successful
                                                                          evaluation of written    approach,                 performance in
                                                                          technical and cost       management plan,          service environments
                                                                          proposals, and oral      small business            similar to NIMA’s
                                                                          presentations for        utilization, past         requirements.
                                                                          qualified vendors.       performance, and          However, to be able
                                                                                                   price. Finally, vendors   to fully meet NIMA’s
                                                                                                   in the competitive        requirements, these
                                                                                                   range had to provide      vendors formed a
                                                                                                   demonstrations of         joint venture, which
                                                                                                   technical network         was awarded the
                                                                                                   operations centers to     contract.
                                                                                                   verify their ability to
                                                                                                   achieve required
                                                                                                   service levels set
                                                                                                   forth in technical
                                                                                                   proposals.
When issuing an          Yes—The services       Yes—The RFP               Yes—The request for      Yes—The NMCI              Yes—Because this
RFP, identify services   and performance        identified the            quote identified         solicitation              was a sole source
with expected            levels were included   government and            services and             documents identified      contract, an RFP was
performance levels       in the original        contractor roles and      expected                 the required services     not issued. However,
and define client and    performance work       responsibilities, and     performance levels       and expected              according to NIMA
provider roles and       statement included     system operational        and specified a          performance levels.       and provider officials,
responsibilities.        in the invitation to   and availability          performance-based,       These documents,          they worked jointly to
                         bid.                   requirements.             incentivized, shared-    along with the            refine the
                                                                          risk relationship with   contract, define the      requirements and
                                                                          the service provider.    client and provider       expected
                                                                          It also defined client   roles and                 performance levels
                                                                          and provider roles       responsibilities.         and to define client
                                                                          and responsibilities.                              and provider roles
                                                                                                                             and responsibilities
                                                                                                                             in the contract.




                                                 Page 64                                               GAO-03-371 DOD IT Services Outsourcing
                                                    Appendix V
                                                    Projects’ Implementation of Phase IV: Select
                                                    the Provider(s)




(Continued From Previous Page)
                                                                    Did the project use the practice?
                          Air Force C4
Practice                  Services                 Army TAC-SWA             MHS/ITO Help Desk Navy NMCI                        NIMA IT/IS
Use third-party           No—This was not          No—The Army              Yes—A private-           Yes—A private-sector      Yes—NIMA used a
assistance with           done because the Air     believed that it had     sector contractor, the   firm assisted with        private-sector firm
expertise in a variety    Force believed that      adequate                 Defense Acquisition      market research and       and the NIMA
of outsourcing            adequate in-house        government               University, and          developing vendor         Acquisition Center to
arrangements when         expertise was            experience and           GovWorks (an             pass/fail criteria.       support the
selecting provider(s),    available.               expertise covering       Interior organization    Another contractor        evaluation of the
including developing                               technical, resource      offering procurement     assisted the Navy         vendor proposal.
the RFP.                                           management, and          services to              with developing SLAs
                                                   contracting areas.       government               and the technical
                                                                            agencies) provided       evaluation of
                                                                            assistance in this       vendors.
                                                                            area.
Conduct due               Yes—The Air Force        Yes—NETCOM               Yes—Minimum              Yes—As part of due        Yes—NIMA
diligence activities to   performed a              evaluated the            vendor qualifications    diligence, the Navy       researched both
verify vendor             technical evaluation     contractor’s financial   were established and     ensured that all          parent companies of
capabilities before       of the vendor’s          and past                 evaluated early in the   bidders had relevant      the new joint venture
signing the contract.     capabilities. Also, as   performance              process for selecting    experience                corporation to ensure
                          part of its due          information before       the provider.            implementing large        that they had
                          diligence activities,    signing the contract.                             seat management           financial and
                          the Air Force                                                              contracts. In addition,   technical backing.
                          reviewed the                                                               past performance,
                          debarred list and                                                          including reference
                          DOD’s Central                                                              checks, was a
                          Contractor Registry.                                                       source-selection
                                                                                                     evaluation factor.
Source: GAO.
                                                    a
                                                     Preferential procurement programs are special commercial source programs, such as Federal Prison
                                                    Industries and the workshops administered by the Committee for the Purchase from the Blind and
                                                    Other Severely Handicapped under the Javis-Wagner O’Day Act.




                                                    Page 65                                              GAO-03-371 DOD IT Services Outsourcing
Appendix VI

Projects’ Implementation of Phase V:
Transition to Provider(s)                                                                     Appendx
                                                                                                    iVI




               In this phase, client organizations transfer responsibility of IT functions to
               one or more providers. A clear definition of responsibilities among the
               various parties and the careful consideration of employees’ needs matched
               against the organization’s needs enable both the client and provider to
               focus on execution and give staff confidence in their future employment. If
               the contractor is assuming responsibility for functions previously
               performed by federal employees, it is especially important that the
               organization communicate a clear transition process. Without such
               communication, an outsourcing project can be negatively affected if
               misinformation and mistrust ensues.

               The 11 practices in this phase that we used to evaluate the five projects are
               as follows:

               • Communicate a clear transition process to all key players from both
                 client and provider organizations.

               • Clearly communicate to employees what is going to happen and when it
                 is going to happen.

               • Establish a client transition team with representatives from across the
                 organization to facilitate the transition.

               • Place the transition under a single program manager.

               • Create client/provider transition teams to address short-term transition
                 tasks as required.

               • Recognize that it takes time to effect transition and plan accordingly.




               Page 66                                   GAO-03-371 DOD IT Services Outsourcing
Appendix VI
Projects’ Implementation of Phase V:
Transition to Provider(s)




• Encourage the transition of staff to the provider, where appropriate,
  using bonuses, stock options, and other appropriate methods.

• Develop employee-retention programs and offer bonuses to keep key
  people, where appropriate.

• When consistent with organizational objectives, assist employees who
  do not want to transfer in finding other jobs, either within an
  organization or at another organization.

• Document key information to preserve organizational knowledge in the
  event that one or more providers change.

• Use change management strategies to help client employees deal with
  the transition.

Figure 10 demonstrates that the five projects in our review were largely
using the practices.




Page 67                                  GAO-03-371 DOD IT Services Outsourcing
Appendix VI
Projects’ Implementation of Phase V:
Transition to Provider(s)




Figure 10: Project Implementation of Phase V: Transition to Provider(s)




Note: Not applicable—The practice was not relevant to the project’s particular circumstances.
No—The agency did not implement the practice. Limited—The agency fully implemented some but not
all aspects of the practice and did not take alternative actions that fully satisfied the practice. Yes—The
agency fully implemented the practice or took an alternative action that fully satisfied its intent.


Table 8 provides information on whether and how the projects
implemented each of the 11 practices in this phase.




Page 68                                                   GAO-03-371 DOD IT Services Outsourcing
                                                   Appendix VI
                                                   Projects’ Implementation of Phase V:
                                                   Transition to Provider(s)




Table 8: Summary of Projects’ Use of Phase V Practices

                                                                  Did the project use the practice?
                        Air Force C4
Practice                Services                  Army TAC-SWA            MHS/ITO Help Desk Navy NMCI                       NIMA IT/IS
Communicate a clear     No—There were two         Yes—The provider        Yes—The contract         Yes—The Navy             Yes—Several NIMA
transition process to   transition plans          developed a phase-in    delivery order and       developed numerous       transformation teams
all key players from    prepared, one by the      plan that addressed     the provider’s           guides and Web sites     were formed,
both client and         government and one        personnel issues, the   transition plan laid     to communicate with      consisting of both
provider                by the provider.          integration of added    out a transition         the NMCI community       client and provider
organizations.          However, the              locations, updating     process. In addition,    about site, technical,   personnel, to develop
                        government’s plan         plans, and finalizing   a kickoff meeting was    and personnel            a single transition
                        contained conflicting     new subcontract/        held between the         transition processes,    plan. The plan was
                        direction. In addition,   sponsorship             client and provider to   procedures, and          developed to
                        provider officials said   arrangements. This      communicate the          tasks.                   formalize
                        that the two separate     phase-in plan was       transition process to                             communications with
                        transition plans were     discussed at a          all key players.                                  all key players.
                        not well coordinated      postaward
                        and that the Air          conference, attended
                        Force did not have a      by representatives
                        strong advocate to        from both the client
                        ensure that the           and provider.
                        transition process
                        was well planned and
                        executed. Both
                        provider and Kirtland
                        Air Force Base
                        officials
                        acknowledged that
                        there were problems
                        with the transition
                        process, including
                        incorrect information
                        about upgrades that
                        would be made
                        before the transition
                        and staff morale
                        problems that
                        hampered
                        knowledge transfer to
                        the provider.




                                                   Page 69                                              GAO-03-371 DOD IT Services Outsourcing
                                                 Appendix VI
                                                 Projects’ Implementation of Phase V:
                                                 Transition to Provider(s)




(Continued From Previous Page)
                                                                 Did the project use the practice?
                       Air Force C4
Practice               Services                 Army TAC-SWA             MHS/ITO Help Desk Navy NMCI                        NIMA IT/IS
Clearly communicate    Yes—The Air Force        Limited—According        Yes—The                  Limited—The Navy          Yes—With help from
to employees what is   kept its employees       to TAC-SWA officials,    government task          has used its normal       a contractor, NIMA
going to happen and    informed of the          the Army                 manager for the Tri-     chain of command to       developed a
when it is going to    outsourcing project      communicated with        Service                  communicate               communications
happen.                through town hall        its employees, but       Infrastructure           transition                plan. The plan
                       meetings and             they did not provide     Management Office        information, but          included town hall
                       internal briefings. As   documentation            held regular             found that the            meetings, global
                       part of this process,    supporting any           meetings with MHS        implementation was        E-mail messages,
                       employees were           communication            program officials to     uneven. As a result,      and a Web site
                       informed about the       efforts. However, the    discuss transition       some staff did not        posting that provided
                       reduction-in-force       provider was             issues. The MHS          know current              employees with
                       process and the          required to submit a     program office also      information about         access to transition
                       procedure for those      phase-in plan, which     periodically sent        how NMCI would            information.
                       interested in being      included a section on    global E-mails to        affect them until the
                       transitioned or          communicating with       MHS staff on             provider was ready to
                       reassigned.              the incumbent            transition activities    contact them
                                                contractors’ staff.      and MHS leadership       regarding their
                                                                         made on-site visits to   possible transition to
                                                                         San Antonio, the         the contractor.
                                                                         location of the help-    However, according
                                                                         desk function.           to the NMCI
                                                                                                  Director’s office, this
                                                                                                  problem was
                                                                                                  somewhat mitigated
                                                                                                  by the provider’s
                                                                                                  Web site that
                                                                                                  provides transition
                                                                                                  information to all
                                                                                                  NMCI
                                                                                                  customers/users.
Establish client       Yes—The                  Yes—According to         Yes—MHS program          Yes—The                   Yes—NIMA’s
transition team with   government’s             TAC-SWA officials,       officials for the Tri-   headquarters-level        transition team
representatives from   transition team          the contracting          Service                  transition team           included
across the             included personnel       officer’s                Infrastructure           consists of               representatives from
organization to        from the functional      representative           Management Office        representatives from      various offices within
facilitate the         areas being              assisted the provider    established transition   the NMCI Director’s       NIMA to help
transition.            outsourced (i.e.,        with the transition.     teams with               Office and the            facilitate the
                       communications and       The officials also       representatives from     program                   transition.
                       IT services) as well     stated that Army         the former Tri-          management office.
                       as personnel from        technical points of      Service Medical          Also, every site has a
                       other offices.           contact at each site     System Support           transition team made
                                                also helped facilitate   Center contract.         up of customer and
                                                the transition to the                             provider personnel.
                                                new contract.




                                                 Page 70                                               GAO-03-371 DOD IT Services Outsourcing
                                              Appendix VI
                                              Projects’ Implementation of Phase V:
                                              Transition to Provider(s)




(Continued From Previous Page)
                                                              Did the project use the practice?
                      Air Force C4
Practice              Services               Army TAC-SWA             MHS/ITO Help Desk Navy NMCI                      NIMA IT/IS
Place transition     Yes—The Air Force       Yes—The Army             Yes—The transition      Yes—NMCI’s               Yes—The transition
under single program established a single    designated the           was assigned to the     Director has overall     process is under a
manager.             program manager,        contracting officer’s    government task         responsibility for the   single program
                     who is referred to as   representative to        manager in the Tri-     transition.              manager.
                     the functional area     oversee the              Service
                     chief.                  transition.              Infrastructure
                                                                      Management Office.
Create                Yes—During the         Yes—Personnel from       Yes—Client/provider     Yes—NMCI and the         Yes—An integrated
client/provider       transition phase,      both the Army and        transition teams were   provider designated      NIMA transformation
transition teams to   management from        provider formed          established to          staff to work together   team comprising
address short-term    both the government    three transition         address short-term      on short-term            NIMA and provider
transition tasks as   and the provider met   teams, one for each      transition tasks. The   transition issues.       staff, was
required.             weekly to track the    site location. These     teams met weekly to                              responsible for
                      progress of the        teams worked with        address any                                      addressing short-
                      transition.            the contracting          transition issues.                               term tasks, including
                                             officer’s                                                                 (1) transition
                                             representative and                                                        planning, (2)
                                             the provider’s project                                                    resources and
                                             manager to deal with                                                      recruitment, (3)
                                             transition issues.                                                        program budget, and
                                                                                                                       (4) contract
                                                                                                                       development and
                                                                                                                       costing.




                                              Page 71                                              GAO-03-371 DOD IT Services Outsourcing
                                             Appendix VI
                                             Projects’ Implementation of Phase V:
                                             Transition to Provider(s)




(Continued From Previous Page)
                                                            Did the project use the practice?
                       Air Force C4
Practice               Services              Army TAC-SWA          MHS/ITO Help Desk Navy NMCI                        NIMA IT/IS
Recognize that it      Yes—The Air Force     Yes—The contract      Yes—The contract         Limited—The Navy          Yes—According to
takes time to effect   allowed 60 days for   allowed a 60-day      provided for a           said that it had          NIMA, it has
transition and plan    the transition.       phase-in period.      transition period of     initially                 scheduled its seven
accordingly.                                                       30 to 90 days.           underestimated the        functional areas to
                                                                   According to MHS         scope of the project      be transitioned to the
                                                                   program officials, the   and the magnitude of      provider over a 3- to
                                                                   transition period        the problems brought      5-year period. The
                                                                   lasted 90 days.          on by legacy              provider will not
                                                                                            applications and          assume
                                                                                            associated                responsibility for a
                                                                                            information               given functional area
                                                                                            assurance and             until the NIMA
                                                                                            cyber-security            transition review
                                                                                            issues. According to      board has given its
                                                                                            NMCI program              approval to the
                                                                                            officials, addressing     provider’s turnover
                                                                                            these problems and        plan.
                                                                                            completing the
                                                                                            operational testing
                                                                                            that was mandated
                                                                                            by the Congress
                                                                                            subsequent to
                                                                                            contract award led to
                                                                                            the transition period
                                                                                            being lengthened
                                                                                            from 2-1/2 to
                                                                                            3-1/2 years. NMCI
                                                                                            officials also said the
                                                                                            extension will allow
                                                                                            the Department of
                                                                                            the Navy to have
                                                                                            time to operate NMCI
                                                                                            as a fully transitioned
                                                                                            enterprise before
                                                                                            having to decide
                                                                                            whether to exercise
                                                                                            the contract option.




                                             Page 72                                             GAO-03-371 DOD IT Services Outsourcing
                                                    Appendix VI
                                                    Projects’ Implementation of Phase V:
                                                    Transition to Provider(s)




(Continued From Previous Page)
                                                                   Did the project use the practice?
                        Air Force C4
Practice                Services                   Army TAC-SWA             MHS/ITO Help Desk Navy NMCI                     NIMA IT/IS
Encourage transition    Yes—The Air Force          Yes—The Army had         Yes—According to       Yes—The NMCI             Yes—The contract
of staff to provider,   staff were given the       no personnel             provider officials,    contract provides        requires the provider
where appropriate,      right to transfer to the   expected to transition   they extended offers   that displaced federal   to actively recruit,
using bonuses, stock    contractor. It was left    to the provider since    to employees from      employees that           hire, and make
options, and other      up to the contractor       the activity being       the prior contractor   transition to the        reasonable efforts to
appropriate methods.    to decide whether to       outsourced had           and some MHS staff.    provider under           retain NIMA staff. If
                        offer incentives or        previously been                                 employment               the provider meets
                        not, which it chose        contracted out.                                 openings as a result     certain target
                        not to do.                 However, the                                    of NMCI will be given    thresholds for
                                                   provider was                                    guaranteed 3 years       recruiting NIMA staff,
                                                   interested in                                   of employment with       it receives a
                                                   retaining existing                              the provider, a 15       monetary incentive.
                                                   contractor staff and                            percent salary           According to the
                                                   had a program to                                increase, and a sign-    provider’s general
                                                   recruit them.                                   on bonus.                manager, the
                                                                                                                            company offered
                                                                                                                            signing bonuses to
                                                                                                                            NIMA employees that
                                                                                                                            wanted to transition.




                                                    Page 73                                             GAO-03-371 DOD IT Services Outsourcing
                                                Appendix VI
                                                Projects’ Implementation of Phase V:
                                                Transition to Provider(s)




(Continued From Previous Page)
                                                               Did the project use the practice?
                       Air Force C4
Practice               Services                Army TAC-SWA           MHS/ITO Help Desk Navy NMCI                       NIMA IT/IS
Develop employee-      No—According to         Not applicable—        Yes—The                  Yes—NMCI program         Yes—According to
retention programs     Kirtland Air Force      Since this activity    government task          officials noted that     NIMA program
and offer bonuses to   Base officials, they    was previously         manager persuaded        the Navy has human       officials, NIMA
keep key people,       did not develop an      contracted out,        key government           resources programs       categorized affected
where appropriate.     employee-retention      employee retention     employees from the       that would allow its     employees into three
                       program because         of Army civilian       Tri-Service Medical      organizations a          tiers, on the basis of
                       they did not believe    personnel was not      System Support           range of options,        whether they would
                       that they could offer   relevant.              Center to temporarily    including bonuses, to    be allowed to
                       incentives.                                    work at the Tri-         help keep key staff.     transition to the
                                                                      Service                  Additionally, the Navy   provider or whether
                                                                      Infrastructure           directed all Navy and    there would be
                                                                      Management Office        Marine Corps             restrictions on their
                                                                      until positions          commands affected        activities if they did
                                                                      became available         by NMCI to develop       transition. This
                                                                      with a contractor that   civilian personnel       distinction was made
                                                                      provides support to      transition plans to      to comply with
                                                                      this office. In          mitigate the impact of   government conflict
                                                                      addition, according to   NMCI                     of interest rules.
                                                                      a program official, to   implementation on        According to the
                                                                      keep expertise in        employees.               program manager,
                                                                      particular                                        NIMA also sought
                                                                      applications, key                                 personnel who
                                                                      employees were                                    wanted to remain at
                                                                      transferred to the                                the agency to help
                                                                      MHS program                                       monitor the
                                                                      executive office and                              provider’s
                                                                      Tri-Service                                       performance and
                                                                      Infrastructure                                    considered the
                                                                      Management Office.                                qualifications of
                                                                                                                        those who wished to
                                                                                                                        serve in this role
                                                                                                                        prior to deciding who
                                                                                                                        to retain.




                                                Page 74                                            GAO-03-371 DOD IT Services Outsourcing
                                                     Appendix VI
                                                     Projects’ Implementation of Phase V:
                                                     Transition to Provider(s)




(Continued From Previous Page)
                                                                     Did the project use the practice?
                          Air Force C4
Practice                  Services                  Army TAC-SWA              MHS/ITO Help Desk Navy NMCI                         NIMA IT/IS
When consistent with      Yes—Kirtland Air          Not applicable—           Yes—According to          Yes—NMCI program          Yes—For staff that
organizational            Force Base officials      According to the          MHS program               officials said that       did not want to
objectives, assist        stated that they have     TAC-SWA                   officials, the            existing Navy civilian    transition, NIMA
employees who do          reduction-in-force        contracting officer, no   government task           personnel programs        provided various
not want to transfer in   procedures to help        Army personnel were       manager assisted          are in place to assist    types of assistance,
finding other jobs,       place civilian            displaced by the          government                employees. They           such as retirement
either within an          employees in other        contract.                 employees with the        noted that two Navy       planning, résumé
organization or at        Air Force jobs.                                     Tri-Service Medical       organizations found       writing, and
another organization.     Employees also                                      System Support            other jobs within their   interviewing skills. In
                          receive priority status                             Center staff in finding   command for their         addition, NIMA
                          for other DOD job                                   new positions with        small number of           reported that if an
                          openings. Further,                                  other MHS                 employees affected        employee did not
                          the base education                                  organizations. For        by NMCI. Other Navy       want to transition to
                          office provided                                     example, he located       organizations are         the provider, that it
                          assistance with                                     open positions and        also reassigning          would consider re-
                          outplacement;                                       contacted the             affected personnel to     adjusting the
                          résumé writing; and,                                organizations.            other government          employee’s work
                          for those nearing                                                             jobs, where               assignment or
                          retirement, planning                                                          applicable.               provide training to
                          advice.                                                                                                 support the
                                                                                                                                  individual’s
                                                                                                                                  placement within
                                                                                                                                  other areas.
Document key              Yes—Kirtland Air          Yes—Documentation         Yes—The Tri-Service       Yes—NMCI                  Yes—The executing
information to            Force Base’s work         produced by the           Medical System            maintains a Web           NIMA transformation
preserve                  functions and             provider, including       Support Center            portal with               team was charged
organizational            workload size are         maintenance logs,         processes and             organizational            with documenting the
knowledge in the          documented in the         become the property       procedures were           knowledge                 activities for each
event that one or         contract.                 of the government.        captured during           information. In           functional area that is
more providers                                                                transition. The           addition, a new           being outsourced.
change.                                                                       provider’s processes      Intranet site is being    This includes
                                                                              and procedures are        developed with up-to-     developing and
                                                                              currently being           date interactive          capturing operational
                                                                              captured. The             organizational            processes and
                                                                              contract also             knowledge relevant        procedures for each
                                                                              includes                  to the many varied        of the functional
                                                                              requirements on           NMCI communities.         areas.
                                                                              documenting key
                                                                              information.




                                                     Page 75                                                 GAO-03-371 DOD IT Services Outsourcing
                                                Appendix VI
                                                Projects’ Implementation of Phase V:
                                                Transition to Provider(s)




(Continued From Previous Page)
                                                               Did the project use the practice?
                     Air Force C4
Practice             Services                  Army TAC-SWA            MHS/ITO Help Desk Navy NMCI                   NIMA IT/IS
Use change           Yes—Kirtland Air          Not applicable—The      Yes—According to         Yes—The Navy has     Yes—NIMA’s change
management           Force Base held           activity had been       MHS program              provided employees   management
strategies to help   town hall meetings        previously contracted   officials, the           affected by NMCI     strategies included
client employees     and internal briefings    out.                    government task          with a range of      training seminars
deal with the        to communicate the                                manager for the Tri-     employment options   offered by its human
transition.          objectives of the                                 Service                  and incentives.      resources office on
                     outsourcing project                               Infrastructure                                résumé writing,
                     and the changes that                              Management Office                             interviewing skills,
                     would take place                                  assisted government                           career transition
                     internally. It also had                           employees with                                workshops, and
                     one-on-one meetings                               finding new positions                         employment trends.
                     with each employee                                and kept them
                     that would be                                     informed of transition
                     displaced.                                        issues. Weekly
                                                                       E-mails were also
                                                                       sent out to all
                                                                       affected employees.
Source: GAO.




                                                Page 76                                             GAO-03-371 DOD IT Services Outsourcing
Appendix VII

Projects’ Implementation of Phase VI: Manage
Provider(s) Performance                                                                                Append
                                                                                                            x
                                                                                                            iVI




               The effectiveness with which the performance of the provider(s) is
               managed—the focus of this phase—is critical to the successful
               implementation of an outsourcing project. Indeed, according to Gartner,
               Inc., an outsourcing project can be thwarted by poorly designed, funded,
               and delivered processes for managing the delivery of services.1 This firm
               also points out that an enterprise needs to retain the resources to oversee
               the planning and implementation of the IT services being delivered by the
               provider to ensure that the contractor meets the client’s business needs
               throughout the life of the agreement.2 Moreover, frequent and clear
               communication between the client and provider ensures that potential
               problems are resolved before they cause disruptions. In addition,
               performance reviews should take place regularly to keep the project on
               course, measure performance levels, and make adjustments as necessary.3

               The 11 practices in this phase that we used to evaluate the five projects are
               as follows:

               • Consider incentives to motivate provider(s) to exceed performance
                 requirements.




               1
                Gartner, Inc., Retain Enough Resources to Manage Outsourcing Deals, Research Note
               COM-16-8425 (June 17, 2002).
               2
                Gartner, Inc., Successful Outsourcing Means Retaining Some Staff, Research Note COM-
               18-9692 (Dec. 18, 2002).
               3
                An Interagency-Industry Partnership in Performance, Seven Steps to Performance-Based
               Services Acquisition, Benchmark Version (January 2002).




               Page 77                                        GAO-03-371 DOD IT Services Outsourcing
Appendix VII
Projects’ Implementation of Phase VI:
Manage Provider(s) Performance




• Use penalties to motivate provider(s) to meet performance
  requirements.

• Periodically undertake studies to assess (1) how the provider’s
  performance compares with the value being delivered to similar clients
  and (2) the extent to which the provider’s performance is improving
  over time.

• Schedule periodic working-level meetings with both the end-user groups
  and the provider to review the provider’s performance.

• Conduct executive-level oversight meetings with the provider’s senior
  management to review provider’s performance.

• Distribute performance data to stakeholders.

• Reserve audit rights on performance data supplied by the provider.

• Ensure that the provider measures and reports on performance.

• Work with the provider to redefine service levels, as appropriate.

• Sample performance data frequently enough to perform trend analysis
  and to permit extrapolation based on historical data.

• Allow employees and possibly stakeholders to rate the provider on a
  regular basis.

As shown by figure 11, the five projects in our review generally
implemented the practices.




Page 78                                  GAO-03-371 DOD IT Services Outsourcing
Appendix VII
Projects’ Implementation of Phase VI:
Manage Provider(s) Performance




Figure 11: Project Implementation of Phase VI: Managing Provider Performance




Note: Not applicable—The practice was not relevant to the project’s particular circumstances.
No—The agency did not implement the practice. Limited—The agency fully implemented some but not
all aspects of the practice and did not take alternative actions that fully satisfied the practice. Yes—The
agency fully implemented the practice or took an alternative action that fully satisfied its intent.


Table 9 depicts whether and how the five projects in our review
implemented each of the 11 practices in this phase.




Page 79                                                   GAO-03-371 DOD IT Services Outsourcing
                                                  Appendix VII
                                                  Projects’ Implementation of Phase VI:
                                                  Manage Provider(s) Performance




Table 9: Summary of Projects’ Use of Phase VI Practices

                                                                 Did the project use the practice?
                        Air Force C4
Practice                Services                 Army TAC-SWA             MHS/ITO Help Desk Navy NMCI                      NIMA IT/IS
Consider incentives     No—The Air Force         No—There are no          Yes—The contract         Yes—The NMCI            Yes—The contract
to motivate             did not include          monetary incentives,     includes incentives      contract includes a     includes monetary
provider(s) to exceed   incentives in the        but, according to        linked to each           one-time payment of     incentives in
performance             contract because the     TAC-SWA officials,       performance              $10 million if the      accordance with an
requirements.           agency expects the       the fact that the        requirement. Each        provider successfully   award fee plan. It
                        contractor to meet       agency will prepare a    requirement has a        completes full          also includes share-
                        the contract’s           performance              positive, negative,      operational             in-savings provisions
                        requirements without     evaluation report can    and acceptable           capability. The         to encourage
                        incentives.              help motivate a          performance range        contract also has       process
                        Specifically,            provider to meet         that provides a basis    incentives for          improvements.
                        according to the         requirements.            for monetary             customer
                        former functional        However, the             incentives (as well as   satisfaction,
                        area chief, because      contracting officer      penalties).              information
                        the contract was         said that incentives                              assurance, and small
                        issued under the         might have been                                   business and small
                        rules established by     useful to motivate the                            disadvantaged
                        OMB Circular A-76,       provider to exceed                                business
                        the focus of the         performance                                       participation.
                        project was on           requirements.
                        achieving cost
                        savings, and
                        incentives were not
                        included in the
                        contract. As a result,
                        he noted that the
                        provider did not have
                        any incentive to be
                        innovative or
                        creative.




                                                  Page 80                                              GAO-03-371 DOD IT Services Outsourcing
                                                    Appendix VII
                                                    Projects’ Implementation of Phase VI:
                                                    Manage Provider(s) Performance




(Continued From Previous Page)
                                                                   Did the project use the practice?
                          Air Force C4
Practice                  Services                 Army TAC-SWA            MHS/ITO Help Desk Navy NMCI                    NIMA IT/IS
Use penalties to           Yes—The contract        Limited—The             Yes—The contract       Yes—The contract        Limited—NIMA did
motivate provider(s)       contains monetary       contract does not       includes monetary      includes monetary       not include monetary
to meet performance        penalties that are      have specific           penalties linked to    penalties in the form   penalties in its
requirements, such         linked to each of the   monetary penalties      each performance       of credits to the       contract. However, if
as                         performance             linked to               requirement. Each      agency if the           the provider does not
                           requirements. If        performance             requirement has        provider fails to       meet certain
•   assess penalties for imposed, they would       requirements.           positive, negative,    perform to the levels   minimum
    failure to perform at reduce the amount of     However, the            and acceptable         specified in the        performance
    required individual the payment owed to        contract states that    performance ranges     SLAs.                   standards, only its
    service as well as     the provider.           unacceptable work       that provide a basis                           costs would be
    aggregate service                              must be redone at       for monetary                                   covered.
    levels;                                        the provider’s          penalties as well as
•   apply penalties in                             expense, and, if the    incentives.
    the form of credit to                          defects and services
    the client;                                    cannot be corrected,
•   increase penalty for                           the government may
    recurring deficient                            reduce the contract’s
    performance;                                   price to reflect the
•   hold back a                                    reduced value of the
    percentage of                                  services performed.
    provider’s pay for a                           TAC-SWA project
    particular service                             officials stated that
    until performance                              monetary penalties
    requirements are                               were not included in
    met;                                           the contract because
•   refund a penalty if                            the Army was
    the provider returns                           concerned that
    to agreed-upon                                 contractors might not
    performance levels                             bid on the contract,
    within a designated                            and administrating
    period of time; and                            this type of contract
•   ensure that the                                is more difficult.
    provider will cover
    costs, but not profit,
    when a particular
    performance
    requirement is not
    met.




                                                    Page 81                                           GAO-03-371 DOD IT Services Outsourcing
                                                   Appendix VII
                                                   Projects’ Implementation of Phase VI:
                                                   Manage Provider(s) Performance




(Continued From Previous Page)
                                                                   Did the project use the practice?
                         Air Force C4
Practice                 Services                 Army TAC-SWA            MHS/ITO Help Desk Navy NMCI                      NIMA IT/IS
Periodically             Limited—(1) This has     No—TAC-SWA              Yes—The provider’s       Limited—(1) NMCI        Limited—(1) At this
undertake studies to     not been done            officials stated that   performance is           only has                time, NIMA has not
assess: (1) how the      because, according       they address            reviewed monthly by      performance data        undertaken such
provider’s               to a project official,   performance issues      the government task      since October 2002.     studies because it
performance              the Air Force was not    at the time that they   manager and in           The Navy intends to     only recently
compares with the        aware of similar         occur and that they     quarterly                do such a study but     transitioned functions
value being delivered    clients.                 have not performed      management reviews       has not established a   to the provider.
to similar clients and   (2) This is done         any studies.            of the provider’s        schedule for it.        However, NIMA
(2) the extent to        through periodic                                 performance.             (2) The provider’s      program officials
which the provider’s     monitoring of                                    According to MHS         improvements over       stated that they
performance is           performance by                                   program officials, the   time are being          intend to use a
improving over time.     quality assurance                                results are compared     monitored monthly.      private-sector firm to
                         evaluators and                                   with peers and                                   periodically compare
                         through monthly                                  reviewed for how the                             the provider’s
                         review meetings held                             provider has                                     performance with
                         by the functional area                           improved over time.                              those of similar
                         chief to identify and                                                                             organizations.
                         address any                                                                                       (2) The provider’s
                         problems that are                                                                                 improvements over
                         starting to occur.                                                                                time are being
                                                                                                                           monitored monthly.
Schedule periodic        Yes—Monthly              Yes—According to        Yes—Periodic             Yes—At each             Yes—Quarterly
working-level            meetings are held at     the provider and        working-level            implemented site, the   performance
meetings with both       which the contractor     TAC-SWA officials,      meetings are held        provider and            management review
the end-user groups      briefs the functional    they hold daily         with the government      government              meetings are held
and the provider to      area chief and other     meetings to discuss     task manager and         managers meet on a      and are attended by
review the provider’s    Kirtland Air Force       any issues.             the program              frequent, as-needed     NIMA and the
performance.             Base                                             management office        basis to review         provider. In addition,
                         representatives.                                 representatives to       performance and         the provider hired an
                                                                          obtain help-desk         resolve any issues.     ombudsman to
                                                                          feedback.                                        obtain anonymous
                                                                                                                           comments from
                                                                                                                           NIMA staff and
                                                                                                                           management on its
                                                                                                                           performance.




                                                   Page 82                                             GAO-03-371 DOD IT Services Outsourcing
                                                  Appendix VII
                                                  Projects’ Implementation of Phase VI:
                                                  Manage Provider(s) Performance




(Continued From Previous Page)
                                                                  Did the project use the practice?
                      Air Force C4
Practice              Services                   Army TAC-SWA              MHS/ITO Help Desk Navy NMCI                     NIMA IT/IS
Conduct executive-    Yes—Air Force              Yes—According to          Yes—Executive           Yes—The NMCI            Yes—This is done
level oversight       schedules executive-       provider and TAC-         oversight meetings      Director and the        through the quarterly
meetings with the     level oversight            SWA officials,            are held quarterly      provider executive      review meetings.
provider’s senior     meetings whenever          quarterly in- progress    with the provider’s     meet weekly to
management to         they are necessary.        reviews are held with     senior management       review performance
review provider’s     A recent meeting           the provider’s            to review               and discuss other
performance.          included the               corporate                 performance.            NMCI
                      president of the           management and the                                implementation
                      provider.                  battalion                                         issues. In addition,
                                                 commander.                                        the Department of
                                                                                                   the Navy recently
                                                                                                   established an
                                                                                                   operations advisory
                                                                                                   board consisting of
                                                                                                   Department of the
                                                                                                   Navy leadership and
                                                                                                   the provider. The
                                                                                                   goal of this board is
                                                                                                   to focus senior
                                                                                                   leadership on issues
                                                                                                   affecting NMCI in
                                                                                                   order to establish
                                                                                                   priorities and make
                                                                                                   decisions.
Distribute            Yes—Performance            Yes—The contracting       Yes—The help-desk       Yes—The NMCI            Yes—Performance
performance data to   data were not              officer’s                 monthly reports,        Navy and Marine         data are distributed
stakeholders.         distributed to             representative            including results       Corps program           at quarterly review
                      stakeholders at the        distributes monthly       against performance     managers are            meetings, which
                      beginning of the           performance reports       metrics, are e-mailed   responsible for         according to the
                      project, but began         to Army stakeholders      to stakeholders         providing SLA           contracting officer,
                      after complaints from      for review.               monthly.                performance data to     are attended by
                      stakeholders about         Performance issues                                their commands.         stakeholders.
                      the need for such          or problems
                      information. Metrics,      identified are
                      such as system             discussed with the
                      reliability or “uptime,”   contracting officer’s
                      are now provided to        representative and
                      stakeholders on line.      the contracting
                                                 officer, and corrective
                                                 actions are planned
                                                 to prevent
                                                 reoccurrence.
Reserve audit rights Yes—Addressed in            Yes—Addressed in          Yes—Addressed in        Yes—Addressed in        Yes—Addressed in
on performance data the contract.                the contract.             the contract.           the contract.           the contract.
supplied by the
provider.




                                                  Page 83                                              GAO-03-371 DOD IT Services Outsourcing
                                                 Appendix VII
                                                 Projects’ Implementation of Phase VI:
                                                 Manage Provider(s) Performance




(Continued From Previous Page)
                                                                Did the project use the practice?
                        Air Force C4
Practice                Services                Army TAC-SWA           MHS/ITO Help Desk Navy NMCI                      NIMA IT/IS
Ensure that provider    Yes—As required by      Yes—The contracting    Yes—The contract         Yes—The contract        Yes—The contract
measures and            the quality control     officer’s              requires the provider    specifies that the      requires the provider
reports on              plan in the contract,   representative         to submit monthly        provider must           to submit data,
performance.            the provider makes      receives               and quarterly            measure and report      including
                        information available   performance            performance reports      on whether its SLA      measurements of
                        on product or service   information in daily   that are then            performance goals       service, quarterly.
                        quality and any         meetings with the      reviewed and             are being met.
                        actions needed to       provider’s project     verified.
                        correct decreases in    manager.
                        quality. In addition,
                        according to the
                        former functional
                        area chief, the
                        provider provides
                        performance
                        information during
                        monthly meetings.
Work with provider to   Yes—This is done on     Yes—According to       Yes—MHS and the          Yes—The Navy and        Yes—The redefinition
redefine service        an as-needed basis.     TAC-SWA and            provider have worked     service provider have   of service levels is
levels, as              For example, the        provider officials,    together to redefine     and continue to         provided for in the
appropriate.            service levels were     they have worked       SLAs. For example,       refine NMCI SLAs to     contract. According
                        redefined in April      together to redefine   the first call closure   ensure more precise     to NIMA, the
                        2002.                   performance            performance metric       performance             redefinition of service
                                                requirements in the    was redefined to be      measures and to         levels is expected to
                                                contract.              more realistic.          more accurately         occur in the future,
                                                                                                capture user            generally at the time
                                                                                                satisfaction with the   that the annual
                                                                                                system. For             contract option is
                                                                                                example, a contract     exercised.
                                                                                                modification
                                                                                                standardized many
                                                                                                of the performance
                                                                                                categories that dealt
                                                                                                with service
                                                                                                availability.




                                                 Page 84                                            GAO-03-371 DOD IT Services Outsourcing
                                                  Appendix VII
                                                  Projects’ Implementation of Phase VI:
                                                  Manage Provider(s) Performance




(Continued From Previous Page)
                                                                 Did the project use the practice?
                        Air Force C4
Practice                Services                 Army TAC-SWA             MHS/ITO Help Desk Navy NMCI                       NIMA IT/IS
Sample performance      Yes—Air Force            Yes—The provider’s       Yes—Performance          Yes—The NMCI             Yes—The contract
data frequently         quality assurance        proposal and quality     data are sampled         Director monitors        requires data to be
enough to perform       evaluators monitor       control plan state       and reported monthly     provider performance     provided periodically
trend analysis and to   performance in           that it will provide     and quarterly. They      to identify trends by    for NIMA to use for
permit extrapolation    accordance with the      various trend            are analyzed and         assessing provider-      trend and other types
based on historical     quality assurance        analyses to the          verified, including      supplied information     of analyses.
data.                   surveillance plan and    government.              any supporting data.     as well as
                        the performance          According to TAC-        In addition, the         information from
                        requirements             SWA project officials,   provider’s knowledge     Navy independent
                        summary. Any trends      these data are           management system        verification and
                        identified are           submitted to the         provides analysis        validation testing and
                        addressed in monthly     contracting officer’s    and trend data to        customer satisfaction
                        status meetings.         representative for       MHS.                     survey results. In
                                                 analysis.                                         addition, NMCI
                                                                                                   program officials said
                                                                                                   that they have
                                                                                                   requested funding to
                                                                                                   develop a
                                                                                                   performance
                                                                                                   measurement data
                                                                                                   repository to support
                                                                                                   trend analyses.
Allow employees and     Limited—The              Yes—The contracting      Yes—The customer         Yes—User surveys         Yes—The provider
possibly                provider conducts        officer’s                satisfaction survey is   are used to measure      surveys NIMA
stakeholders to rate    annual customer          representative and       one method used to       satisfaction with        employees on their
the provider on a       satisfaction surveys,    on-site points of        rate the provider. In    specific services        satisfaction with the
regular basis (e.g.,    but it is not required   contact obtain           addition, according to   such as E-mail           help-desk function.
scorecards and          to submit the results    feedback from            MHS program              access, help desk,       The results of these
quarterly report        to the Air Force.        employees on a           officials, stakeholder   etc., and overall user   surveys are reviewed
cards).                 Kirtland Air Force       regular basis.           input is also obtained   satisfaction with the    by NIMA during
                        Base also obtains ad                              from program             service provider’s       quarterly meetings.
                        hoc feedback from                                 management reviews       performance.             In addition,
                        employees who, after                              and meetings held by                              stakeholders are also
                        they report a                                     the government task                               responsible for
                        problem, are asked                                manager.                                          providing monthly
                        to provide                                                                                          assessments of
                        information about                                                                                   contractor
                        how well the problem                                                                                performance.
                        was addressed and                                                                                   Moreover, customer
                        their overall                                                                                       satisfaction is a
                        satisfaction level.                                                                                 factor in determining
                                                                                                                            the amount of the
                                                                                                                            semiannual award
                                                                                                                            fee earned by the
                                                                                                                            provider.
Source: GAO.




                                                  Page 85                                               GAO-03-371 DOD IT Services Outsourcing
Appendix VIII

Projects’ Implementation of Phase VII: Ensure
Services are Provided                                                                          Appendx
                                                                                                     iI
                                                                                                     V




                Although outsourcing focuses on the provider’s ability to perform, the
                client organization is ultimately responsible for ensuring that services are
                provided and that end-user needs are met. The previous phases addressed
                the extensive preparation that must precede the provider’s assuming
                responsibility for the client organization’s services. This phase
                encompasses practices designed to ensure that an effective oversight
                approach is established. In addition, over the course of the outsourcing
                project, the client learns more about the capabilities of the provider, and
                market conditions may change. As a result, it is important to monitor
                service levels internally as well as maintain an external view of the
                performance of other providers in order to identify opportunities to
                improve and ensure that the outsourcing arrangement maintains its value
                to the client.

                The six practices in this phase that we used to evaluate the five projects are
                as follows:

                • Monitor the provider’s work to anticipate issues for resolution.

                • Make sure that the provider uses the standard tools and processes
                  defined as part of the operational model.

                • Use provider performance data to continuously improve processes.

                • Pursue improvement based on customer satisfaction surveys.

                • Ensure that an appropriately empowered individual from the client
                  organization oversees the work.

                • Set realistic time frames that are agreed to by the provider.



                Page 86                                   GAO-03-371 DOD IT Services Outsourcing
Appendix VIII
Projects’ Implementation of Phase VII:
Ensure Services are Provided




As illustrated by figure 12, the five projects in our review largely
implemented the practices.



Figure 12: Project Implementation of Phase VII: Ensure Services Are Provided




Note: Not applicable—The practice was not relevant to the project’s particular circumstances.
No—The agency did not implement the practice. Limited—The agency fully implemented some but not
all aspects of the practice and did not take alternative actions that fully satisfied the practice. Yes—The
agency fully implemented the practice or took an alternative action that fully satisfied its intent.


Table 10 provides details on whether and how each project implemented
each of the six practices associated with this phase.




Page 87                                                   GAO-03-371 DOD IT Services Outsourcing
                                                  Appendix VIII
                                                  Projects’ Implementation of Phase VII:
                                                  Ensure Services are Provided




Table 10: Summary of Projects’ Use of Phase VII Practices

                                                                  Did the project use the practice?
                        Air Force C4
Practice                Services                 Army TAC-SWA              MHS/ITO Help Desk Navy NMCI                         NIMA IT/IS
Monitor the             Yes—Quality              Yes—The contracting       Yes—The provider’s        Yes—Navy                  Yes—NIMA reviews
provider’s work to      assurance evaluators     officer’s                 work is monitored         representatives at        the provider’s work in
anticipate issues for   monitor the quality of   representative            through monthly           each implemented          quarterly program
resolution.             the provider’s work      reports monthly on        reports and meetings      site (including           management reviews
                        and identify problems    contractor                as well as through        headquarters)             and as needed in
                        or trends. The results   performance. In           on-site meetings and      monitor the               response to issues
                        and any problems         addition, he meets        readiness                 provider’s work and       raised by NIMA’s
                        are reported to the      daily with provider       assessments.              identify issues.          performance
                        contracting officer      officials to discuss                                According to NMCI         monitoring officials.
                        and the functional       performance results.                                program officials, the
                        area chief.                                                                  NMCI Enterprise
                                                                                                     Management System
                                                                                                     enables the
                                                                                                     government to
                                                                                                     monitor the
                                                                                                     performance of the
                                                                                                     network and
                                                                                                     associated service
                                                                                                     delivery.
Make sure the           Yes—The provider         Yes—According to          Yes—According to          Yes—The Navy has          Yes—The tools are
provider uses the       uses software tools      TAC-SWA project           MHS program               monitored the             documented in the
standard tools and      to help monitor          officials, the provider   officials, the provider   service provider’s        quality management
processes defined as    system performance       uses various              uses the standard         use of standard tools     plan. For example,
part of the             against performance      standard tools            tools outlined in the     and processes. Tools      the provider is using
operational model.      standards to monitor     outlined in its           contract.                 are being used to         a specific quality and
                        and manage the           proposal.                                           support legacy            process
                        help-desk function.                                                          migration, client         improvement
                                                                                                     installation, and help-   methodology.
                                                                                                     desk procedures.
Use provider            Yes—Although             Yes—The contracting       Yes—For example,          Yes—The Navy uses         Yes—NIMA’s
performance data to     Kirtland Air Force       officer’s                 MHS used provider         the provider’s data to    performance
continuously improve    Base does not rely       representative            performance data to       help determine ways       monitors use
processes.              on data provided by      monitors provider         improve the               to improve                provider data to
                        its provider, its        performance data,         customer satisfaction     processes, such as        assess progress and
                        quality assurance        and any potential         survey process.           to improve NMCI           ways to improve
                        evaluators               areas for                                           implementation            performance.
                        continually monitor      improvement are                                     procedures and the
                        provider                 incorporated in the                                 timeliness of help-
                        performance.             monthly report.                                     desk problem
                                                                                                     resolution.




                                                  Page 88                                                 GAO-03-371 DOD IT Services Outsourcing
                                                 Appendix VIII
                                                 Projects’ Implementation of Phase VII:
                                                 Ensure Services are Provided




(Continued From Previous Page)
                                                                Did the project use the practice?
                       Air Force C4
Practice               Services                 Army TAC-SWA             MHS/ITO Help Desk Navy NMCI                      NIMA IT/IS
Pursue improvement Limited—The Air              Yes—Surveys are          Yes—Negative            Yes—Surveys of           Yes—To pursue
based on customer     Force does not            received by the          feedback that is        user satisfaction with   improvement, NIMA
satisfaction surveys. perform, or require       contracting officer’s    received based on       specific services and    uses both customer
                      the provider to           technical                the results of the      overall satisfaction     satisfaction surveys
                      perform, customer         representatives and      customer satisfaction   with provider            and interviews with
                      surveys. However,         provided as input for    surveys is reviewed     performance are          senior-level
                      the provider has          the contracting          by the MHS help-        used to identify areas   customers.
                      initiated such            officer’s                desk manager.           needing
                      surveys but is not        representative                                   improvement.
                      required to distribute    report.
                      the results to the Air
                      Force.
Ensure that an         Yes—The functional       Yes—The contracting      Yes—The                 Yes—The NMCI             Yes—The contracting
appropriately          area chief oversees      officer’s                government task         Director oversees the    officer’s
empowered              the work, and the        representative is the    manager is the          work and discusses       representative and a
individual from the    quality assurance        empowered                individual              performance with the     staff of technical
client organization    evaluators support       individual from          empowered to            NMCI provider            monitors oversee the
oversees the work.     this oversight effort.   NETCOM.                  oversee the work of     executive during         work.
                                                                         the provider.           weekly meetings.
Set realistic time     Yes—Kirtland Air         Yes—Time frames          Yes—The monthly         Yes—According to         Yes—NIMA and the
frames that are        Force Base and the       for new tasks are        performance reports     NMCI program             provider have agreed
agreed to by the       provider agree that      established by           and quarterly           officials, Navy’s        on dates associated
provider.              reasonable time          contract                 program reviews set     practice is to work      with specific actions.
                       frames for               modifications that are   time frames that        with the provider in
                       performance have         agreed to by the         have been agreed to     establishing
                       now been set. Some       provider.                by the provider.        schedules.
                       revisions were made
                       to them in April 2002.
Source: GAO




                                                 Page 89                                              GAO-03-371 DOD IT Services Outsourcing
Appendix IX

Comments from the Department of Defense                                 Appendx
                                                                              IiX




Note: GAO comments
supplementing those in
the report text appear
at the end of this
appendix.




See comment 1.




See comment 2.




                         Page 90   GAO-03-371 DOD IT Services Outsourcing
Appendix IX
Comments from the Department of Defense




Page 91                                   GAO-03-371 DOD IT Services Outsourcing
                 Appendix IX
                 Comments from the Department of Defense




See comment 3.




                 Page 92                                   GAO-03-371 DOD IT Services Outsourcing
               Appendix IX
               Comments from the Department of Defense




               The following are GAO’s comments on the Department of Defense’s letter
               dated April 8, 2003.



GAO Comments   1. Addressed in the Agency Comments and Our Evaluation section of this
                  report.

               2. On January 29, 2003, NIMA granted the provider authorization to use
                  government sources in performing the IT/IS contract that was limited
                  to “products, services, and supplies that support the performance of the
                  specific functional areas and miscellaneous items required under this
                  contract.” Accordingly, we did not modify this report.

               3. Addressed in the Agency Comments and Our Evaluation section of this
                  report.




               Page 93                                   GAO-03-371 DOD IT Services Outsourcing
Appendix X

GAO Contact and Staff Acknowledgments                                                         Append
                                                                                                   x
                                                                                                   i
                                                                                                   X




GAO Contact       Linda J. Lambert, (202) 512-9556




Staff             Season Dietrich, James Houtz, Anjalique Lawrence, Patricia Slocum, and
                  Thomas Wright made key contributions to this report.
Acknowledgments




(310239)          Page 94                                GAO-03-371 DOD IT Services Outsourcing
GAO’s Mission            The General Accounting Office, the audit, evaluation and investigative arm of
                         Congress, exists to support Congress in meeting its constitutional responsibilities
                         and to help improve the performance and accountability of the federal government
                         for the American people. GAO examines the use of public funds; evaluates federal
                         programs and policies; and provides analyses, recommendations, and other
                         assistance to help Congress make informed oversight, policy, and funding
                         decisions. GAO’s commitment to good government is reflected in its core values of
                         accountability, integrity, and reliability.


Obtaining Copies of      The fastest and easiest way to obtain copies of GAO documents at no cost is
                         through the Internet. GAO’s Web site (www.gao.gov) contains abstracts and full-
GAO Reports and          text files of current reports and testimony and an expanding archive of older
                         products. The Web site features a search engine to help you locate documents
Testimony                using key words and phrases. You can print these documents in their entirety,
                         including charts and other graphics.
                         Each day, GAO issues a list of newly released reports, testimony, and
                         correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site
                         daily. The list contains links to the full-text document files. To have GAO e-mail this
                         list to you every afternoon, go to www.gao.gov and select “Subscribe to GAO
                         Mailing Lists” under “Order GAO Products” heading.


Order by Mail or Phone   The first copy of each printed report is free. Additional copies are $2 each. A check
                         or money order should be made out to the Superintendent of Documents. GAO
                         also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a single
                         address are discounted 25 percent. Orders should be sent to:
                         U.S. General Accounting Office
                         441 G Street NW, Room LM
                         Washington, D.C. 20548
                         To order by Phone:     Voice: (202) 512-6000
                                                TDD: (202) 512-2537
                                                Fax: (202) 512-6061


To Report Fraud,         Contact:
                         Web site: www.gao.gov/fraudnet/fraudnet.htm
Waste, and Abuse in      E-mail: fraudnet@gao.gov
Federal Programs         Automated answering system: (800) 424-5454 or (202) 512-7470



Public Affairs           Jeff Nelligan, Managing Director, NelliganJ@gao.gov (202) 512-4800
                         U.S. General Accounting Office, 441 G Street NW, Room 7149
                         Washington, D.C. 20548
United States                  Presorted Standard
General Accounting Office      Postage & Fees Paid
Washington, D.C. 20548-0001           GAO
                                 Permit No. GI00
Official Business
Penalty for Private Use $300
Address Service Requested