oversight

DOD Business Systems Modernization: Improvements to Enterprise Architecture Development and Implementation Efforts Needed

Published by the Government Accountability Office on 2003-02-28.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                United States General Accounting Office

GAO             Report to the Chairman and Ranking
                Minority Member, Subcommittee on
                Readiness and Management Support,
                Committee on Armed Services, U.S.
                Senate
February 2003
                DOD BUSINESS
                SYSTEMS
                MODERNIZATION
                Improvements to
                Enterprise
                Architecture
                Development and
                Implementation
                Efforts Needed




GAO-03-458
                a
                                               February 2003


                                               DOD BUSINESS SYSTEMS
                                               MODERNIZATION

Highlights of GAO-03-458, a report to the      Improvements to Enterprise Architecture
Chairman and Ranking Minority Member,
Subcommittee on Readiness and                  Development and Implementation Efforts
Management Support, Committee on
Armed Services, U.S. Senate                    Needed


The Department of Defense (DOD)                DOD has undertaken a challenging and ambitious task to, within 1 year,
is developing an enterprise                    develop a departmentwide blueprint for modernizing its over 1,700 time-
architecture, or corporate                     worn, inefficient, and nonintegrated business processes and supporting
modernization blueprint, to guide              information technology (IT) assets. Such a blueprint, commonly called an
and constrain its ongoing and                  enterprise architecture, is an essential modernization management tool. We
planned business system
investments. GAO was asked to
                                               support the Secretary of Defense’s decision to develop an architecture and
review DOD’s processes and                     the department’s goal of acquiring systems that provide timely, reliable, and
controls for developing the                    relevant information.
enterprise architecture and
ensuring that ongoing IT                       Successfully doing so requires the application of effective enterprise
investments are consistent with its            architecture and IT investment management processes and controls. While
enterprise architecture                        DOD is following some of these enterprise architecture practices, it is not
development efforts.                           following others, in part because it is focused on meeting its ambitious
                                               schedule. More specifically, with respect to developing the architecture,
                                               DOD has yet to (1) establish the requisite architecture development
                                               governance structure and process controls needed to ensure that ownership
To assist DOD in successfully                  of and accountability for the architecture are vested with senior leaders
developing an enterprise                       across the department, (2) clearly communicate to intended architecture
architecture and using it to gain              stakeholders the purpose, scope, and approach to developing the initial and
control over its ongoing business              subsequent versions of the architecture, and their roles and responsibilities,
system investments, we are making              and (3) define and implement an independent quality assurance process.
recommendations to the Secretary               Until it follows these practices, DOD increases the risk of developing an
of Defense to ensure that DOD (1)              architecture that will be limited in scope, be resisted by those responsible
expands its use of effective                   for implementing it, and will not support effective systems modernization.
architecture development
processes and controls and (2)
                                               DOD has taken initial steps aimed at improving its management of ongoing
strengthens controls over its
ongoing business systems                       business system investments. However, DOD has yet to establish the
investments.                                   necessary departmental investment governance structure and process
                                               controls needed to adequately align ongoing investments with its
DOD concurred with our                         architectural goals and direction. Instead, DOD continues to allow its
recommendations and described                  component organizations to make their own parochial investment decisions,
recently completed, ongoing, and               following different approaches and criteria. This stovepiped decision-making
planned efforts to address them.               process has contributed to the department’s current complex, error-prone
                                               environment of over 1,700 systems. In particular, DOD has not established
                                               and applied common investment criteria to its ongoing IT system projects
                                               using a hierarchy of investment review and funding decision-making bodies,
                                               each composed of representatives from across the department. DOD also
                                               has not yet conducted a comprehensive review of its ongoing IT investments
                                               to ensure that they are consistent with its architecture development efforts.
                                               Until it takes these steps, DOD will likely continue to lack effective control
                                               over the billions of dollars it is currently spending on IT projects.
www.gao.gov/cgi-bin/getrpt?GAO-03-458
To view the full report, including the scope
and methodology, click on the link above.
For more information, contact Gregory Kutz,
(202) 512-9095 (kutzg@gao.gov) or
Randolph Hite, (202) 512-3439
(hiter@gao.gov).
Contents



Letter                                                                                                                   1
                             Recommendations for Executive Action                                                        4
                             Agency Comments and Our Evaluation                                                          5


Appendixes
              Appendix I:    Briefing to Subcommittee Staff                                                              7
             Appendix II:    Comments from the Under Secretary of Defense                                               60
             Appendix III:   GAO Contacts and Staff Acknowledgments                                                     64
                             GAO Contacts                                                                               64
                             Acknowledgments                                                                            64




                              This is a work of the U.S. Government and is not subject to copyright protection in the
                              United States. It may be reproduced and distributed in its entirety without further
                              permission from GAO. It may contain copyrighted graphics, images or other materials.
                              Permission from the copyright holder may be necessary should you wish to reproduce
                              copyrighted materials separately from GAO’s product.




                             Page i                    GAO-03-458 DOD Architecture Development and Implementation
A
United States General Accounting Office
Washington, D.C. 20548



                                    February 28, 2003                                                                       Leter




                                    The Honorable John Ensign
                                    Chairman
                                    The Honorable Daniel K. Akaka
                                    Ranking Minority Member
                                    Subcommittee on Readiness and Management Support
                                    Committee on Armed Services
                                    United States Senate

                                    In May 2001,1 we reported that the Department of Defense (DOD) had
                                    neither an enterprise architecture for its financial and financial-related
                                    business operations, nor the management structure, processes, and
                                    controls in place to effectively develop and implement one. In September
                                    2002, the Secretary of Defense designated improving financial management
                                    operations, which include not only finance and accounting but also
                                    business areas such as logistics, acquisition, and personnel management,
                                    as 1 of the department’s top 10 priorities. In addition, the Secretary
                                    established a program to develop and implement an enterprise
                                    architecture. In response to your request, we determined whether DOD is
                                    (1) following effective processes and controls in developing its enterprise
                                    architecture and (2) ensuring that ongoing information technology (IT)
                                    investments are consistent with its enterprise architecture development
                                    efforts.

                                    On January 31, 2003, we briefed your offices on the results of this review
                                    and the recommendations we are making to the Secretary of Defense. This
                                    report transmits those briefing materials, including our scope and
                                    methodology, as appendix I.

                                    DOD has undertaken a challenging and ambitious task to, within 1 year,
                                    develop a departmentwide enterprise architecture (blueprint) for
                                    modernizing its business operations and systems. We support DOD’s goals
                                    of developing an architecture to guide and constrain its modernization
                                    efforts and acquiring systems that provide timely, reliable, and relevant
                                    information. Toward these goals, DOD has taken a number of positive
                                    steps, including


                                    1
                                     U.S. General Accounting Office, Information Technology: Architecture Needed to Guide
                                    Modernization of DOD’s Financial Operations, GAO-01-525 (Washington, D.C.: May 17,
                                    2001).




                                    Page 1                    GAO-03-458 DOD Architecture Development and Implementation
                           • designating improving financial management operations as 1 of its top
                             10 priorities;

                           • establishing a program office responsible for managing the enterprise
                             architecture development effort;

                           • capturing key data needed to develop the “As Is” architecture, such as
                             documenting its inventory of over 1,700 business systems; and

                           • requiring DOD Comptroller review and approval of IT investments that
                             meet certain criteria.

                           Our May 20012 report provided a number of fundamental steps on how
                           DOD should approach the development of its enterprise architecture. At
                           that time, we had also recommended that the department limit business
                           system investments until the enterprise architecture is developed. While
                           DOD has taken some positive actions, as specified above, the department
                           has yet to implement some of our recommendations and certain best
                           practices for developing and implementing the architecture. The following
                           discussion summarizes those key practices DOD has yet to employ.



Architecture Development   Successful architecture development requires the application of proven
                           management practices. Thus far, DOD has not implemented certain
                           practices. Specifically, DOD has yet to

                           • establish the requisite architecture development governance structure
                             needed to ensure that ownership of and accountability for the
                             architecture is vested with senior leaders across the department;

                           • develop and implement a strategy to effectively communicate the
                             purpose and scope, approach to, and roles and responsibilities of
                             stakeholders in developing the enterprise architecture; and

                           • fully define and implement an independent quality assurance process.

                           Not implementing these practices increases DOD’s risk of developing an
                           architecture that will be limited in scope, be resisted by those responsible


                           2
                           GAO-01-525.




                           Page 2                 GAO-03-458 DOD Architecture Development and Implementation
                             for implementing it, and will not support effective systems modernization.
                             DOD recognizes the need to follow these practices, and attributes its delays
                             in doing so to tight schedule demands, unawareness of certain best
                             practices, and competing resource priorities. Among other things, it plans
                             to strengthen the architecture governance and management structure.



Architecture                 Ensuring that ongoing IT investments are consistent with DOD’s
Implementation and Control   architecture development efforts requires the application of proven
                             investment management practices. To date, DOD has not implemented
of Ongoing Investments       certain practices. Specifically, DOD has yet to establish an investment
                             management governance structure that includes

                             • a hierarchy of investment review boards composed of representatives
                               from across the department who are assigned investment selection and
                               control responsibilities based on project threshold criteria;

                             • a standard set of investment review and decision-making criteria for use
                               by all boards, including criteria to ensure architectural compliance and
                               consistency; and

                             • a specified, near-term date by which ongoing investments have to be
                               subjected to this investment review process, and by which decisions
                               should be made as to whether to proceed with each investment.

                             Until the investment management governance structure is established,
                             DOD component organizations will continue to make their own parochial
                             investment decisions, following different approaches and criteria. As we
                             have previously reported,3 this stovepiped decision-making process has
                             contributed to the department’s current complex, error-prone systems
                             environment. This deeply embedded cultural resistance to a more holistic
                             decision-making process is a substantial risk to successful development
                             and implementation of the enterprise architecture. DOD’s leadership plans
                             to strengthen its governance and oversight over ongoing IT investments.




                             3
                              U.S. General Accounting Office, DOD Financial Management: Important Steps Underway
                             But Reform Will Require a Long-term Commitment, GAO-02-784T (Washington, D.C.: June
                             4, 2002).




                             Page 3                  GAO-03-458 DOD Architecture Development and Implementation
Recommendations for   To assist DOD in its efforts to effectively develop and implement an
                      enterprise architecture, and guide and constrain its business system
Executive Action      investments, and to address the problems discussed during the briefing, we
                      reiterate the recommendations that we made in our May 2001 report4 that
                      DOD has yet to implement. In addition, we recommend that the Secretary
                      of Defense ensure that

                      • the enterprise architecture executive committee members are singularly
                        and collectively made explicitly accountable to the Secretary for
                        delivery of the enterprise architecture, including approval of each
                        version of the architecture;

                      • the enterprise architecture program is supported by a proactive
                        marketing and communication program; and

                      • the quality assurance function (1) includes the review of adherence to
                        process standards and reliability of reported program performance, (2)
                        is made independent of the program management function, and (3) is
                        not performed by subject matter experts involved in the development of
                        key architecture products.

                      Additionally, we recommend that the Secretary gain control over ongoing
                      IT investments by

                      • establishing a hierarchy of investment review boards, each responsible
                        and accountable for selecting and controlling investments that meet
                        defined threshold criteria, and each composed of the appropriate level
                        of executive representatives, depending on the threshold criteria, from
                        across the department;

                      • establishing a standard set of criteria to include (1) alignment and
                        consistency with the DOD enterprise architecture and (2) our open
                        recommendations governing limitations in business system investments
                        pending development of the architecture; and

                      • directing these boards to immediately apply these criteria in completing
                        reviews of all ongoing IT investments, and to not fund investments that



                      4
                      GAO-01-525.




                      Page 4                GAO-03-458 DOD Architecture Development and Implementation
                         do not meet these criteria unless they are otherwise justified by explicit
                         criteria waivers.



Agency Comments and   In written comments on a draft of this report (see appendix II), the Under
                      Secretary of Defense (Comptroller) stated that the department concurred
Our Evaluation        with our recommendations and described recently completed, ongoing,
                      and planned efforts to address them. For example, the department stated
                      that it is currently developing a new architecture development and
                      implementation governance structure and a marketing and communication
                      strategy to facilitate ongoing development activities. Additionally, the
                      department stated that it is providing for the independence of its quality
                      assurance function by organizationally moving it under the Director of
                      Business Modernization and Systems Integration. DOD stated that it would
                      also require the reporting of quality assurance information to the
                      architecture Executive Steering Committee. We did not verify or evaluate
                      the extent to which the efforts described in DOD's comments will address
                      our recommendation.

                      Regarding the scope of quality assurance reviews, the department stated
                      that it has established and implemented a quality assurance function that
                      includes review of architecture products, program performance, and
                      architecture development process standards. We agree that the quality
                      assurance function includes review of architecture products. However,
                      neither during our review nor in its comments did the department provide
                      documentary evidence to support that its quality assurance reviews
                      address program performance and adherence to architecture development
                      process standards. Further, as stated in our report, quality assurance
                      function officials told us that they were never tasked to perform such
                      reviews.


                      We are sending copies of this report to the Chairmen and Ranking Minority
                      Members of other Senate and House committees and subcommittees that
                      have jurisdiction and oversight responsibilities for the Department of
                      Defense. We are also sending copies to the Secretary of Defense; the
                      Secretary of the Army; the Secretary of the Navy; the Secretary of the Air
                      Force; the Under Secretary of Defense (Comptroller); the Under Secretary
                      of Defense (Acquisition, Technology, and Logistics); the Under Secretary of
                      Defense (Personnel & Readiness); the Assistant Secretary of Defense
                      (Command, Control, Communications, and Intelligence); the Director of



                      Page 5                 GAO-03-458 DOD Architecture Development and Implementation
the Defense Finance and Accounting Service; and the Director of the Office
of Management and Budget. Copies will also be available at no charge on
our Web site at www.gao.gov.

Should you or your staff have any questions on matters discussed in this
report, please contact us at (202) 512-9095 or (202) 512-3439, respectively.
We can also be reached by e-mail at kutzg@gao.gov or hiter@gao.gov. GAO
contacts and key contributors to this report are listed in appendix III.




Gregory D. Kutz
Director, Financial Management and Assurance




Randolph C. Hite
Director, Information Technology Architecture
and System Issues




Page 6                GAO-03-458 DOD Architecture Development and Implementation
Appendix I

Briefing to Subcommittee Staff                                                                              AA
                                                                                                             ppp
                                                                                                               ep
                                                                                                                ned
                                                                                                                  n
                                                                                                                  x
                                                                                                                  id
                                                                                                                   e
                                                                                                                   x
                                                                                                                   Iis




             DOD BUSINESS SYSTEMS MODERNIZATION: Improvements to
             Enterprise Architecture Development and Implementation Efforts
             Needed



                              Briefing for the Staff of the
                   Subcommittee on Readiness and Management Support,
                           Senate Armed Services Committee


                                        January 31, 2003




                                                                                            1




                               Page 7              GAO-03-458 DOD Architecture Development and Implementation
                    Appendix I
                    Briefing to Subcommittee Staff




                                                            Outline of Briefing


•   Introduction
•   Objectives
•   Scope and Methodology
•   Results in Brief
•   Background
•   Results
•   Conclusions
•   Recommendations
•   Agency Comments




                                                                                      2




                    Page 8                    GAO-03-458 DOD Architecture Development and Implementation
                      Appendix I
                      Briefing to Subcommittee Staff




                                                                       Introduction


In September 2002, the Secretary of Defense designated improving
financial management operations, which includes not only finance and
accounting but also such business areas as logistics, acquisition, and
personnel management, as 1 of the department’s top 10 priorities.

Effectively managing such a large and complex endeavor requires, among
other things, a well-defined and enforced blueprint for operational and
technological change, commonly referred to as an enterprise architecture.
The Department of Defense (DOD) is in the process of developing an
enterprise architecture for its business systems modernization efforts. At
the same time, it continues to invest billions of dollars in existing and new
systems.

The use of enterprise architectures is an information technology (IT)
management best practice. Our experience with federal agencies has
shown that attempting a major modernization effort without a well-defined
and enforceable enterprise architecture results in systems that are
duplicative, are not well integrated, are unnecessarily costly to maintain
and interface, and do not effectively optimize mission performance.
                                                                                        3




                      Page 9                    GAO-03-458 DOD Architecture Development and Implementation
                    Appendix I
                    Briefing to Subcommittee Staff




                                                                        Objectives


As agreed, our objectives were to determine whether DOD is

   • following effective processes and controls in developing its
     enterprise architecture and
   • ensuring that ongoing IT investments are consistent with its
     enterprise architecture development efforts.




                                                                                      4




                    Page 10                   GAO-03-458 DOD Architecture Development and Implementation
                    Appendix I
                    Briefing to Subcommittee Staff




                                                     Scope and Methodology

To accomplish our objectives, we

   • used relevant federal guidance and best practices to define the
     controls and processes needed for effective architecture
     development and implementation (see app. I);

   • reviewed DOD documentation associated with developing the
     architecture, including steering committees’ charters, contractor
     work orders and deliverables (e.g., development methodology and
     program management plan), risk management and quality assurance
     process definitions and results, and ongoing system review processes
     and results;

   • attended various DOD architecture development program meetings
     and workshops to determine, among other things, the program’s
     scope, strategy, methodology, stakeholder understanding and
     commitment, and progress;
                                                                                      5




                    Page 11                   GAO-03-458 DOD Architecture Development and Implementation
                 Appendix I
                 Briefing to Subcommittee Staff




                                       Scope and Methodology (cont.)


• compared the architecture development and implementation
  management processes and controls to applicable criteria, including
  recognized best practices;

• reviewed DOD’s and the military services’ policies and procedures to
  obtain an understanding of their processes for developing and
  reviewing their IT budgets and selecting and controlling their IT
  investments; and

• reviewed and analyzed DOD’s fiscal year 2003 IT budget request to
  determine the amount of IT funding requested for selected systems.




                                                                                   6




                 Page 12                   GAO-03-458 DOD Architecture Development and Implementation
                 Appendix I
                 Briefing to Subcommittee Staff




                                       Scope and Methodology (cont.)


• To augment our document reviews and analyses, we interviewed
  officials from various DOD organizations and contractors, including
   • the Office of the Under Secretary of Defense (Comptroller), the
     Financial Management Modernization Program Management
     Office, and the Financial Management Modernization Program
     Support Office;
   • the Office of the Assistant Secretary of Defense for Command,
     Control, Communications and Intelligence/Chief Information
     Officer (CIO);
   • the Offices of the Army, Navy, and Air Force Assistant Secretaries
     for Financial Management and Comptrollers;
   • the Offices of the Army, Navy, and Air Force CIOs;
   • International Business Machines (IBM); and
   • MITRE Corporation.



                                                                                   7




                 Page 13                   GAO-03-458 DOD Architecture Development and Implementation
                     Appendix I
                     Briefing to Subcommittee Staff




                                           Scope and Methodology (cont.)


We did not independently validate cost and budget information provided by
DOD, or the completeness or accuracy of draft architecture products.

We conducted our work primarily at DOD headquarters offices in
Washington, D.C., and Arlington, Virginia.

The work was performed from July 2002 through January 2003 in
accordance with U.S. generally accepted government auditing standards.




                                                                                       8




                     Page 14                   GAO-03-458 DOD Architecture Development and Implementation
                              Appendix I
                              Briefing to Subcommittee Staff




                                                                                Results in Brief

DOD has undertaken a challenging and ambitious task to, within 1 year,
develop a departmentwide enterprise architecture, or blueprint, for
modernizing its business operations and systems 1. We support DOD’s
goals of developing an architecture to guide and constrain its modernization
efforts and acquiring systems that provide timely, reliable, and relevant
information. Toward these goals, DOD has taken a number of positive
steps, including

 • designating improving financial management operations as 1 of its top
   10 priorities;
 • establishing a program office responsible for managing the enterprise
   architecture development effort;
 • capturing key data needed to develop the “As Is” architecture, such as
   documenting its inventory of over 1,700 business systems; and
 • requiring DOD Comptroller review and approval of IT investments that
   meet certain criteria.
1Section1004 of Public Law 107-314, December 2, 2002, recently required the Secretary of Defense to
develop a financial management enterprise architecture that meets certain minimum requirements by May 1,
2003.
                                                                                                           9




                              Page 15                     GAO-03-458 DOD Architecture Development and Implementation
                      Appendix I
                      Briefing to Subcommittee Staff




                                                       Results in Brief (cont.)

While DOD has taken positive steps and is continuing to take further
actions to address some of our concerns, the department is still not meeting
certain critical best practices for developing and implementing the
architecture.

Architecture Development
Successful architecture development requires the application of proven
management practices. However, DOD has yet to

 • establish the requisite architecture development governance structure
   needed to ensure that ownership of and accountability for the
   architecture is vested with senior leaders across the department;
 • develop and implement a strategy to effectively communicate the
   purpose and scope, approach to, and roles and responsibilities of
   stakeholders in developing the enterprise architecture; and
 • fully define and implement an independent quality assurance process.

                                                                                       10




                      Page 16                   GAO-03-458 DOD Architecture Development and Implementation
                      Appendix I
                      Briefing to Subcommittee Staff




                                                        Results in Brief (cont.)

DOD recognizes the need to follow these practices, and attributes its delays
in doing so to tight schedule demands, unawareness of certain best
practices, and competing resource priorities. Among other things, it plans to
strengthen the architecture governance and management structure.

However, until each of these practices is followed, DOD has increased the
risk of developing an architecture that will be limited in scope, will be
resisted by those responsible for implementing it, and thus will not support
effective systems modernization.

Architecture Implementation and Control of Ongoing Investments
Ensuring that ongoing IT investments are consistent with DOD’s
architecture development efforts requires the application of proven
investment management practices. However, DOD has yet to establish an
investment management governance structure that includes
    • a hierarchy of investment review boards composed of
       representatives from across the department who are assigned
       investment selection and control responsibilities based on project
       threshold criteria;

                                                                                       11




                      Page 17                   GAO-03-458 DOD Architecture Development and Implementation
                       Appendix I
                       Briefing to Subcommittee Staff




                                                         Results in Brief (cont.)

   • a standard set of investment review and decision-making criteria for
     use by all boards, including criteria to ensure architectural
     compliance and consistency; and
   • a specified, near-term date by which ongoing investments have to be
     subjected to this investment review process, and by which decisions
     should be made as to whether to proceed with each investment.

Instead, DOD continues to allow its component organizations to
independently make their own investment decisions, following different
approaches and criteria. As we have previously reported,2 this stovepiped
decision-making process has contributed to the department’s complex,
error-prone systems environment. This deeply embedded cultural
resistance to a more holistic decision-making process is a substantial risk
to successful development and implementation of the enterprise
architecture. Its leadership plans to strengthen its governance and
oversight of its ongoing IT investments.
2U.S. General Accounting Office, DOD Financial Management: Important Steps Underway
But Reform Will Require a Long-term Commitment, GAO-02-784T (Washington, D.C.: June
4, 2002).

                                                                                        12




                       Page 18                   GAO-03-458 DOD Architecture Development and Implementation
                     Appendix I
                     Briefing to Subcommittee Staff




                                                       Results in Brief (cont.)

Until these practices are followed, DOD will likely continue to invest in
business systems that are duplicative, not integrated, and overly costly and
do not optimally support mission operations.




                                                                                      13




                     Page 19                   GAO-03-458 DOD Architecture Development and Implementation
                     Appendix I
                     Briefing to Subcommittee Staff




                                                       Results in Brief (cont.)

Recommendations
To assist DOD in successfully developing an enterprise architecture and
using it to gain control over its ongoing business system investments, we
are making recommendations to the Secretary of Defense to ensure that
DOD (1) expands its use of effective architecture development processes
and controls, and (2) strengthens controls over its ongoing business
systems investments.

In commenting on a draft of this briefing, DOD officials stated that they
generally agreed with our findings, conclusions and recommendations,
with the exception of our assessment of the adequacy of its quality
assurance function. With regard to quality assurance, they stated that this
function currently addresses adherence to process standards and reported
program performance. However, they did not provide evidence that
supported this statement.




                                                                                      14




                     Page 20                   GAO-03-458 DOD Architecture Development and Implementation
                             Appendix I
                             Briefing to Subcommittee Staff




                                                                                   Background

We have long reported that DOD’s serious financial management and
related business system problems result in a lack of information needed to
make sound decisions, and that these problems hinder the efficiency of
operations and leaves the department vulnerable to fraud, waste, and abuse.
Such problems led us in 1995 to put both DOD financial management and
systems modernization on our list of high-risk areas in the federal
government, a designation that continues today.3

To assist DOD in addressing these high risk areas, the DOD Inspector
General, the military service audit agencies, and GAO have conducted
numerous audits and reviews and made scores of recommendations that
are aimed at correcting the root causes of its problems. The department has
acknowledged that its present financial management environment has
serious inadequacies and does not, for the most part, comply with the
framework for financial reform set out by Congress, such as the Chief
Financial Officers Act of 1990.
3U.S.General Accounting Office, High-Risk Series: An Overview, GAO/HR-95-1 (Washington, D.C.: February
1995); High-Risk Series: Defense Financial Management, GAO/HR-97-3 (Washington, D.C.: February 1997);
High-Risk Series: An Update, GAO/01-263 (Washington, D.C.: January 2001); and Performance and
Accountability Series: Major Management Challenges and Program Risks – Department of Defense,
GAO/03-98 (Washington, D.C.: January 2003).

                                                                                                    15




                             Page 21                    GAO-03-458 DOD Architecture Development and Implementation
                         Appendix I
                         Briefing to Subcommittee Staff




                                                                Background (cont.)

Recent GAO Reviews Identified Need for an Enterprise Architecture
In May 2001, we reported4 that the department had neither an enterprise
architecture for its financial and financial-related business operations, nor
the management structure, processes, and controls in place to effectively
develop and implement one. We also reported that the department planned
to spend billions of dollars on new and modified business systems that
would function independently from one another and outside the context of
an enterprise architecture.

We concluded that if the department continued down this path, it would
only perpetuate its existing business operations and systems environment,
which was
    • duplicative,
    • not interoperable,
    • unnecessarily costly to maintain and interface, and
    • not optimizing mission performance and accountability.
4U.S. General Accounting Office, Information Technology: Architecture Needed to Guide
Modernization of DOD’s Financial Operations, GAO-01-525 (Washington, D.C.: May 17, 2001).

                                                                                          16




                         Page 22                   GAO-03-458 DOD Architecture Development and Implementation
                     Appendix I
                     Briefing to Subcommittee Staff




                                                            Background (cont.)

To address its problems, we recommended that the Secretary of Defense,
among other things,
    • immediately designate the modernization program a departmental
      priority and that the Deputy Secretary of Defense lead this effort;
    • issue a policy that directed the development, implementation, and
      maintenance of an enterprise architecture;
    • empower an organization consisting of senior management from
      across the department with the authority to develop and maintain the
      architecture and serve as the department’s investment review board,
      and hold it accountable for both;
    • appoint a Chief Architect and establish and adequately staff and fund
      a program office to develop and maintain the architecture; and
    • ensure that the Chief Architect
       • establishes architecture management processes and controls,
       • defines and implements the architecture development approach
         and methodology,
       • develops the baseline architecture, the target architecture, and
         the sequencing plan, and
       • maintains the architecture.

                                                                                      17




                     Page 23                   GAO-03-458 DOD Architecture Development and Implementation
                     Appendix I
                     Briefing to Subcommittee Staff




                                                            Background (cont.)

We further recommended that, until the enterprise architecture is
developed and a departmentwide investment management review process
is established, DOD components limit business system investments to

   • deployment of systems that have already been fully tested and
     involve no additional development or acquisition cost,
   • stay-in-business maintenance needed to keep existing systems
     operational,
   • management controls needed to effectively invest in modernized
     systems, and
   • new systems or existing system changes that are congressionally
     directed or are relatively small, cost-effective, and low risk and can
     be delivered in a relatively short time frame.




                                                                                      18




                     Page 24                   GAO-03-458 DOD Architecture Development and Implementation
                         Appendix I
                         Briefing to Subcommittee Staff




                                                                Background (cont.)

In addition, we testified in March 2002,5 that despite well-intentioned prior
efforts to reform financial management operations, the department has
failed largely because of
     • a lack of sustained top-level leadership and management
        accountability for correcting problems;
     • deeply embedded cultural resistance to change, including military
        service parochialism and stovepiped operations;
     • a lack of results-oriented goals and performance measures and
        monitoring; and
     • inadequate incentives for seeking change.

We concluded that our experience has shown that well-intentioned
initiatives will only succeed if there are the right incentives,


5U.S.General Accounting Office, DOD Financial Management: Integrated Approach,
Accountability, Transparency, and Incentives Are Keys to Effective Reform, GAO-02-497T
(Washington, D.C.: Mar. 6, 2002).


                                                                                          19




                         Page 25                   GAO-03-458 DOD Architecture Development and Implementation
                     Appendix I
                     Briefing to Subcommittee Staff




                                                            Background (cont.)

transparency, and accountability and that for DOD, this means having,
among other things,

    • the direct, active support and involvement of the Secretary of
      Defense and
    • an enterprisewide architecture to guide and direct modernization
      investments.




                                                                                      20




                     Page 26                   GAO-03-458 DOD Architecture Development and Implementation
                      Appendix I
                      Briefing to Subcommittee Staff




                                                            Background (cont.)

What Is an Enterprise Architecture?
An enterprise architecture provides a clear and comprehensive picture of
an entity, whether it is an organization (e.g., federal department, military
service, or agency) or a functional or mission area that cuts across more
than one organization (e.g., financial management or homeland security).
This picture consists of snapshots of the enterprise’s current operational
and technological environment and its target environment, as well as a
capital investment road map for transitioning from the current to the target
environment.

Enterprise architecture development, implementation, and maintenance are
basic tenets of effective IT management. Managed properly, these
architectures can clarify and help optimize the interdependencies and
interrelationships among an organization’s business operations and the
underlying IT infrastructure and applications that support these operations.


                                                                                       21




                      Page 27                   GAO-03-458 DOD Architecture Development and Implementation
                     Appendix I
                     Briefing to Subcommittee Staff




                                                           Background (cont.)

Employed in concert with other important IT management controls, such as
institutional investment management practices, enterprise architectures can
greatly increase the chances that organizations’ operational and IT
environments will be configured in such a way as to optimize mission
performance.

History and Status of DOD Enterprise Architecture Efforts
Following our May 2001 report,6 the Secretary directed the development
and implementation of a departmentwide enterprise architecture, and
established a program to accomplish this. In doing so, the Secretary
assigned responsibility for the program to the DOD Comptroller, in
coordination with the Under Secretary of Defense (Acquisition, Technology,
and Logistics) and the DOD CIO.


6GAO-01-525.




                                                                                      22




                     Page 28                   GAO-03-458 DOD Architecture Development and Implementation
                     Appendix I
                     Briefing to Subcommittee Staff




                                                            Background (cont.)

In October 2001, the DOD Comptroller established the Financial
Management Modernization Executive Committee to oversee the
architecture and the systems modernization efforts, and the Financial
Management Modernization Steering Committee to advise and guide the
program.

In April 2002, DOD awarded a contract to IBM for approximately $95
million to begin developing the architecture.

DOD has divided its architecture development into two phases.
  Phase I (April 2002 - November 2002) focused on
      • documenting the current or “As Is” architecture and
      • developing what DOD termed a “strawman” architecture, which
        was defined to mean a version of the “To Be” architecture that
        (1) was unconstrained by existing laws, regulations, policies, and
        procedures, (2) addressed selected, previously identified
        deficiencies, and (3) incorporated leading commercial business
        practices.
                                                                                      23




                     Page 29                   GAO-03-458 DOD Architecture Development and Implementation
                      Appendix I
                      Briefing to Subcommittee Staff




                                                             Background (cont.)

    Phase II (October 2002 - April 2003) is focused on developing
       • a version of the “To Be” architecture that (1) is constrained by
         relevant laws, regulations, policies, and procedures, (2) addresses
         all previously identified deficiencies, and (3) incorporates leading
         commercial business practices;
       • a strategy for transitioning from the “As Is” to the “To Be”
         architectural environments; and
       • obtaining stakeholder participation and commitment.

Architecture and Systems Modernization Costs
For fiscal years 2002 and 2003, Congress appropriated approximately $98
million and $96 million, respectively, to support DOD’s effort to develop and
implement the enterprise architecture and transition plan.




                                                                                       24




                      Page 30                   GAO-03-458 DOD Architecture Development and Implementation
                     Appendix I
                     Briefing to Subcommittee Staff




                                                            Background (cont.)

DOD’s Current Business Systems Environment
According to DOD, its current business systems environment consists of
over 1,700 systems to support operations and management decision making,
such as accounting, acquisition, finance, logistics, and personnel.
As we have previously reported,7 this environment was not designed to be,
but rather has evolved into the overly complex and error-prone operation
that exists today, including (1) little standardization across DOD
components, (2) multiple systems performing the same tasks, (3) the same
data stored in multiple systems, and (4) manual data entry into multiple
systems.
To operate and maintain this environment, DOD’s fiscal year 2003 IT budget
request was approximately $18 billion.



7GAO-02-784T.




                                                                                      25




                     Page 31                   GAO-03-458 DOD Architecture Development and Implementation
                      Appendix I
                      Briefing to Subcommittee Staff




                                                             Objective 1: Results

DOD Has Taken Some Positive Steps Towards Developing the
Architecture; However, Key Processes and Controls Are Not Being
Followed

DOD has undertaken a challenging and ambitious task: to, within 1 year,
develop a departmentwide enterprise architecture, or blueprint, for
modernizing its business operations and systems. We agree with DOD’s
goal of developing and implementing an architecture to guide and constrain
investments in systems that provide timely, reliable, and relevant
information for informed decision making. We also support DOD’s efforts
and recognize that the department has taken positive steps in developing
the architecture. Specifically, the department has

    • designated improving financial management operations as 1 of its top
      10 priorities;
    • issued a memorandum directing the development, implementation,
      and maintenance of an enterprise architecture;
    • established a program office responsible for managing the enterprise
      architecture development effort;
                                                                                       26




                      Page 32                   GAO-03-458 DOD Architecture Development and Implementation
                      Appendix I
                      Briefing to Subcommittee Staff




                                                   Objective 1: Results (cont.)


    • selected an architecture framework and repository tool to assist in
      developing the architecture;
    • gathered key data needed to develop the “As Is” architecture, such as
      documenting its inventory of over 1,700 business systems; and
    • engaged DOD subject matter experts (business and systems) to
      assist in developing the architecture.

However, DOD is not following other architecture development best
practices that are designed to reduce the risk that enterprise architecture
programs will fail or fall short of their potential.




                                                                                       27




                      Page 33                   GAO-03-458 DOD Architecture Development and Implementation
                    Appendix I
                    Briefing to Subcommittee Staff




                                                 Objective 1: Results (cont.)


1. DOD has not assigned accountability for enterprise
architecture development to a DOD-wide executive entity.

Enterprise architectures are corporate assets that are intended to
represent the strategic direction of the enterprise. As such, best practices
recommend that an organization establish an Enterprise Architecture
Executive Steering Committee, consisting of top executives, to direct and
oversee the architecture program, and to be accountable for approving
the initial and subsequent versions of the architecture. Sustained support
and commitment by this committee to the architecture, as well as the
committee’s ownership of it, are critical to a successful enterprise
architecture development effort.




                                                                                     28




                    Page 34                   GAO-03-458 DOD Architecture Development and Implementation
                    Appendix I
                    Briefing to Subcommittee Staff




                                                 Objective 1: Results (cont.)

DOD has established two executive committees to provide program
guidance, both consisting of senior leaders from across the department.
However, these committees are not responsible for directing and
overseeing the architecture effort, and they are not accountable for
approving the architecture. Instead, the responsibility of each is limited
to providing guidance to the program office, and advising the DOD
Comptroller on the program.

We observed and were told by department officials that the perception
within the department is that ownership and accountability for the
architecture resides solely with the DOD Comptroller, and not with
DOD’s military services, agencies, and the Office of the Secretary of
Defense principal staff assistants, such as the Under Secretary of
Defense (Acquisition, Technology and Logistics). To address this, the
program office recently developed a draft proposal to improve the
program accountability structure and is currently soliciting stakeholders’
comments. Program officials stated that the new structure will identify
DOD business areas and establish business (domain) owners who will be
accountable for all aspects of the architecture within their domain. DOD
plans to implement its revised accountability structure by no later than
May 2003.
                                                                                     29




                    Page 35                   GAO-03-458 DOD Architecture Development and Implementation
                    Appendix I
                    Briefing to Subcommittee Staff




                                                 Objective 1: Results (cont.)

According to a program official, the department assigned accountability
for the enterprise architecture effort to the Comptroller in response to a
recommendation in a study directed by the Secretary of Defense.

Until DOD corrects this accountability structure, the success of its
enterprise architecture program is at risk. Past DOD initiatives to
improve its business processes and systems, such as the Corporate
Information Management (CIM) initiative, failed in part because they did
not (1) obtain and sustain the departmentwide senior management
leadership, commitment, and support needed to succeed and (2)
effectively overcome deeply embedded cultural resistance to change,
including military service parochialism.




                                                                                     30




                    Page 36                   GAO-03-458 DOD Architecture Development and Implementation
                    Appendix I
                    Briefing to Subcommittee Staff




                                                 Objective 1: Results (cont.)

 2. DOD has not developed an effective strategy for
communicating with architecture stakeholders.

As noted earlier, enterprise architectures are corporate assets to be
owned and used by the major business area executives. As such, it is
critical that architecture plans and activities be transparent to, and have
the support of, these stakeholders and their teams.

To this end, best practices recommend that, before developing the
architecture, organizations initiate a marketing program to educate
stakeholders about the value of the architecture and emphasize the
agency head’s support and commitment. Once initial participation and
commitment is achieved, best practices recommend that an agency
develop and implement an enterprise architecture communications
strategy to facilitate the exchange of information and to keep business
area executives and their teams informed and engaged.



                                                                                     31




                    Page 37                   GAO-03-458 DOD Architecture Development and Implementation
                   Appendix I
                   Briefing to Subcommittee Staff




                                                Objective 1: Results (cont.)

DOD does not have a marketing program or a communications plan for
its enterprise architecture efforts. Instead, it has attempted to
communicate the program’s purpose, scope, approach, and stakeholder
responsibilities as part of its development efforts. For example, it has
(1) conducted a program kickoff meeting to provide a general
explanation of the enterprise architecture goals, objectives, and
processes, (2) involved business area subject matter experts in the
development process, and (3) conducted workshops to discuss initial
versions of architecture products.

According to Army, Navy, and Air Force stakeholders we interviewed,
DOD has not clearly communicated the architecture program’s purpose,
scope, approach, and stakeholder responsibilities.

Further, our observations at workshops held to brief stakeholders on the
program corroborated these statements. That is, we observed that in
many cases workshop participants did not have a clear understanding of
the program.

                                                                                    32




                   Page 38                   GAO-03-458 DOD Architecture Development and Implementation
                   Appendix I
                   Briefing to Subcommittee Staff




                                                Objective 1: Results (cont.)

 • For example, numerous questions were raised at these workshops
   regarding the purpose of the initial version of the “To Be”
   architecture (i.e., strawman architecture) and the participants’ roles
   and responsibilities for reviewing it.

DOD officials agreed that some stakeholders are confused about the
program’s scope and strategy, as well as their roles and responsibilities,
and acknowledged that the department’s lack of a marketing program
and communications plan has contributed to the problem. DOD program
officials stated that the department recognizes that implementing a
communications and change management strategy is critical to the
success of architecture implementation. As a result, the program
manager stated that the department plans to have its contractor develop
such a marketing/communications strategy to be completed by April
2003. The department has not yet issued a statement of work and did not
provide a timeframe by which it will.



                                                                                    33




                   Page 39                   GAO-03-458 DOD Architecture Development and Implementation
                   Appendix I
                   Briefing to Subcommittee Staff




                                                Objective 1: Results (cont.)


According to the program manager, the department did not prepare this
strategy earlier because of competing priorities and tight schedule
demands.

Without a sufficiently open and transparent architecture development
program that ensures full stakeholder understanding, DOD increases the
risk of developing an architecture that is limited in scope, resisted by
business owners, and not supportive of effective business systems
modernization.




                                                                                    34




                   Page 40                   GAO-03-458 DOD Architecture Development and Implementation
                     Appendix I
                     Briefing to Subcommittee Staff




                                                  Objective 1: Results (cont.)

3. DOD has not established an effective process for ensuring
architecture quality.

Best practices recommend that entities establish and implement a quality
assurance function to ensure (1) architecture products meet prescribed
quality standards and measures, (2) reported program performance (e.g.,
satisfaction of cost and schedule commitments) is reliable, and (3) program
management is adhering to relevant process standards. Furthermore, these
best practices recommend that the quality assurance function be
independent of the enterprise architecture program, reporting directly to
an enterprise architecture executive steering committee.

DOD has established a quality assurance function. However, the
department has not developed a plan that documents whether the scope of
this function’s activities extends to all 3 areas, and our review of
documentation and interviews with responsible officials showed that
quality assurance activities only include the first of the three.
Compounding this scope limitation, the function is not independent of the
program management office.

                                                                                      35




                     Page 41                   GAO-03-458 DOD Architecture Development and Implementation
                   Appendix I
                   Briefing to Subcommittee Staff




                                                Objective 1: Results (cont.)

 • With respect to the three areas, program officials stated that quality
   reviews include only architecture products. Based on our
   examination of completed quality reviews, we confirmed that they
   addressed architecture products. In contrast, program office
   documentation states that the quality assurance function will also
   include adherence to relevant process standards; however, this is not
   occurring. Our examination of completed quality reviews showed
   that none addressed adherence to process standards. Further, as
   stated by these officials and confirmed by the documents, the quality
   assurance function is not addressing reported program performance.

In addition, the quality assurance function is not independent. That is,
the quality assurance manager reports directly to the program director,
and not to the architecture steering committee; and the same subject
matter experts responsible for developing the architecture products are
performing certain quality assurance reviews. For example, the same
subject matter experts who developed the strawman architecture were
involved in the quality assurance review of it. In addition,

                                                                                    36




                   Page 42                   GAO-03-458 DOD Architecture Development and Implementation
                   Appendix I
                   Briefing to Subcommittee Staff




                                                Objective 1: Results (cont.)

        • The quality assurance function raised concerns regarding the
          quality of the architecture development methodology and the
          program management plan. However, because the quality
          assurance function reports to the program office, program
          officials still approved these documents without adequately
          addressing the concerns. Moreover, these concerns were
          never brought to the attention of the executive steering
          committee.

According to program officials, the department did not prepare a quality
assurance plan because of schedule demands and staffing limitations.
Further, they stated that the quality assurance function was never tasked
to ensure that process standards were being met or that reported
program performance data were reliable. The program control team
leader also stated that the quality assurance function was not reviewing
program performance data because this was a program office
responsibility. With regard to independence, the quality assurance
manager stated that the department was not aware of the requirement to
report to an executive steering committee.

                                                                                    37




                   Page 43                   GAO-03-458 DOD Architecture Development and Implementation
                   Appendix I
                   Briefing to Subcommittee Staff




                                                Objective 1: Results (cont.)

According to program officials, DOD plans to reconsider the scope of
their quality assurance activities and provide for independent reporting
to the executive steering committee.

Until these deficiencies are corrected, the department lacks reasonable
assurance that it is producing high-quality architecture products.




                                                                                    38




                   Page 44                   GAO-03-458 DOD Architecture Development and Implementation
                      Appendix I
                      Briefing to Subcommittee Staff




                                                             Objective 2: Results


DOD Does Not Have the Means in Place for Ensuring That Ongoing
IT Investments Are Consistent with Its Enterprise Architecture
Efforts

DOD has taken certain steps to control ongoing investments. Specifically,
the DOD Comptroller has

 • issued memorandums that set forth the requirement for Comptroller
   review and approval of IT investments that meet certain criteria—for
   example, systems currently under development are not supposed to go
   beyond the pilot/prototype phase without the Comptroller’s written
   approval; and
 • assessed selected IT projects as part of the budget process, which
   resulted in a reduction of over $222 million in IT funding for fiscal years
   2003 through 2005.

However, these steps do not incorporate three key IT investment control
best practices.

                                                                                       39




                      Page 45                   GAO-03-458 DOD Architecture Development and Implementation
                      Appendix I
                      Briefing to Subcommittee Staff




                                                   Objective 2: Results (cont.)


1. DOD has not established a corporate investment management
governance structure for IT systems.

Best practices recommend that an organization establish an investment
review board, or a hierarchy of boards associated with investments that
meet different thresholds, to select and control IT investments. Among
other things, this board(s) should be composed of representatives from
across the organization and have the authority to make IT investment (i.e.,
funding) decisions.

DOD has not established an investment review board structure consisting
of a hierarchy of boards that are (1) assigned portfolios of investments
based on certain threshold criteria, (2) comprised of representatives from
across the department, and (3) responsible for selecting which IT
investments to fund and for controlling those that are funded. Instead, the
DOD components continue to independently make their own investment
decisions.


                                                                                       40




                      Page 46                   GAO-03-458 DOD Architecture Development and Implementation
                      Appendix I
                      Briefing to Subcommittee Staff




                                                   Objective 2: Results (cont.)


As we have previously reported,8 this stovepiped investment decision-
making process is the result of deeply embedded cultural resistance to
change and military service parochialism.

According to program officials, DOD leadership plans to strengthen its
governance and oversight of its ongoing IT investments. DOD officials
stated that this revised governance structure will incorporate a hierarchy of
investment review boards.

As a result, the department does not have a critical structure in place to
effectively select and control its IT investments, and runs the risk of
continuing to invest in systems that perpetuate its existing incompatible,
duplicative, and overly costly environment of over 1,700 business systems
that do not optimally support mission performance.


8GAO-02-784T.




                                                                                       41




                      Page 47                   GAO-03-458 DOD Architecture Development and Implementation
                      Appendix I
                      Briefing to Subcommittee Staff




                                                   Objective 2: Results (cont.)

2. DOD is not yet using an explicit set of standard criteria for
selecting and controlling its IT investments, to include consistency
and compliance with its ongoing architecture development efforts.

Best practices recommend that investment review boards use common
investment criteria when selecting among competing investment options
and controlling ongoing investments. One critical criterion is alignment of
the investment with the enterprise architecture.

DOD has established some and is establishing other IT investment criteria
that according to program officials will be used by review boards to select
and control investments. However, this criteria has not been finalized or
made part of an investment review process. Further, the criteria does not
implement our open recommendations which would limit current
investments to

 • deployment of systems that have already been fully tested and involve
   no additional development or acquisition cost,
 • stay-in-business maintenance needed to keep existing systems
   operational,
                                                                                       42




                      Page 48                   GAO-03-458 DOD Architecture Development and Implementation
                      Appendix I
                      Briefing to Subcommittee Staff




                                                   Objective 2: Results (cont.)

 • management controls needed to effectively invest in modernized
   systems, and
 • new systems or existing system changes that are congressionally
   directed or are relatively small, cost-effective, and low risk and can be
   delivered in a relatively short time frame.

DOD officials told us that they are in the process of developing a structured
approach, including criteria for assessing a system’s technical
characteristics, economic justification, interoperability with other systems,
and compliance with federal financial management standards.

Without common investment criteria, the department will be unable to
ensure that the new and ongoing IT projects are consistent with the
strategic direction of the department, as is yet to be defined and captured in
the enterprise architecture. Absent this control, DOD will likely continue to
invest in systems that perpetuate its existing incompatible, duplicative, and
overly costly systems environment that does not optimally support mission
performance.

                                                                                       43




                      Page 49                   GAO-03-458 DOD Architecture Development and Implementation
                     Appendix I
                     Briefing to Subcommittee Staff




                                                  Objective 2: Results (cont.)

3. DOD has not conducted a comprehensive review of its ongoing
 IT investments to ensure that they are consistent with its
 architecture development efforts.

 Best practices recommend that an organization review ongoing
 investments periodically to ensure, among other things, that funds are
 being spent in a manner that is consistent with the organization’s
 enterprise architecture.

 While the DOD Comptroller has reviewed selected investments, DOD does
 not have a plan to review the full complement of the department’s IT
 investments in this manner, nor has it established a date by which it will.

    •   The DOD Comptroller reviews are limited primarily to major
        automated information systems acquisitions, which represent $1.4
        billion (approximately 8 percent) of the $18 billion fiscal year 2003
        IT budget request for business systems.


                                                                                      44




                     Page 50                   GAO-03-458 DOD Architecture Development and Implementation
                 Appendix I
                 Briefing to Subcommittee Staff




                                              Objective 2: Results (cont.)

According to program officials, the Comptroller’s office does not have
the resources to conduct such reviews.
  • Only four individuals are currently conducting these reviews.
  • DOD officials stated that they intend to hire 9 additional
       individuals to conduct these reviews.

Until the department exercises effective control over its ongoing IT
investments, it will likely continue to invest in systems that perpetuate
its existing incompatible, duplicative, and overly costly systems
environment that does not optimally support mission performance.




                                                                                  45




                 Page 51                   GAO-03-458 DOD Architecture Development and Implementation
                      Appendix I
                      Briefing to Subcommittee Staff




                                                                       Conclusions


DOD is aggressively developing an enterprise architecture that is intended
to guide and constrain its business systems modernization. Moreover, its
stated goal is to acquire systems that provide timely, reliable, and relevant
information for management to make informed decisions. As such, DOD is
committed to doing the right thing. However, unless it goes about it the
right way, it is unlikely that its enterprise architecture development effort
will result in a quality product, meaning a corporate asset and decision-
making tool that DOD business executives buy into and follow. It is
imperative that DOD position itself for architecture success by doing the
up-front things called for in best practices and avoiding a schedule-driven
strategy that bypasses these keys to success. Our open recommendations to
DOD addressing how it should approach development of its enterprise
architecture are founded on these keys to success. Unless the department
embraces and follows through in implementing these recommendations,
and the best practices on which they are founded, it increases the risk that
its architecture efforts will fall far short of providing a clear and
comprehensive blueprint for effective and efficient business systems
modernization.



                                                                                       46




                      Page 52                   GAO-03-458 DOD Architecture Development and Implementation
                      Appendix I
                      Briefing to Subcommittee Staff




                                                             Conclusions (cont.)


At the same time, it is essential for DOD to get control over the funds
currently being invested in business systems. Thus far, DOD’s efforts to do
so are limited in scope. As a result, it has only scratched the surface in
controlling billions of dollars of ongoing investments in light of its
architecture development efforts. The keys to its success in gaining control
are to apply the proven IT investment management best practices that are
the foundation of our open recommendations for implementing the
architecture.




                                                                                       47




                      Page 53                   GAO-03-458 DOD Architecture Development and Implementation
                     Appendix I
                     Briefing to Subcommittee Staff




                                                             Recommendations

To assist DOD in its efforts to effectively develop and implement an
enterprise architecture, and guide and constrain its business system
investments, and to address the problems discussed in this briefing, we
reiterate the recommendations that we made in our May 2001 report9 that
DOD has yet to implement. In addition, we recommend that the Secretary
of Defense ensure that

 • the enterprise architecture executive committee members are
   singularly and collectively explicitly accountable to the Secretary for
   delivery of the enterprise architecture, including approval of each
   version of the architecture;
 • the enterprise architecture program provides for ensuring stakeholder
   commitment and buy-in through a proactive marketing and
   communication program; and

9GAO-01-525.




                                                                                      48




                     Page 54                   GAO-03-458 DOD Architecture Development and Implementation
                     Appendix I
                     Briefing to Subcommittee Staff




                                                      Recommendations (cont.)


 • the quality assurance function (1) includes the review of adherence to
   process standards and reliability of reported program performance, (2)
   is made independent of the program management function, and (3) is
   not performed by subject matter experts involved in the development of
   key architecture products.

Additionally, we recommend that the Secretary gain control over ongoing IT
investments by
 • establishing a hierarchy of investment review boards, each responsible
    and accountable for selecting and controlling investments that meet
    defined threshold criteria, and each composed of the appropriate level
    of executive representatives, depending on the threshold criteria, from
    across the department;
 • establishing a standard set of criteria, to include (1) alignment and
    consistency with the DOD enterprise architecture and (2) our prior yet
    to be implemented recommendations concerning limitations in business
    system investments pending development of the architecture; and

                                                                                      49




                     Page 55                   GAO-03-458 DOD Architecture Development and Implementation
                    Appendix I
                    Briefing to Subcommittee Staff




                                                     Recommendations (cont.)


• directing these boards to immediately apply these criteria in completing
  reviews of all ongoing IT investments and to not fund investments that
  do not meet these criteria, unless they are otherwise justified by
  explicit criteria waivers.




                                                                                     50




                    Page 56                   GAO-03-458 DOD Architecture Development and Implementation
                      Appendix I
                      Briefing to Subcommittee Staff




                                                              Agency Comments

We provided a draft of this briefing to DOD officials representing the Office
of the Under Secretary of Defense (Comptroller), the Department of the
Navy’s Chief Information Officer, and the Secretary of the Air Force’s
Financial Management Program. Among these officials were the Deputy
Under Secretary of Defense (Financial Management) and the Deputy Chief
Financial Officer.

In commenting on the draft, these officials stated that they generally agreed
with our findings, conclusions and recommendations, with the exception of
our assessment of the adequacy of its quality assurance function. In the
areas where they did agree, they stated that the department is taking
actions that are consistent with our recommendations.

With regard to quality assurance, they stated that this function currently
addresses adherence to process standards and reported program
performance. However, they did not provide us with any evidence that
supported this statement.

                                                                                       51




                      Page 57                   GAO-03-458 DOD Architecture Development and Implementation
                     Appendix I
                     Briefing to Subcommittee Staff




                                                      Agency Comments (cont.)

These officials also provided updated and clarifying information, which we
have incorporated as appropriate in the briefing.




                                                                                      52




                     Page 58                   GAO-03-458 DOD Architecture Development and Implementation
                     Appendix I
                     Briefing to Subcommittee Staff




                                                        Appendix I: Guidance


Chief Information Officer Council. A Practical Guide to Federal Enterprise
Architecture, Version 1. Washington, D.C.: February 2001.

U.S. General Accounting Office. Information Technology Investment
Management: A Framework for Assessing and Improving Process
Maturity. GAO/AIMD-10.1.23, Version 1. Washington, D.C.: May 2000.

U.S. General Accounting Office Assessing Risks and Returns: A Guide for
Evaluating Federal Agencies’ IT Investment Decision-making.
GAO/AIMD-10.1.13, Version 1. Washington, D.C.: February 1997.




                                                                                      53




                     Page 59                   GAO-03-458 DOD Architecture Development and Implementation
Appendix II

Comments from the Under Secretary of
Defense                                                                          Appendx
                                                                                       Ii




              Page 60   GAO-03-458 DOD Architecture Development and Implementation
Appendix II
Comments from the Under Secretary of
Defense




Page 61                  GAO-03-458 DOD Architecture Development and Implementation
Appendix II
Comments from the Under Secretary of
Defense




Page 62                  GAO-03-458 DOD Architecture Development and Implementation
Appendix II
Comments from the Under Secretary of
Defense




Page 63                  GAO-03-458 DOD Architecture Development and Implementation
Appendix III

GAO Contacts and Staff Acknowledgments                                                        Appendx
                                                                                                    iI




GAO Contacts      Cynthia Jackson, (202) 512-5086
                  Jenniffer Wilson, (202) 512-9192



Acknowledgments   In addition to the individuals named above, key contributors to this report
                  included Johnny Bowen, Teressa Broadie-Gardner, Christopher DePerro,
                  Eric Essig, Brian Johnson, Neelaxi Lakhmani, John Ledford, Evelyn Logue,
                  Mai Nguyen, Sanford Reigle, Darby Smith, Stacey Smith, Al Steiner, and
                  Randolph Tekeley.




(192085)          Page 64               GAO-03-458 DOD Architecture Development and Implementation
GAO’s Mission            The General Accounting Office, the audit, evaluation and investigative arm of
                         Congress, exists to support Congress in meeting its constitutional responsibilities
                         and to help improve the performance and accountability of the federal government
                         for the American people. GAO examines the use of public funds; evaluates federal
                         programs and policies; and provides analyses, recommendations, and other
                         assistance to help Congress make informed oversight, policy, and funding
                         decisions. GAO’s commitment to good government is reflected in its core values of
                         accountability, integrity, and reliability.


Obtaining Copies of      The fastest and easiest way to obtain copies of GAO documents at no cost is
                         through the Internet. GAO’s Web site (www.gao.gov) contains abstracts and full-
GAO Reports and          text files of current reports and testimony and an expanding archive of older
                         products. The Web site features a search engine to help you locate documents
Testimony                using key words and phrases. You can print these documents in their entirety,
                         including charts and other graphics.
                         Each day, GAO issues a list of newly released reports, testimony, and
                         correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site
                         daily. The list contains links to the full-text document files. To have GAO e-mail this
                         list to you every afternoon, go to www.gao.gov and select “Subscribe to GAO
                         Mailing Lists” under “Order GAO Products” heading.


Order by Mail or Phone   The first copy of each printed report is free. Additional copies are $2 each. A check
                         or money order should be made out to the Superintendent of Documents. GAO
                         also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a single
                         address are discounted 25 percent. Orders should be sent to:
                         U.S. General Accounting Office
                         441 G Street NW, Room LM
                         Washington, D.C. 20548
                         To order by Phone:     Voice: (202) 512-6000
                                                TDD: (202) 512-2537
                                                Fax: (202) 512-6061


To Report Fraud,         Contact:
                         Web site: www.gao.gov/fraudnet/fraudnet.htm
Waste, and Abuse in      E-mail: fraudnet@gao.gov
Federal Programs         Automated answering system: (800) 424-5454 or (202) 512-7470



Public Affairs           Jeff Nelligan, Managing Director, NelliganJ@gao.gov (202) 512-4800
                         U.S. General Accounting Office, 441 G Street NW, Room 7149
                         Washington, D.C. 20548
United States                  Presorted Standard
General Accounting Office      Postage & Fees Paid
Washington, D.C. 20548-0001           GAO
                                 Permit No. GI00
Official Business
Penalty for Private Use $300
Address Service Requested