oversight

DOD Business Systems Modernization: Important Progress Made to Develop Business Enterprise Architecture, but Much Work Remains

Published by the Government Accountability Office on 2003-09-19.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                 United States General Accounting Office

GAO              Report to Congressional Committees




September 2003
                 DOD BUSINESS
                 SYSTEMS
                 MODERNIZATION
                 Important Progress
                 Made to Develop
                 Business Enterprise
                 Architecture, but
                 Much Work Remains




GAO-03-1018
                 a
                                                September 2003


                                                DOD BUSINESS SYSTEMS
                                                MODERNIZATION

Highlights of GAO-03-1018, a report to          Important Progress Made to Develop
congressional committees
                                                Business Enterprise Architecture, but
                                                Much Work Remains


The National Defense                            DOD has expended tremendous effort and resources and made important
Authorization Act for Fiscal Year               progress in complying with the act’s requirements aimed at developing and
2003 directed the Department of                 effectively implementing a well-defined business enterprise architecture.
Defense (DOD) to develop an                     Further, DOD’s initial version of its architecture provides a foundation from
enterprise architecture and a                   which to build and ultimately produce a well-defined architecture. For
transition plan that meets certain
requirements. The act also
                                                example, the “As Is” environment includes an inventory of about 2,300
directed DOD to have a process for              existing systems and their characteristics that support DOD’s current
controlling its system investments.             business operations; and the “To Be” environment addresses, to at least
As required by the act, GAO                     some degree, how DOD intends to operate in the future, what information
assessed DOD’s actions to comply                will be needed to support these future operations, and what technology
with the act’s requirements and                 standards should govern the design of future systems. Further, DOD has
recently issued a report to                     established some of the architecture management capabilities advocated by
congressional defense committees.               best practices and federal guidance, such as having a program office,
This report provides further details            designating a chief architect, and using an architecture development
of GAO’s assessment results                     methodology and automated tool.
regarding (1) the extent to which
DOD’s actions complied with the
requirements of the act and                     At the same time, DOD’s initial architecture does not yet adequately address
(2) DOD’s plans for further                     the act’s requirements and other relevant architectural requirements
development and implementation                  governing the scope and content. For example, critical federal requirements
of its architecture.                            governing the “To Be” architecture, such as federal accounting requirements
                                                for recording revenue, are not included in the initial architecture. Other
                                                items not included are
To further assist DOD in its efforts            •   descriptions of the current business operations in terms of entities and
to effectively develop and
implement an enterprise
                                                    people who perform the functions, processes, and activities and the
architecture and to guide and                       locations where these are performed;
constrain its business system                   •   descriptions of the systems to be developed or acquired to support
investments, GAO is making                          future business operations; and
recommendations to the Secretary                •   time frames for phasing out existing systems.
of Defense aimed at improving
DOD’s plans for developing the                  Furthermore, DOD has not yet implemented an effective investment
next version of the architecture                management process for selecting and controlling ongoing and planned
and implementing the institutional              business system investments. Until it does, DOD remains at risk of spending
means for selecting and controlling             billions of dollars on duplicative, stove-piped, nonintegrated systems that do
both planned and ongoing business               not optimize mission performance and accountability and, therefore, do not
system investments. DOD
concurred with 9 of our 10
                                                support the department’s business transformation goals.
recommendations, partially
concurred with the remaining one,               Overall, our findings indicate that DOD has taken positive first steps, but
and described completed, ongoing,               much remains to be accomplished before DOD will have the kind of
or planned actions to address them.             blueprint and associated investment controls to successfully modernize its
                                                business operations and supporting systems. According to program officials
                                                and the initial version of the transition plan, DOD intends to extend and
www.gao.gov/cgi-bin/getrpt?GAO-03-1018
To view the full product, including the scope
                                                evolve the architecture to include the missing scope and detail; however, it
and methodology, click on the link above.       has not yet defined specific plans outlining how this will be accomplished.
For more information, contact Gregory Kutz,
(202) 512-9095 (kutzg@gao.gov) or
Randolph Hite, (202) 512-3439
(hiter@gao.gov).
Contents



Letter                                                                                                     1
                             Results in Brief                                                              3
                             Background                                                                    6
                             DOD Has Taken Positive First Steps in Complying with Enterprise
                               Architecture Legislative Requirements, but Much Remains to be
                               Accomplished                                                               16
                             DOD’s Plans for Evolving and Extending Its Enterprise Architecture
                               and for Improving Business System Investment Decision Making
                               Are Unclear                                                                50
                             Conclusions                                                                  52
                             Recommendations                                                              53
                             Agency Comments and Our Evaluation                                           55


Appendixes
              Appendix I:    SEC. 1004. [of Public Law 107-314] Development and
                             Implementation of Financial Management Enterprise
                             Architecture                                                                 58
             Appendix II:    Scope and Methodology                                                        60
             Appendix III:   Status of Prior Recommendations on DOD’s Business
                             Enterprise Architecture                                                      64
             Appendix IV:    Comments from the Department of Defense                                      68
              Appendix V:    GAO Contacts and Staff Acknowledgments                                       73
                             GAO Contacts                                                                 73
                             Acknowledgments                                                              73


Tables                       Table 1: Interrelationship Between Domains and Business Process
                                      Areas                                                                9
                             Table 2: Summary of GAO’s Enterprise Architecture (EA) Maturity
                                      Framework Stages                                                    18
                             Table 3: Assessment of DOD’s Enterprise Architecture Efforts
                                      Against GAO’s Enterprise Architecture Maturity
                                      Framework                                                           20
                             Table 4: Summary of GAO Analysis of JFMIP Requirements                       25
                             Table 5: Detailed Analysis of the Extent to Which DOD’s “As Is”
                                      Architecture Satisfies Key Elements                                 33
                             Table 6: Detailed Analysis of the Extent to Which DOD’s “To Be”
                                      Architecture Satisfies Key Elements                                 38




                             Page i                          GAO-03-1018 DOD Business Enterprise Architecture
          Contents




          Table 7: Detailed Analysis of the Extent to Which DOD’s Transition
                   Plan Satisfies Key Architectural Elements                                     45


Figures   Figure 1: Interdependent C4ISR Views                                                   12
          Figure 2: Summary of Extent to Which Version 1.0 of DOD’s
                    Enterprise Architecture Satisfies Key Elements
                    Governing Architectural Content                                              30




           This is a work of the U.S. government and is not subject to copyright protection in the
           United States. It may be reproduced and distributed in its entirety without further
           permission from GAO. However, because this work may contain copyrighted images or
           other material, permission from the copyright holder may be necessary if you wish to
           reproduce this material separately.




          Page ii                                GAO-03-1018 DOD Business Enterprise Architecture
A
United States General Accounting Office
Washington, D.C. 20548



                                    September 19, 2003                                                                            Leter




                                    Congressional Committees

                                    Our research of successful public and private sector organizations shows
                                    that attempting a large-scale systems modernization program without
                                    having a well-defined modernization blueprint—commonly called an
                                    enterprise architecture—and effective investment management controls,
                                    results in systems that are duplicative, are not well-integrated, are
                                    unnecessarily costly to maintain and interface, and do not effectively
                                    optimize mission performance. Accordingly, in May 20011 we
                                    recommended that the Department of Defense (DOD) develop an
                                    enterprise architecture to guide and constrain its almost $20 billion annual
                                    investment in business systems and establish the investment controls
                                    needed to implement this architecture. In July 2001, DOD initiated a
                                    program to, among other things, address our recommendations, and began
                                    developing a DOD-wide business enterprise architecture (BEA) in April
                                    2002. This effort is an essential part of the Secretary of Defense’s broad
                                    initiative to “transform the way the department works and what it works
                                    on.”

                                    The National Defense Authorization Act for Fiscal Year 20032 required DOD
                                    to develop by May 1, 2003, a financial management enterprise architecture3
                                    and a transition plan for implementing the architecture that meet certain
                                    requirements. The act also requires DOD to control expenditures for
                                    financial system improvements while the architecture and transition plan
                                    are being developed and after they are completed. The act states that the
                                    enterprise architecture shall describe an information infrastructure that, at
                                    a minimum, would enable DOD to achieve certain capabilities, such as
                                    complying with all federal accounting, financial management, and


                                    1
                                      U.S. General Accounting Office, Information Technology: Architecture Needed to Guide
                                    Modernization of DOD’s Financial Operations, GAO-01-525 (Washington, D.C.: May 17,
                                    2001).
                                    2
                                      Bob Stump National Defense Authorization Act for Fiscal Year 2003, Pub. L. No. 107-314, §
                                    1004, 116 Stat. 2458, 2629, Dec. 2, 2002.
                                    3
                                      In May 2003, the DOD Comptroller changed the architecture name from the Financial
                                    Management Enterprise Architecture to the BEA to reflect the transformation of
                                    departmentwide business operations and supporting systems, including accounting and
                                    finance, budget formulation, acquisition, inventory management, logistics, personnel, and
                                    property management systems.




                                    Page 1                                 GAO-03-1018 DOD Business Enterprise Architecture
reporting requirements. The act also requires development of a transition
plan for implementing the enterprise architecture that includes, among
other things, a schedule for phasing out existing systems that will not
become part of the “To Be” environment. Finally, before the architecture
and transition plan are approved, the act requires DOD to review proposed
obligations of funds in amounts exceeding $1 million for financial system
improvements to determine if they meet specific conditions called for in
the act. Once the architecture and transition plan are approved, the act
requires DOD to ensure that obligations exceeding $1 million for financial
system investments are consistent with the architecture and the transition
plan.

The act also directs us to submit to congressional defense committees,
within 60 days of DOD’s approval of its enterprise architecture and its
transition plan, an assessment of DOD’s actions taken to comply with these
requirements. (See app. I for a copy of section 1004 of the act.) We
recently issued a report to satisfy this requirement.4 This report provides
specific details on our assessment results regarding (1) the extent to which
DOD’s actions complied with the requirements of the act and (2) DOD’s
plans for further development and implementation of its enterprise
architecture. It also makes recommendations to the Secretary of Defense
for improving DOD’s architecture development, maintenance, and
implementation efforts, including restricting investments in systems until
the architecture is sufficiently defined and effective controls are in place
for ensuring new investments are aligned with it.

We performed our work from March 2003 through June 2003 in accordance
with U.S. generally accepted government auditing standards. Details on
our scope and methodology are included in appendix II. We requested
comments on a draft of this report from the Secretary of Defense or his
designee. Written comments from the Under Secretary of Defense
(Comptroller) are addressed in the “Agency Comments and Our
Evaluation” section of this report and are reprinted in appendix IV.




4
  U.S. General Accounting Office, Business Systems Modernization: Summary of GAO’s
Assessment of Department of Defense’s Initial Business Enterprise Architecture, GAO-03-
877R (Washington, D.C.: July 7, 2003).




Page 2                               GAO-03-1018 DOD Business Enterprise Architecture
Results in Brief   As we reported in February 2003,5 DOD undertook a challenging and
                   ambitious task—to develop within 1 year a departmentwide architecture
                   for modernizing its current financial and business operations and systems.
                   Thus far, DOD has expended tremendous effort and resources and made
                   important progress in complying with the legislative requirements aimed at
                   developing and effectively implementing a well-defined enterprise
                   architecture. Concerning progress, the department has established some
                   of the architecture management capabilities advocated by best practices
                   and federal guidance,6 such as having a program office, designating a chief
                   architect, and using an architecture development methodology and
                   automated tool. Further, DOD’s initial version of its BEA provides a
                   foundation from which to build and ultimately produce a well-defined
                   business enterprise architecture. For example, the “As Is” descriptions
                   include an inventory of about 2,300 systems in operation or under
                   development, and their characteristics, that support DOD’s current
                   business operations. The “To Be” descriptions address, to at least some
                   degree, how DOD intends to operate in the future, what information will be
                   needed to support these future operations, and what technology standards
                   should govern the design of future systems.

                   At the same time, the initial version does not yet adequately address the
                   act’s requirements and other relevant architectural requirements.7 For
                   example,

                   • While DOD has incorporated many relevant external requirements from
                     152 federal sources in developing its “To Be” architecture products,


                   5
                     U.S. General Accounting Office, DOD Business Systems Modernization: Improvements to
                   Enterprise Architecture Development and Implementation Efforts Needed, GAO-03-458
                   (Washington, D.C.: Feb. 28, 2003).
                   6
                     U.S. General Accounting Office, Information Technology: A Framework for Assessing
                   and Improving Enterprise Architecture Management (Version 1.1), GAO-03-584G
                   (Washington, D.C.: April 2003).
                   7
                     See for example, Office of Management and Budget, Federal Enterprise Architecture
                   Business Reference Model, Version 1.0 (2002); Chief Information Officer Council, A
                   Practical Guide to Federal Enterprise Architecture, Version 1.0 (February 2001); Office of
                   Management and Budget Circular A-130, Management of Federal Information Resources
                   (Nov. 28, 2000); M.A. Cook, Building Enterprise Information Architectures: Reengineering
                   Information Systems (Upper Saddle River, N.J., Prentice Hall: 1996); and National Institute
                   of Standards and Technology, Information Management Directions: The Integration
                   Challenge, Special Publication 500-167 (September 1989).




                   Page 3                                 GAO-03-1018 DOD Business Enterprise Architecture
   critical federal requirements governing the “To Be” architecture, such as
   federal accounting requirements for recording revenue, are not
   included. As a result, the architecture’s descriptions of certain business
   processes, such as those associated with revenue accounting and
   reporting, which include over $70 billion earned annually by working
   capital fund activities, are not complete.

• The “As Is” environment provides little descriptive content and does not
  satisfy 90 percent of the architectural elements required by relevant
  guidance, such as descriptions of the current business operations in
  terms of the entities/people who perform the functions, processes, and
  activities, and the locations where the functions, processes, and
  activities are performed. As a result, DOD does not have a sufficiently
  described picture of its current environment to permit development of a
  meaningful and useful transition plan.

• The “To Be” architecture does not provide sufficient descriptive content
  related to future business operations and supporting technology to
  permit effective acquisition and implementation of system solutions and
  associated operational change. For example, it does not include
  descriptions of the actual systems to be developed or acquired to
  support future business operations and the physical infrastructure (e.g.,
  hardware and software) that will be needed to support the business
  systems. As a result, the “To Be” environment lacks the details needed
  to provide DOD with a common vision and frame of reference for
  defining a transition plan to guide and constrain capital investments,
  and thus to effectively leverage technology to orchestrate logical,
  systematic change and optimize enterprisewide mission performance.

• The transition plan does not possess the attributes needed to provide a
  temporal roadmap for moving from the “As Is” to the “To Be”
  environment and is basically a plan to develop a transition plan. For
  example, such information as time frames for phasing out existing
  systems within DOD’s current inventory of about 2,300 systems, and
  resource requirements for implementing the “To Be” architecture, are
  not part of the transition plan. As a result, DOD does not yet have a
  meaningful and reliable basis for managing the disposition of its existing
  inventory of about 2,300 systems or for sequencing the introduction of
  modernized business operations and supporting systems.




Page 4                           GAO-03-1018 DOD Business Enterprise Architecture
Moreover, DOD has not yet defined and implemented an effective approach
to select and control business systems investments8 for obligations
exceeding $1 million while the architecture is being developed and after it
is completed. Since enactment of the act, as of June 6, 2003, DOD had
approved one business system improvement for $10 million that met this
$1 million threshold and was currently reviewing four others. Our analysis
of DOD’s fiscal years 2003 and 2004 information technology (IT) budget
requests shows that over 200 business systems in each year’s budget,
totaling about $4 billion per year, could involve obligations of funds that
meet the $1 million threshold. This indicates that the majority of the
billions of dollars that DOD invests in business system improvements
annually have not been subject to the scrutiny of the DOD Comptroller now
called for in the act. Overall, our findings indicate that DOD has taken
positive first steps, but much remains to be accomplished before DOD will
have the kind of blueprint and associated investment controls needed to
successfully modernize its financial management operations and
supporting business systems.

DOD’s position is that, to varying degrees, the initial version of its
architecture fully satisfies the act’s requirements, but it also recognizes that
the architecture needs to be expanded and extended to provide a sufficient
basis for guiding and constraining investment decisions. DOD’s position is
also that it has taken steps to implement the act’s requirements regarding
approving system investments, but that it needs to do more to effectively
select and control business system investments. DOD attributes the
current state of its architecture and investment management processes to
the limited time it has had to define and implement each, in part because it
was overly optimistic in estimating what it could deliver by May 1, 2003.
Until DOD develops and provides for effective implementation of a well-
defined enterprise architecture, its ability to modernize its business and
systems environments in a way that minimizes risk and maximizes return
on investment will be severely hindered.

According to program officials and the initial version of the transition plan,
DOD intends to extend and evolve the architecture to include the missing
scope and detail. However, it has not defined specific plans outlining how


8
  Business systems include financial and nonfinancial systems, such as civilian personnel,
finance, health, logistics, military personnel, procurement, and transportation, with the
common element being the generation or use of financial data to support DOD’s business
operations.




Page 5                                  GAO-03-1018 DOD Business Enterprise Architecture
             this will be accomplished. Rather, DOD’s current plan is to develop a
             strategy for producing the next version of its architecture, and managing
             ongoing and planned investments. Among other things, this strategy is to
             provide for

             • determining the resources needed to further develop the architecture;

             • developing a methodology for integrating the architecture with other
               internal and external architectures;

             • establishing an approach for maintaining its existing systems inventory;
               and

             • evaluating the architecture for completeness, accuracy, and integration
               of end-to-end business processes and system functions.

             In addition, DOD program documentation provides for initiating pilot
             projects in the near term that are to demonstrate and implement a portion
             of the architecture and be usable across the department. However, DOD
             officials stated that the pilot projects are intended to validate
             departmentwide business processes and not to implement production
             systems. Because of these differing views of what the pilot projects are
             intended to achieve, the purpose and scope of these projects remain
             unclear, and specific projects have yet to be selected.

             To assist DOD, we are making recommendations to the Secretary of
             Defense aimed at improving its plans for developing the next version of the
             architecture and implementing the institutional means for selecting and
             controlling both planned and ongoing system investments. DOD concurred
             with 9 of our 10 recommendations, partially concurred with the remaining
             one, and described actions recently completed, ongoing, or planned to
             implement them.



Background   DOD is one of the largest and most complex organizations in the world. In
             fiscal year 2002, DOD reported that its operations involve about $700
             billion in assets, nearly $1.5 trillion in liabilities, approximately 3.3 million
             military and civilian personnel, and disbursements of over $346 billion.
             Moreover, execution of these operations spans a wide range of defense
             organizations, including the military services and their respective major
             commands and functional activities, numerous large defense agencies and
             field activities, and various combatant and joint operational commands



             Page 6                             GAO-03-1018 DOD Business Enterprise Architecture
that are responsible for military operations for specific geographic regions
or theaters of operations. To execute these military operations, the
department performs an assortment of interrelated and interdependent
business functions, including logistics management, procurement,
healthcare management, and financial management.

The department’s pervasive problems in performing these business
functions are well chronicled by the DOD Inspector General, the military
service audit agencies, and us. Of the 25 areas in the federal government
that we have designated as high risk, 6 are DOD program areas (i.e.,
systems modernization management, financial management, contract
management, inventory management, support infrastructure management,
and weapon systems acquisition), and DOD shares responsibility for 3 of
the governmentwide high-risk areas (i.e., strategic human capital
management, protecting information systems and critical infrastructures,
and federal real property).9 DOD’s problems in each of these areas hinder
the efficiency of operations, and leave the department vulnerable to fraud,
waste, and abuse.

To support its business functions, DOD reports that it currently relies on
about 2,300 systems, including accounting, acquisition, logistics, and
personnel. As we have previously reported,10 this environment was not
designed to be, but rather has evolved into, an overly complex, and error-
prone IT environment, including (1) little standardization across DOD,
(2) multiple systems performing the same tasks, (3) the same data stored in
multiple systems, and (4) manual data entry into multiple systems. For
fiscal year 2003, DOD requested approximately $26 billion in IT funding to
support a wide range of military operations as well as DOD business
system operations. Approximately $18 billion—nearly $5.2 billion for
business systems and $12.8 billion primarily for business systems
infrastructure—relates to the operation, maintenance, and modernization


9
  U.S. General Accounting Office, High Risk Series: An Update, GAO-03-119 (Washington,
D.C.: January 2003); U.S. General Accounting Office, High Risk Series: Strategic Human
Capital Management, GAO-03-120 (Washington, D.C.: January 2003); U.S. General
Accounting Office, High Risk Series: Protecting Information Systems Supporting the
Federal Government and the Nation’s Critical Infrastructures, GAO-03-121 (Washington,
D.C.: January 2003); and U.S. General Accounting Office, High Risk Series: Federal Real
Property, GAO-03-122 (Washington, D.C.: January 2003).
10
  U.S. General Accounting Office, DOD Financial Management: Important Steps
Underway But Reform Will Require a Long-term Commitment, GAO-02-784T (Washington,
D.C.: June 4, 2002).




Page 7                                GAO-03-1018 DOD Business Enterprise Architecture
of DOD’s business systems. The remaining $8 billion relates primarily to
command and control systems, including the infrastructure to support
these systems.

One of the seven key elements we have reported11 as necessary to
successfully execute the department’s business systems modernization
program is establishing and implementing an enterprise architecture.
Subsequently, in its fiscal year 2002 Performance and Accountability
Report, DOD acknowledged that deficiencies in its systems and business
processes hindered the department’s ability to collect and report financial
and performance information that is accurate, reliable, and timely. The
report noted that to address its systemic problems and assist in the
transformation of the department’s business operations, the department
had undertaken the development and implementation of a business
enterprise architecture, or modernization blueprint. Table 1 shows the
scope of DOD’s business operations, including business domains owners,
and related business process areas and supporting functions.




11
  U.S. General Accounting Office, Department of Defense: Status of Financial
Management Weaknesses and Progress Toward Reform, GAO-03-931T (Washington D.C.:
June 25, 2003).




Page 8                             GAO-03-1018 DOD Business Enterprise Architecture
Table 1: Interrelationship Between Domains and Business Process Areas

Domain                              Domain owner                      Business process areas            Business process functions
Acquisition/Procurement             Under Secretary of Defense        Procurement and Acquisition       Identifying a need and procuring and
                                    (Acquisition, Technology and                                        acquiring goods and services to satisfy the
                                    Logistics)                                                          need (includes managing contracts and
                                                                                                        purchase card programs).
Finance, Accounting       Under Secretary of Defense                  Accounting                        Identifying, measuring, recording, and
Operations, and Financial (Comptroller/Chief Financial                                                  communicating economic information
Management                Officer)                                                                      about an organization (includes
                                                                                                        developing and maintaining DOD standard
                                                                                                        accounting structure, policies, and cost
                                                                                                        accounting).

                                                                      Collection, Accounts              Recording, tracking, managing,
                                                                      Receivable, and Cash              monitoring, liquidating, and collecting
                                                                      Management                        amounts due to DOD (includes reconciling
                                                                                                        fund balance with Treasury).

                                                                      Payables and Disbursing           Receiving payment requests, determining
                                                                                                        payment due dates, and issuing payment.

                                                                      Financial and Management          Providing accurate, reliable, and timely
                                                                      Reporting                         financial and management information.
Human Resource                      Under Secretary of Defense        Human Resource                    Facilitating entry to the organization,
Management                          (Personnel and Readiness)         Management                        developing and managing careers,
                                                                                                        managing benefits and pay, executing
                                                                                                        policies and procedures, and managing
                                                                                                        employee information.
Logistics                           Under Secretary of Defense        Logistics                         Planning, controlling, and carrying out the
                                    (Acquisition, Technology and                                        efficient and effective movement and
                                    Logistics)                                                          maintenance of forces (includes inventory
                                                                                                        and personal property management).
Strategic Planning and              Under Secretary of Defense        Strategic Planning and            Strategic planning, developing the
Budgeting                           (Comptroller/Chief Financial      Budgeting                         programs and the budget, and executing
                                    Officer)                                                            the budget.
Installations and                   Under Secretary of Defense        Real Property and                 Managing all real property and
Environment                         (Acquisition, Technology and      Environmental Liabilities         environmental controls.
                                    Logistics)
Technical Infrastructure            Assistant Secretary of Defense    All of the above                  Providing foundation for enterprise data
                                    (Networks and Information                                           management, reporting, enterprise and
                                    Integration)/Chief Information                                      technical services, and standards and
                                    Officera                                                            policy (includes information assurance).
Source: GAO analysis of DOD data.
                                                       a
                                                        Formerly known as Assistant Secretary of Defense (Command, Control, Communications and
                                                       Intelligence)/Chief Information Officer.




                                                       Page 9                                     GAO-03-1018 DOD Business Enterprise Architecture
Enterprise Architecture Is     Effective use of enterprise architectures is a trademark of successful public
Critical to DOD’s Ability to   and private organizations. For a decade, we have promoted the use of
                               architectures, recognizing them as a crucial means to a challenging goal:
Improve Its Business           agency operational structures that are optimally defined, in both business
Functions                      and technological environments. The Congress, the Office of Management
                               and Budget (OMB), and the federal Chief Information Officer (CIO)
                               Council also have recognized the importance of an architecture-centric
                               approach to modernization. For example, the Clinger-Cohen Act of 199612
                               mandates that an agency’s CIO develop, maintain, and facilitate the
                               implementation of an architecture within the agency and that the agency’s
                               decisions to invest in IT satisfy specified criteria and take into account the
                               agency’s business processes. Further, OMB has issued guidance13 that,
                               among other things, requires system investments to be consistent with
                               these architectures.

What Is an Enterprise          An enterprise architecture provides a clear and comprehensive picture of
Architecture?                  an entity, whether it is an organization (e.g., federal department or agency)
                               or a functional or mission area that cuts across more than one organization
                               (e.g., financial management). This picture consists of snapshots of both the
                               enterprise’s current or “As Is” operational and technological environment
                               and its target or “To Be” environment, as well as a capital investment
                               roadmap for transitioning from the current to the target environment.
                               These snapshots further consist of “views,” which are basically one or
                               more architecture products that provide conceptual or logical
                               representations of the enterprise.

                               The concept of enterprise architectures dates back to the mid-1980s. At
                               that time, John Zachman, widely recognized as a leader in the field of
                               enterprise architecture, identified the need to use a logical construction
                               blueprint (i.e., an architecture) for defining and controlling the integration
                               of systems and their components.14 Accordingly, Zachman developed a
                               structure or “framework” for defining and capturing an architecture. In his
                               work, Zachman drew parallels to the field of classical architecture and later

                               12
                                 The Clinger-Cohen Act of 1996, Pub. L. No. 104-106, Div. E, Title LI, sections 5122 and 5125,
                               110 Stat. 679, 683-85, Feb. 10, 1996 (codified, as amended, at 40 U.S.C. sections 11312 and
                               11315(b)(2)).
                               13
                                    OMB Circular No. A-130, Management of Federal Information Resources (Nov. 28, 2000).
                               14
                                  J.A. Zachman, “A Framework for Information Systems Architecture,” IBM Systems
                               Journal 26, no. 3 (1987).




                               Page 10                                  GAO-03-1018 DOD Business Enterprise Architecture
to the aircraft manufacturing industry, in which different work products
(e.g., architect plans, contractor plans, shop plans, and bills of lading)
represent different views of the planned building or aircraft. Similarly,
Zachman’s framework identified the kinds of work products needed for
people to understand and thus build a given system or entity. This
framework provides for six windows from which to view the enterprise,
which Zachman terms “perspectives” on how a given entity operates: those
of (1) the strategic planner, (2) the system user, (3) the system designer,
(4) the system developer, (5) the subcontractor, and (6) the system itself.
Zachman also proposed six abstractions or models associated with each of
these perspectives: these models cover (1) how the entity operates,
(2) what the entity uses to operate, (3) where the entity operates, (4) who
operates the entity, (5) when entity operations occur, and (6) why the entity
operates. Zachman’s framework provides a way to identify and describe an
entity’s existing and planned component parts and the parts’ relationships
before one begins the costly and time-consuming efforts associated with
developing or transforming the entity.

Since Zachman introduced his framework, a number of other frameworks
have been proposed. In February 1998, DOD directed its components to
use its Command, Control, Communications, Computers, Intelligence,
Surveillance, and Reconnaissance (C4ISR) Architecture Framework,
Version 2.0. The C4ISR architecture framework defines the type and
content of architectural artifacts, as well as the relationships among
artifacts, that are needed to produce a useful enterprise architecture.
Briefly, the framework decomposes an enterprise architecture into three
primary views (windows into how the enterprise operates): the
operational, systems, and technical views. According to DOD, the three
interdependent views are needed to ensure that IT systems are developed
and implemented in an interoperable and cost-effective manner. (See fig. 1
for an illustration of the three views.)




Page 11                          GAO-03-1018 DOD Business Enterprise Architecture
Figure 1: Interdependent C4ISR Views


                                                                      Operational view


                                       of                              Identifies warfighter                      inf
                                     s
                                  vel nts                                relationships and                           or P
                                 e me                                                                                  m r
                                                                                                                        a


                                            eq al l
                                    e                                   information needs




                                                                                                                 oc e
                                                                                                                  tio
                                                 r
                                      ng mod        s, nts




                                                                                                                   es xch
                                              ui




                                                                                                                      n
                                                  de me




                                                                                                                       sin an
                                                no uire
                                          r
                                          r
                              exc inte




                                                                                                                          g a ge
                                        e




                                                                                                                           Ba a
                                                  q

                                an to




                                                                                                                             nd e q u
                                                                                                                              sic nd n
                                    re
                                   ha
                          ion and




                            s, ns
                                  d




                                                                                                                                 lev
                         ine atio




                                                                                                                                  tec ew



                                                                                                                                    r
                infor ssing




                                                                                                                                     els ements
                                                                                                                                      hn
                              i
                  , ne soc




                                                                                                                                        olo pabilities




                                                                                                                                         of
                                                                                                                                          ir
       activities s as
                      mat
                        e




                      edl




                                                                                                                                             gy s
                   Proc




                                                                                                                                              ca
                    m




                                                                                                                                                 uppo
            Syste




                                                                                                                                                     rtability
              Systems view                                                                                        Technical view

            Relates capabilities
                                                                                                                      Prescribes
            and characteristics
                                                                                                                       standards
              to operational
                                                                                                                    and conventions
              requirements                                       S
                                                              to s pecifi
                                                                  atis c capabilities identified vels
                                                                  andfy information exchange le
                                                                       opera                    ts
                                                                             tional requiremen                     /
                                                                                                                 on
                                                  Te
                                                     ch                                                  n t ati
                                                          n ic                                          e
                                                               a                                      em s
                                                          pro l criteri                           mpl    e
                                                               cure a governing interoperable i abiliti
                                                                   m ent                         cap
                                                                         of the selected syste m


Source: C4ISR Architecture Framework, Version 2.



More recently, OMB established the Federal Enterprise Architecture
Program Management Office to develop a federated enterprise architecture
according to a collection of five “reference models”:

• The Business Reference Model is intended to describe the business
  operations of the federal government independent of the agencies that
  perform them, including defining the services provided to state and local
  governments.




Page 12                                                                   GAO-03-1018 DOD Business Enterprise Architecture
• The Performance Reference Model is to provide a common set of
  general performance outputs and measures for agencies to use to
  achieve business goals and objectives.

• The Data and Information Reference Model is to describe, at an
  aggregate level, the type of data and information that support program
  and business line operations, and the relationships among these types.

• The Service Component Reference Model is to identify and classify IT
  service (i.e., application) components that support federal agencies and
  promote the reuse of components across agencies.

• The Technical Reference Model is to describe how technology is
  supporting the delivery of service components, including relevant
  standards for implementing the technology.

These post-Zachman frameworks differ in their nomenclatures and
modeling approach. However, the frameworks consistently provide for
defining an enterprise’s operations in both (1) logical terms, such as
interrelated business processes and business rules, information needs and
flows, and work locations and users, and (2) technical terms, such as
hardware, software, data, communications, and security attributes and
performance standards. The frameworks also provide for defining these
perspectives both for the enterprise’s “As Is” and “To Be” environments, as
well as a transition plan for moving from the “As Is” to the “To Be”
environment.

The importance of developing, implementing, and maintaining an
enterprise architecture is a basic tenet of both organizational
transformation and IT management. Managed properly, an enterprise
architecture can clarify and help optimize the interdependencies and
relationships among an organization’s business operations and the
underlying IT infrastructure and applications that support these
operations. Employed in concert with other important management
controls, such as portfolio-based capital planning and investment control
practices, architectures can greatly increase the chances that
organizations’ operational and IT environments will be configured so as to
optimize mission performance. Our experience with federal agencies has
shown that investing in IT without defining these investments in the




Page 13                         GAO-03-1018 DOD Business Enterprise Architecture
                            context of an architecture often results in systems that are duplicative, not
                            well integrated, and unnecessarily costly to maintain and interface.15

                            Subsection 1004(b) of the act (see app. I) also defines elements required of
                            DOD’s enterprise architecture that are consistent with the above
                            mentioned enterprise architecture guidance and requirements.
                            Specifically, DOD’s financial management enterprise architecture (its BEA)
                            must define an information infrastructure that would enable DOD to meet
                            certain requirements and it must include policies, procedures, data
                            standards, and system interface requirements applicable uniformly
                            throughout DOD.



Prior Reviews of DOD’s      For the last 2 years, GAO has addressed the need for and reviewed DOD’s
Enterprise Architecture     efforts to develop an enterprise architecture for modernizing its business
                            operations and systems and made recommendations to assist DOD in
Efforts Have Identified
                            successfully developing the architecture and using it to gain control over its
Challenges and Weaknesses   ongoing business system investments.

                            In particular, we reported in May 200116 that the department had neither an
                            enterprise architecture for its financial and financial-related business
                            operations, nor the management structure, processes, and controls in place
                            to effectively develop and implement one. We also reported that the
                            department planned to spend billions of dollars on new and modified
                            business systems that would function independently from one another and
                            outside the context of an enterprise architecture. We concluded that if the
                            department continued down this path, it would only perpetuate its existing
                            business operations and systems environment, which we described as
                            duplicative, not interoperable, unnecessarily costly to maintain and
                            interface, and not optimizing mission performance and accountability. We
                            made eight recommendations to the Secretary of Defense aimed at
                            providing the means for effectively developing and implementing an

                            15
                              See, for example, U.S. General Accounting Office, DOD Business Systems
                            Modernization: Improvements to Enterprise Architecture Development and
                            Implementation Efforts Needed, GAO-03-458 (Washington, D.C.: February 2003); U.S.
                            General Accounting Office, Information Technology: DLA Should Strengthen Business
                            Systems Modernization Architecture and Investment Activities, GAO-01-631 (Washington,
                            D.C.: June 2001); and U.S. General Accounting Office, Information Technology: INS Needs
                            to Better Manage the Development of Its Enterprise Architecture, AIMD-00-212
                            (Washington, D.C.: August 2000).
                            16
                                 GAO-01-525.




                            Page 14                              GAO-03-1018 DOD Business Enterprise Architecture
enterprise architecture. Of the eight recommendations, DOD has fully
implemented one, partially implemented five, and has not yet implemented
two.

In February 2003,17 we reported that while DOD is following some
enterprise architecture and IT investment management processes and
controls, it is not following others, in part, because it was focused on
meeting an ambitious schedule. More specifically, with respect to
developing the architecture, we reported that DOD had yet to (1) establish
a governance structure and process controls needed to ensure ownership
of and accountability for the architecture across the department, (2) clearly
communicate to intended stakeholders its purpose, scope, and approach to
developing the architecture, and (3) define and implement an independent
quality assurance process.

We also reported in our February 2003 report that, while DOD had taken
some initial steps aimed at improving its management of ongoing business
system investments, DOD had yet to (1) establish an investment
governance structure and process to align ongoing investments with its
architectural goals and direction, (2) establish and apply common
investment criteria to its ongoing IT system projects, and (3) conduct a
comprehensive review of its ongoing IT investments to ensure they are
consistent with architecture development efforts. We reiterated our earlier
recommendations and made six new recommendations to the Secretary of
Defense to assist DOD in successfully developing an enterprise
architecture and using it to gain control over its ongoing business system
investments. Of the six recommendations we made, DOD has partially
implemented two but has not yet implemented the remaining four.

In March 2003,18 we reported on DOD’s draft version of the BEA, dated
February 7, 2003, and concluded that it did not include a number of items
recommended by relevant architectural guidance and that DOD’s plans
would not fully satisfy the act’s requirements. For example, the draft
architecture did not include a “To Be” security view, which defines the
security requirements, including relevant standards to be applied in
implementing security policies, procedures, and controls. We did not make


17
     GAO-03-458.
18
  U.S. General Accounting Office, Information Technology: Observations on Department
of Defense’s Draft Enterprise Architecture, GAO-03-571R (Washington, D.C.: Mar. 28, 2003).




Page 15                                GAO-03-1018 DOD Business Enterprise Architecture
                              recommendations because this draft was a work in process that was
                              changing daily. However, DOD officials agreed with our preliminary
                              assessment of the architecture and stated that subsequent versions of the
                              architecture would provide these missing details.

                              See appendix III for details on the status of all our recommendations,
                              including our assessment of DOD’s actions.



DOD Has Taken                 DOD has made important progress in complying with the legislative
                              requirements to develop and effectively implement a well-defined
Positive First Steps in       enterprise architecture. The department has (1) elected an incremental
Complying with                approach to developing its architecture, (2) adopted some of the
                              architecture management capabilities advocated by best practices and
Enterprise                    federal guidance,19 such as designating a chief architect, and (3) developed
Architecture                  initial versions of architecture products that provide a foundation upon
Legislative                   which to build. Nevertheless, DOD’s initial architecture lacks sufficient
                              scope and content to fully satisfy legislative requirements, satisfy relevant
Requirements, but             architecture guidance, and make informed decisions about systems
Much Remains to be            investments. Moreover, DOD has yet to implement an effective investment
Accomplished                  management process to select and control ongoing and planned business
                              system investments.



DOD Is Following an           Our research and experience show that for major program investments,
Incremental Approach to       such as the development of an enterprise architecture, successful
                              organizations approach product development in an incremental fashion,
Developing Its Architecture   meaning that they initially develop a foundational product that is expanded
                              and extended through a series of follow-on products that add more
                              capability and value. In doing so, these organizations can effectively
                              mitigate the enormous risk associated with trying to deliver a large and
                              complex product that requires the execution of many activities over an
                              extended period of time as a single monolithic product. In effect, this
                              incremental approach permits a large undertaking to be broken into a
                              series of smaller projects, or incremental versions, that can be better
                              controlled to provide reasonable assurance that expectations are met.




                              19
                                   GAO-03-584G.




                              Page 16                          GAO-03-1018 DOD Business Enterprise Architecture
                            For its enterprise architecture development program, we have recognized
                            and told DOD that given its plans, capabilities, and status, it would not be
                            able to produce and approve a complete version of its architecture by May
                            1, 2003. Accordingly, we advised DOD to adopt an incremental approach to
                            developing and implementing its architecture and to represent its
                            architecture product to stakeholders as an initial version and to define its
                            plans for evolving and extending this initial version to satisfy the act and
                            create a well-defined blueprint. Recognizing these obstacles, DOD has
                            adopted an incremental approach. Specifically, DOD has designated its
                            architecture as version 1.0 and has committed to building on this in
                            producing subsequent versions. According to DOD, the next significant
                            delivery of the BEA is currently planned for May 2004.



DOD Has Recently Adopted    Effective process controls for managing architecture development,
Some, but Not All Key       maintenance, and implementation are recognized hallmarks of successful
                            public and private organizations. According to guidance published by the
Elements of Architecture    federal CIO Council,20 effective architecture management consists of a
Management Best Practices   number of practices, conditions, and structures. In April 2003, we
                            published version 1.1 of our enterprise architecture management maturity
                            framework, which arranges the core elements of the CIO Council’s
                            guidance into five hierarchical stages.21 The framework provides an
                            explicit benchmark for gauging the effectiveness of architecture
                            management and provides a roadmap for making improvements. Table 2
                            summarizes the framework’s five stages of maturity.




                            20
                              Chief Information Officer Council, A Practical Guide to Federal Enterprise Architecture,
                            Version 1.0 (February 2001).
                            21
                                 GAO-03-584G.




                            Page 17                                GAO-03-1018 DOD Business Enterprise Architecture
Table 2: Summary of GAO’s Enterprise Architecture (EA) Maturity Framework Stages

Stage                                Description
Stage 1: Creating EA awareness       Organization does not have plans to develop and use an architecture, or it has plans that do not
                                     demonstrate an awareness of the value of having and using an architecture. While stage 1
                                     agencies may have initiated some EA activity, these agencies’ efforts are ad hoc and
                                     unstructured, lack institutional leadership and direction, and do not provide the management
                                     foundation necessary for successful EA development.
Stage 2: Building the EA             Organization recognizes that the EA is a corporate asset by vesting accountability for it in an
management foundation                executive body that represents the entire enterprise. At this stage, an organization assigns EA
                                     management roles and responsibilities and establishes plans for developing EA products and for
                                     measuring program progress and product quality; it also commits the resources necessary for
                                     developing an architecture—people, processes, and tools.
Stage 3: Developing the EA           Organization focuses on developing architecture products according to the selected framework,
(includes all elements in stage 2)   methodology, tool, and established management plans. Roles and responsibilities assigned in
                                     the previous stage are in place, and resources are being applied to develop actual EA products.
                                     The scope of the architecture has been defined to encompass the entire enterprise, whether
                                     organization-based or function-based.
Stage 4: Completing the EA           Organization has completed its EA products, meaning that the products have been approved by
(includes all elements in stage 3)   the EA steering committee or an investment review board, and by the CIO. Further, an
                                     independent agent has assessed the quality (i.e., completeness and accuracy) of the EA
                                     products. Additionally, evolution of the approved products is governed by a written EA
                                     maintenance policy approved by the head of the organization.
Stage 5: Leveraging the EA to        Organization has secured senior leadership approval of the EA products and a written
manage change                        institutional policy stating that IT investments must comply with the architecture, unless granted
(includes all elements in stage 4)   an explicit compliance waiver. Further, decision makers are using the architecture to identify and
                                     address ongoing and proposed IT investments that are conflicting, overlapping, not strategically
                                     linked, or redundant. Also, the organization tracks and measures EA benefits or return on
                                     investment, and adjustments are continuously made to both the EA management process and
                                     the EA products.
Source: GAO.


                                            The state of DOD’s implementation of key enterprise architecture
                                            management practices, conditions, and structures currently places it at
                                            stage 1 of our maturity framework. Specifically, it has satisfied about 80
                                            percent of the core elements associated with building the enterprise
                                            architecture management foundation—stage 2 of our framework—but only
                                            about 41 percent (9 of the 22) of the core elements associated with stages
                                            3, 4, and 5. According to our framework, effective architecture
                                            management is generally not achieved until an enterprise has a completed
                                            and approved architecture that is being effectively maintained and is being
                                            used to leverage organizational change and support investment decision
                                            making; having these characteristics is equivalent to having satisfied all
                                            stage 3 core elements and many stage 4 and 5 elements.




                                            Page 18                                 GAO-03-1018 DOD Business Enterprise Architecture
With respect to stage 2 core elements, DOD has, for example, established a
program office, assigned a chief architect, and selected a framework
(C4ISR) and an automated tool (e.g., the System Architect by Popkin
Software). However, the department has not satisfied two of the stage 2
core elements that are critical to effective enterprise architecture
management. For example, a committee or group representing the
enterprise has not yet been established to guide, direct, and approve the
architecture. Instead, the current version of its architecture has been
guided and directed by the Business Modernization and Systems
Integration (BMSI) program office. Although the Secretary of Defense has
established Financial Management Modernization Executive and Steering
Committees for the enterprise architecture, which are made up of senior
leaders from across the department, to provide program guidance, these
committees are not accountable for approving the architecture. Instead,
the responsibility of each committee is limited to providing guidance to the
BMSI program office and advising the DOD Comptroller. However, DOD
officials told us that the Executive Committee has approved the
architecture; yet there were no minutes of the Executive Committee
documenting this decision. Without an accountable corporate entity to
lead the architectural effort, there is increased risk that the architecture
will not represent a corporate decision-making tool and will not be viewed
and endorsed as a departmentwide asset.

Further, DOD does not have a written and approved policy for architecture
development, which is a stage 3 core element. Without such a policy that,
for example, identifies the major players in the development process and
provides for architecture guidance, direction, and approval, DOD has been,
and will continue to be, challenged in achieving departmentwide
architecture commitment and support.

The department also has yet to implement numerous stage 4 and 5 core
elements. For example, DOD has not (1) documented and approved a
policy for architecture maintenance, (2) fully implemented an independent
verification and validation function that covers architecture products and
architecture management processes, and (3) made the architecture an
integral component of its IT investment management process.

According to program officials, the department set overly optimistic
expectations and time frames for its enterprise architecture program,
which resulted in the need to establish architecture management process
controls concurrent with developing architecture products. Until the
department implements the core elements of our enterprise architecture



Page 19                          GAO-03-1018 DOD Business Enterprise Architecture
                                                  management maturity framework, it is unlikely that it will be able to either
                                                  produce and maintain a well-defined architecture or effectively implement
                                                  what is produced.

                                                  Table 3 provides the results of our assessment of DOD’s satisfaction of
                                                  each of the core elements of our maturity framework.



Table 3: Assessment of DOD’s Enterprise Architecture Efforts Against GAO’s Enterprise Architecture Maturity Framework

Stage                   Core element                                Satisfied?    Comments
Stage 1: EA             Agency is aware of EA.                     Yes            In July 2001, the Secretary of Defense issued a
Awareness                                                                         memorandum directing the development and
                                                                                  implementation of a departmentwide enterprise
                                                                                  architecture.
Stage 2: Building the   Adequate resources exist (funding,         Yes            According to DOD, it has adequate program
EA Management           people, tools, and technology).                           funding. It requested approximately $196.3 million
Foundation                                                                        for the program and received about $186.8 million.
                                                                                  Further, it reports that it has skilled staff
                                                                                  (government employees and contractors) for its
                                                                                  architecture program. In addition, DOD is using
                                                                                  automated tools and technology, such as System
                                                                                  Architect by Popkin Software and the Dynamic
                                                                                  Object-Oriented Requirements System by
                                                                                  Telelogic.
                        Committee or group representing the        No             DOD has not assigned responsibility for directing,
                        enterprise is responsible for directing,                  overseeing, and approving the EA to a committee
                        overseeing, and approving the EA.                         or group comprised of representatives from across
                                                                                  the department.
                        Program office responsible for EA          Yes            DOD has established a program office that is
                        development and maintenance exists.                       responsible for EA development and maintenance.
                        Chief architect exists.                    Yes            DOD has assigned a chief architect.
                        EA is being developed using a              Yes            The EA is being developed using DOD’s C4ISR
                        framework, methodology, and automated                     architecture framework and the Federal Enterprise
                        tool.                                                     Architecture Program Management Office
                                                                                  Reference Models. Further, DOD has a
                                                                                  methodology that adapts its development
                                                                                  contractor’s architecture methodology to the C4ISR
                                                                                  framework. In addition, DOD is using automated
                                                                                  tools, such as System Architect by Popkin
                                                                                  Software, to build the EA, and Dynamic Object-
                                                                                  Oriented Requirements System by Telelogic, as
                                                                                  the repository for requirements information.
                        EA plans call for describing both the “As Yes             EA plans call for describing the “As Is” and the “To
                        Is” and the “To Be” environments of the                   Be” environments, and a sequencing plan.
                        enterprise, as well as a sequencing plan
                        for transitioning from the “As Is” to the “To
                        Be.”



                                                  Page 20                          GAO-03-1018 DOD Business Enterprise Architecture
(Continued From Previous Page)
Stage                 Core element                               Satisfied?   Comments
                      EA plans call for describing both the “As Yes           EA plans call for describing both the “As Is” and the
                      Is” and the “To Be” environments in terms               “To Be” environments in terms of business,
                      of business, performance,                               performance, information/data, application/service,
                      information/data, application/service, and              and technology.
                      technology.
                      EA plans call for business, performance, No             According to DOD, EA plans will not address
                      information/data, application/service, and              security for the “As Is” environment, but will
                      technology descriptions to address                      address security for the “To Be” environment.
                      security.
                      EA plans call for developing metrics for   Yes          EA plans call for developing EA metrics for
                      measuring EA progress, quality,                         measuring progress, quality, compliance, and
                      compliance, and return on investment.                   return on investment.
Stage 3: Developing Written/approved organization policy         No           DOD has a policy for developing the Global
EA Products (includes exists for EA development.                              Information Grid (GIG),a which requires that all
all elements from                                                             other departmental architectures be in alignment
stage 2)                                                                      with the GIG. While this policy outlines the roles
                                                                              and responsibilities for development, maintenance,
                                                                              and implementation of the GIG, it does not do so
                                                                              for other architectures, such as the BEA. Such
                                                                              policy should describe, for example, the scope of
                                                                              the BEA and the processes for BEA oversight,
                                                                              control, and validation. The policy should also
                                                                              identify the major players in the development
                                                                              process, including the chief architect, steering
                                                                              committee, and CIO. The GIG policy does not
                                                                              provide this information for the BEA.
                      EA products are under configuration        Yes          Configuration of EA products is being managed by
                      management.                                             an automated tool.
                      EA products describe or will describe      Yes          According to plans, EA products will describe the
                      both the “As Is” and the “To Be”                        “As Is” and the “To Be” environments, as well as a
                      environments of the enterprise, as well as              sequencing plan.
                      a sequencing plan for transitioning from
                      the “As Is” to the “To Be.”
                      Both the “As Is” and the “To Be”           Yes          According to plans, the “As Is” and the “To Be”
                      environments are described or will be                   environments will be described in terms of
                      described in terms of business,                         business, performance, information/data,
                      performance, information/data,                          application/service, and technology.
                      application/service, and technology.
                      Business, performance, information/data, No             According to DOD, descriptions will not address
                      application/service, and technology                     security for the “As Is” environment, but will
                      descriptions address or will address                    address security for the “To Be” environment.
                      security.
                      Progress against EA plans is measured      Yes          DOD is measuring and reporting progress against
                      and reported.                                           EA plans. For example, the verification and
                                                                              validation contractor reports on the quality of EA
                                                                              products, and the development contractor reports
                                                                              weekly on its progress (cost, schedule,
                                                                              performance).




                                            Page 21                            GAO-03-1018 DOD Business Enterprise Architecture
(Continued From Previous Page)
Stage                    Core element                               Satisfied?   Comments
Stage 4: Completing      Written/approved organization policy       No           There is no written/approved policy for EA
EA Products              exists for EA maintenance.                              maintenance.
(includes all elements
from stage 3)
                         EA products and management processes No                 EA products undergo verification and validation;
                         undergo independent verification and                    however, as we previously reported,b this function
                         validation.                                             is not independent of the program office. Further,
                                                                                 management processes are not verified and
                                                                                 validated.
                         EA products describe both the “As Is” and No            Version 1.0 of the EA describes the “As Is” and the
                         the “To Be” environments of the                         “To Be” environments of the enterprise, as well as
                         enterprise, as well as a sequencing plan                a sequencing plan; however, as discussed later in
                         for transitioning from the “As Is” to the “To           this report, this version is missing needed scope
                         Be.”                                                    and detail.
                         Both the “As Is” and the “To Be”         No             Version 1.0 of the EA describes the “As Is” and the
                         environments are described in terms of                  “To Be” environments of the enterprise, as well as
                         business, performance, information/data,                a sequencing plan; however, as discussed later in
                         application/service, and technology.                    this report, this version is missing needed scope
                                                                                 and detail.
                         Business, performance, information/data, No             Version 1.0 of the EA describes the “As Is” and the
                         application/service, and technology                     “To Be” environments of the enterprise, as well as
                         descriptions address security.                          a sequencing plan; however, as discussed later in
                                                                                 the report, this version is missing needed scope
                                                                                 and detail. Further, according to DOD, security will
                                                                                 not be addressed for the “As Is” environment.
                         Organization CIO has approved current      Yes          The CIO is a member of the executive committee.
                         version of EA.                                          According to a DOD program official, the executive
                                                                                 committee has approved the EA.
                         Committee or group representing the       Yes           According to a DOD program official, the executive
                         enterprise or the investment review board               committee has approved the EA.
                         has approved current version of EA.
                         Quality of EA products is measured and     Yes          The verification and validation contractor reviews
                         reported.                                               and reports on the quality of EA products.
Stage 5: Leveraging      Written/approved organization policy       No           There is no written/approved policy requiring IT
the EA for Managing      exists for IT investment compliance with                investment compliance with the EA.
Change                   EA.
(includes all elements
from stage 4)
                         Process exists to formally manage EA       No           There is no defined process for managing changes
                         change.                                                 to the EA.
                         EA is integral component of IT investment No            The EA is not being used to make investment
                         management process.                                     selection and control decisions, and thus is not
                                                                                 part of the investment management process.
                         EA products are periodically updated.      Yes          According to DOD, it plans to periodically update
                                                                                 the EA; the next significant version is to be issued
                                                                                 in May 2004. An initial version (1.0) was issued in
                                                                                 May 2003.




                                               Page 22                            GAO-03-1018 DOD Business Enterprise Architecture
(Continued From Previous Page)
Stage                               Core element                               Satisfied?         Comments
                                    IT investments comply with EA.             No                 The EA is not being used to select and control
                                                                                                  investments, and thus investments do not comply
                                                                                                  with the architecture.
                                    Organization head has approved current     Yes                According to a program official, the Secretary of
                                    version of EA.                                                Defense orally delegated approval authority to the
                                                                                                  DOD Comptroller, who approved version 1.0 of the
                                                                                                  EA.
                                    Return on EA investment is measured        No                 Metrics and processes for measuring EA benefits
                                    and reported.                                                 have not been developed, thus precluding return
                                                                                                  on investment measurement. DOD is not capturing
                                                                                                  the full cost of its EA investment—no cost
                                                                                                  information for fiscal year 2001 exists and actual
                                                                                                  government salary expenses are unknown. As of
                                                                                                  May 28, 2003, the department had contractually
                                                                                                  obligated about $116 million and disbursed about
                                                                                                  $48.5 million.
                                    Compliance with EA is measured and         No                 Metrics for measuring EA compliance have not
                                    reported.                                                     been developed, thus precluding measuring and
                                                                                                  reporting on compliance.
Source: GAO analysis of DOD data.
                                                        a
                                                         DOD defines the GIG as the globally interconnected, end-to-end set of information, capabilities,
                                                        associated processes, and personnel for collecting, processing, storing, disseminating, and managing
                                                        information on demand to warfighters, policy makers, and support personnel.
                                                        b
                                                         GAO-03-458.




Initial Version of                                      DOD has expended tremendous effort and resources and made important
Architecture Provides a                                 progress in complying with the legislative requirements aimed at
                                                        developing and implementing a well-defined enterprise architecture.
Foundation Upon Which to                                Further, DOD’s initial version of its BEA provides a foundation from which
Build                                                   to build and ultimately produce a well-defined business enterprise
                                                        architecture. However, the initial architecture does not adequately address
                                                        the act’s requirements and other relevant architectural requirements.22
                                                        Specifically, the initial version of the architecture does not adequately
                                                        describe the accounting and financial management requirements, and the


                                                        22
                                                          See for example, Office of Management and Budget, Federal Enterprise Architecture
                                                        Business Reference Model, Version 1.0 (2002); Chief Information Officer Council, A
                                                        Practical Guide to Federal Enterprise Architecture, Version 1.0 (February 2001); Office of
                                                        Management and Budget Circular A-130, Management of Federal Information Resources
                                                        (Nov. 28, 2000); M.A. Cook, Building Enterprise Information Architectures: Reengineering
                                                        Information Systems (Upper Saddle River, N.J., Prentice Hall: 1996); and National Institute
                                                        of Standards and Technology, Information Management Directions: The Integration
                                                        Challenge, Special Publication 500-167 (September 1989).




                                                        Page 23                                    GAO-03-1018 DOD Business Enterprise Architecture
                              “As Is” and “To Be” environments and the transition plan are not
                              sufficiently complete to provide a basis for guiding and constraining
                              investment decisions.

Architecture Does Not         DOD elicited and incorporated about 4,000 external requirements in its “To
Adequately Address Federal    Be” architecture from 152 federal sources. Our review of 1,767 of the
Requirements and Accounting   external requirements—specifically those that were elicited from the Joint
Standards                     Financial Management Improvement Program (JFMIP) 23—identified 340
                              JFMIP requirements (about 19 percent) that were not adequately addressed
                              in version 1.0 of the “To Be” architecture. Specifically, DOD (1) omitted
                              some JFMIP requirements that are significant, (2) included some that are
                              invalid, and (3) included some that are not appropriate to its business
                              operations.

                              Mission requirements are one of the key bases for determining the scope
                              and content for enterprise architectures. One source of requirements is the
                              legal, regulatory, and other external constraints that define the
                              environment within which an enterprise, such as DOD, must operate. If a
                              given architecture is not developed to adequately recognize these
                              constraints, and these missing constraints are significant, the architecture
                              will not provide a sufficient frame of reference for informed decision
                              making. The act specifies that the architecture should enable DOD to
                              comply with all federal accounting, financial management, and reporting
                              requirements. JFMIP requirements arise from various public laws,
                              regulations, bulletins, circulars, federal accounting standards, and leading
                              practices and are applicable government wide. Agencies must use these
                              requirements, in addition to agency-unique mission requirements, in
                              planning and implementing their financial management improvement
                              projects.

                              DOD’s “To Be” architecture omitted significant JFMIP requirements. For
                              example, DOD’s architecture did not include any relevant revenue
                              requirements. These requirements are significant to DOD because they
                              affect the accounting of and reporting for DOD’s revenue, which include at
                              least $70 billion earned annually by DOD working capital fund activities.
                              Department and contractor officials agreed that these system requirements


                              23
                                JFMIP is a joint undertaking of the Department of the Treasury, GAO, the Office of
                              Management and Budget, and the Office of Personnel Management, working with each
                              other, other agencies, and the private sector to improve financial management in the federal
                              government. JFMIP issues a series of requirements in support of agency operations.




                              Page 24                                 GAO-03-1018 DOD Business Enterprise Architecture
were either excluded or not adequately addressed and stated that a
subsequent version of the architecture would include or modify the
requirements. Table 4 summarizes the JFMIP requirements that we
reviewed and the numbers of missing or incomplete requirements we
identified.



Table 4: Summary of GAO Analysis of JFMIP Requirements

                                                        Number of        Percent of
                                         Number         missing or       missing or
                                        of JFMIP       incomplete       incomplete
JFMIP requirements                  requirements     requirements     requirements
Revenue                                       220              220              100
Acquisition                                   112               10                9
Core Financial                                430                 4               1
Human Resources and Payroll                   203               37               18
Managerial Cost Accounting                     67               30               45
Inventory                                     141                 7               5
Travel                                        166               12                7
Property Management                            78                 0               0
Benefit                                       350               20                6
Total                                       1,767              340               19
Source: GAO analysis of DOD data.




Page 25                             GAO-03-1018 DOD Business Enterprise Architecture
As another example, the “To Be” architecture did not include requirements
governing accounting for and reporting of national defense plant, property,
and equipment (PP&E)24 that became valid shortly before the architecture
was approved. These requirements significantly affect the accounting of
and reporting requirements for a reported 600,000 aircraft, ships, combat
vehicles, missiles and other weapons systems, and related equipment. The
architecture did not incorporate requirements for these accounting
standards25 even though (1) the Federal Accounting Standards Advisory
Board and the three sponsoring agencies26 responsible for federal
accounting standards approved them in October 2002 and (2) DOD already
recognized these new PP&E requirements in its fiscal year 2002
Performance and Accountability Report and has begun to implement
them. According to DOD and contractor officials, they used outdated
requirements because the new standard was not in effect when they were
identifying and linking national defense PP&E requirements to activities,
processes, and entities depicted by the architecture. As a result, DOD must
now revise its architecture to reflect these requirements, activities, and
processes to ensure compliance with the new accounting standard.

Lastly, the architecture includes options for doing business at the federal
level that were not appropriate to DOD’s business operations (i.e., do not
reflect external requirements constraining DOD’s operations). For
example, Statement of Federal Financial Accounting Standards (SFFAS)
No. 3, Accounting for Inventory and Related Property, requires operating
materials and supplies to be primarily accounted for using the consumption
method and allows the purchases method to be used as an exception only
under certain conditions.27 Because DOD’s operating supplies and
materials are considered significant, DOD reports the value of its almost


24
 National Defense PP&E assets include weapons systems and support PP&E owned by
DOD or its component entities for use in the performance of military missions.
25
 SFFAS No. 23, Eliminating the Category National Defense Property, Plant, and
Equipment, May 2003.
26
 The three sponsoring agencies are the Department of the Treasury, the Office of
Management and Budget, and the General Accounting Office.
27
  The consumption method of accounting requires operating materials and supplies to be
capitalized when purchased and reported as an operating expense when they are consumed.
Under the purchases method, operating materials and supplies are reported as an operating
expense when they are purchased if their amounts are insignificant, they are in the hands of
the end user for use in normal operations, or it is not cost effective to apply the
consumption method.




Page 26                                 GAO-03-1018 DOD Business Enterprise Architecture
                                   $91 billion of operating materials and supplies using the consumption
                                   method of accounting. Developing the architecture to allow DOD to use
                                   the purchases method to account for its inventory and related property
                                   may result in inappropriate use of this method and, therefore, inconsistent
                                   practices and supporting system solutions among DOD components.

                                   According to DOD and contractor officials, both the omission and limited
                                   definition of relevant federal requirements is partly due to not having a fully
                                   functioning quality assurance process to verify and validate the
                                   requirements when the requirements were elicited. In March 2003,
                                   following our previous recommendation to strengthen quality assurance
                                   activities, DOD increased its quality assurance activities. These officials
                                   stated that, as part of their current quality assurance process, they are
                                   identifying additional requirements and deleting existing requirements that
                                   are duplicative or deemed not mandatory. As a result, version 1.0 of the
                                   BEA does not adequately reflect and recognize critical external
                                   requirements, and thus is not yet sufficiently complete for making informed
                                   decisions about system investments.

Initial Version of DOD             As previously discussed, the various frameworks used to develop an
Architecture Is Not Sufficiently   enterprise architecture consistently (1) describe the architectures for both
Complete to Satisfy Act and        the enterprise’s “As Is” and “To Be” environments in logical (e.g., business,
Guide and Constrain                performance, application, information) as well as technical (e.g., hardware,
Modernization Investments          software, data) terms, and (2) define a capital investment sequencing plan
                                   to transition from the “As Is” to the “To Be” environment. However, the
                                   frameworks do not prescribe the degree to which the component parts
                                   should be described to be considered correct, complete, understandable,
                                   and usable—essential attributes of any architecture. This is because the
                                   depth and detail of the descriptive content depend on the architecture’s
                                   intended purpose.

                                   In the case of DOD, the act specifies that its enterprise architecture should
                                   enable the department to (1) comply with all federal accounting, financial
                                   management, and reporting requirements, (2) routinely produce timely,
                                   accurate, and reliable financial information for management purposes,
                                   (3) integrate budget, accounting, and program information and systems,
                                   and (4) provide for the systematic measurement of performance.
                                   Moreover, DOD’s stated intention is to use the architecture as the basis for
                                   departmentwide business transformation and systems modernization.




                                   Page 27                           GAO-03-1018 DOD Business Enterprise Architecture
Collectively, these purposes necessitate that the architecture provide
considerable depth and detail, as well as logical and rational structuring
and internal linkages. More specifically, they mean that the architecture
should contain sufficient scope and detail to permit, for example,
(1) elimination of duplicative business operations and systems,
(2) standardization and integration of business operations and
interoperability of supporting systems, (3) maximum use of enterprisewide
services, and (4) alignment with related shared solutions, like OMB’s e-gov
initiatives. Moreover, this scope and detail28 should be accomplished in a
way that (1) provides flexibility in adapting to changes in the enterprise’s
internal and external environments, (2) facilitates its usefulness and
comprehension by varying perspectives/users/stakeholders, and
(3) provides for properly sequencing investments to recognize, for
example, the investments’ respective dependencies and relative business
value.

Version 1.0 of DOD’s enterprise architecture does not contain sufficient
scope and detail to either satisfy the act’s requirements or effectively guide
and constrain departmentwide business transformation and systems
modernization. This is based on our decomposition of version 1.0 into
various parts and components and comparison of it against relevant
benchmarks. More specifically, we first divided the architecture into the
three primary component parts specified in the act and recognized in best
practices and federal guidance: the “As Is” architecture, the “To Be”
architecture, and the transition plan. We then divided the “As Is” and the
“To Be” architectures into five architectural components similar to those in
OMB’s architecture reference models: the business, information/data,
services/applications, technical, and performance components; we added
security as a sixth component because of its recognized importance and
relevance to the other five. We then compared version 1.0 to relevant




28
  Subsection 1004(b)(2) of the act also specifies that DOD’s enterprise architecture shall
“include policies, procedures, data standards, and system interface requirements that are to
apply uniformly throughout the Department.”




Page 28                                 GAO-03-1018 DOD Business Enterprise Architecture
criteria29 governing the content of key architectural elements for the
transition plan and these six components of the “As Is” and “To Be”
architectures. Based on this comparison, we determined whether version
1.0 of the architecture generally satisfied, did not satisfy,30 or partially
satisfied31 each architectural element.

Overall, DOD’s “As Is” architecture did not satisfy 26 of 29 key elements,
and partially satisfied the remaining 3; its “To Be” architecture did not
satisfy 4 of 30 key elements, and partially satisfied the remaining 26; and its
transition plan partially satisfied 2 of the 3 elements, but did not satisfy the
remaining 1 (see fig. 2). This means that version 1.0 of DOD’s enterprise
architecture does not satisfy the requirements of the act and is not
sufficiently complete to provide an adequate basis for guiding and
constraining investments in modernized systems. Program officials agreed
that this version of the architecture is not complete, but stated that it fully
satisfies the act, because it addresses, to at least some degree, each of the
act’s requirements. They added that they have accomplished as much in
completing the architecture as was possible in the time available, and that
the department was overly optimistic in estimating what it could produce
by May 1, 2003. We agree that DOD set unrealistic expectations of the
scope and level of architectural definition it could provide by this time, but
do not agree, as discussed in detail in the previous section and the sections
that follow, that it satisfies the act’s requirements. We also believe that the
state of DOD’s architecture is also due to weaknesses in its architecture
development management controls that are discussed in the previous
section, and that our prior recommendations were aimed at correcting.




29
  See for example, Office of Management and Budget, Federal Enterprise Architecture
Business Reference Model, Version 1.0 (2002); Chief Information Officer Council, A
Practical Guide to Federal Enterprise Architecture, Version 1.0 (February 2001); Office of
Management and Budget, Management of Federal Information Resources, Circular No. A-
130 (Nov. 28, 2000); Cook, M.A., Building Enterprise Information Architectures:
Reengineering Information Systems (Prentice Hall Inc.: 1996); and National Institute of
Standards and Technology, Information Management Directions: The Integration
Challenge, Special Publication 500-167 (September 1989).
30
     The architecture does not satisfy any aspects of this key architectural element.
31
 The architecture partially satisfies some aspects of this key architectural element but does
not satisfy at least one significant aspect.




Page 29                                    GAO-03-1018 DOD Business Enterprise Architecture
Figure 2: Summary of Extent to Which Version 1.0 of DOD’s Enterprise Architecture Satisfies Key Elements Governing
Architectural Content

                                       Yes                  No          Partially                Total
           "As Is" environment           0                  26                  3                  29
           "To Be" environment           0                   4                26                   30
           Transition plan               0                   1                  2                    3

                    "As Is"                     "To Be"                             Transition
                  environment                 environment                              plan

                          10%
                                                    13%
                                                                                             33%


                      90%                         87%                                  67%



                     No

                     Partially

Source: GAO analysis of DOD data.


                                         In addition, the structure of the “To Be” architecture contains internal
                                         linkage, consistency, and navigation limitations that constrain its ease of
                                         use and understandability. For example, explicit linkages among
                                         (1) services/applications, (2) organizations using the services/applications,
                                         and (3) technical standards governing the services/applications were
                                         missing, as were linkages between certain business and information/data
                                         artifacts. This is important because dependencies exist among these
                                         architectural components and a well-defined architecture makes such
                                         dependencies explicit to ensure that systems are implemented in a
                                         nonduplicative and integrated fashion. We also found instances of internal
                                         inconsistencies. For example, one artifact (a table) indicated that DOD had
                                         not selected any standards for certain security services, while another
                                         artifact identified selected standards for these services. In another
                                         instance, four artifacts listed identified requirement sources for security,
                                         such as the Computer Security Act of 198732 and OMB Circular A-130;
                                         however, the artifacts’ respective lists varied and no single list included all
                                         the requirement sources. In another instance, some terms contained within


                                         32
                                              The Computer Security Act of 1987, Pub. L. No. 100-235, 101 Stat. 1724, Jan. 8, 1988.




                                         Page 30                                    GAO-03-1018 DOD Business Enterprise Architecture
the architecture (e.g., availability, integrity checks, confidentiality, and
authentication) were not consistently defined, and the architecture did not
explain the basis for these inconsistencies. For example, one artifact
defined authentication as the “standard practices followed to authenticate
the identity of system users,” while another artifact defined it as a “security
measure designed to establish the validity of a transmission, message, or
originator, or a means of verifying an individual’s authorization to receive
specific categories of information.”

Such inconsistencies in the architecture can in turn lead to
misinterpretations, and thus incompatibilities, in how systems are
implemented. Additionally, the architecture did not include user
instructions or guidance, making it difficult to navigate and use. For
example, (1) certain artifacts (e.g., diagrams) could not be read on-line
because there was no “zoom” capability enabling enlargement, and
(2) specific content, such as the applicability of security standards to
specific security services, took three persons several days to locate. The
complete results of our analysis of each of version 1.0’s parts and related
components are discussed in detail below.

“As Is” Architecture: This architecture is intended to capture the current
state of enterprise operations in sufficient scope and detail to permit
meaningful analysis of gaps between such things as current and future
processes, data, standards, and systems. It thus should describe, for those
areas of the enterprise that are likely to change, the current set of business
processes and performance measures that are in place and operating and,
among other things, the information/data, services/applications, and
technology that are in place to support these processes and measures.
According to relevant guidance,33 the “As Is” architecture should contain,
for example, a description of (1) the actual business operations currently
being performed to support the organization’s mission, including the
entities/people that perform the functions, processes, and activities, and
the locations where the functions, processes, and activities are being


33
  See for example, Office of Management and Budget, Federal Enterprise Architecture
Business Reference Model, Version 1.0 (2002); Chief Information Officer Council, A
Practical Guide to Federal Enterprise Architecture, Version 1.0 (February 2001); Office of
Management and Budget, Management of Federal Information Resources, Circular No. A-
130 (Nov. 28, 2000); Cook, M.A., Building Enterprise Information Architectures:
Reengineering Information Systems (Prentice Hall Inc.: 1996); and National Institute of
Standards and Technology, Information Management Directions: The Integration
Challenge, Special Publication 500-167 (September 1989).




Page 31                                GAO-03-1018 DOD Business Enterprise Architecture
performed, (2) the information/data used by the functions, processes, and
activities, (3) the systems that support the functions, processes, and
activities, including system interfaces, (4) the types of technology (e.g.,
hardware and software) and associated technical standards that comprise
the physical systems environment, (5) the security standards and tools
used to secure and protect systems and data, and (6) the metrics for
evaluating the effectiveness of mission operations and supporting system
performance in achieving mission goals and objectives. Without this
information, an enterprise would be extremely challenged in identifying the
proper sequence of changes needed to move from its current operating
environment to its future, target environment. As stated by one leading
industry authority on enterprise architecture,34 an organization will be
unable to effectively plan and manage its modernization efforts, and will
waste IT dollars, if it does not have a clear picture of its “As Is”
environment.

Version 1.0 of DOD’s “As Is” architecture provides little of this descriptive
content. On the positive side, it includes an inventory of about 2,300
existing systems that support DOD’s current business operations, including
certain characteristics about each (e.g., system owner, purpose and
business process it supports, and vendor). However, the majority of
systems do not have descriptions of system interfaces, and the inventory
has not been validated and continues to change significantly. For example,
DOD’s current “As Is” systems inventory of about 2,300 systems has
increased approximately 35 percent when compared to its “As Is” inventory
of about 1,700 business systems in October 2002. In addition, DOD’s
architecture does not describe (1) the current business operations in terms
of the entities/people who perform the functions, processes, and activities,
and the locations where the functions, processes, and activities are
performed, (2) the data/information being used by the functions, processes,
and activities, (3) the types of technology and associated technical
standards being employed, (4) the security standards and tools being used,
and (5) the performance metrics being used. Instead, it merely provides a
listing of the names of the current business processes. As a result, DOD
does not have a sufficiently described picture of its current environment to
permit development of a meaningful and useful transition plan. See table 5
for the detailed results of our analysis of DOD’s “As Is” architecture.




34
     Troux Technologies, Inc. http://www.eacommunity.com/sponsor/troux_061903.htm.




Page 32                                GAO-03-1018 DOD Business Enterprise Architecture
Table 5: Detailed Analysis of the Extent to Which DOD’s “As Is” Architecture Satisfies Key Elements

                                                                Element satisfied?
Key architectural element                                 Yes       No      Partially   Explanation of partially satisfied
Business
A description of the strategic goals, which defines                 X
what an organization wants to achieve.
A business strategy, which defines how the strategic                X
goals and objectives will be achieved.
An inventory of key policies, procedures, and                       X
standards governing how business operations are
executed and managed.
A description of key business processes and how                             X           The architecture contains (1) a list of the names of
they support the agency’s mission, including the                                        the business processes and (2) a high-level (1-
organizational units responsible for performing the                                     page) graphical depiction of these processes. It
business processes and the locations where the                                          does not contain detailed descriptions of existing
business processes are being performed.                                                 business operations that include, for example,
                                                                                        information flows among activities, organizational
                                                                                        units and locations that perform the business
                                                                                        processes, and the technological characteristics of
                                                                                        the systems that perform these processes.
An analysis of deficiencies in the “As Is” business                         X           The architecture contains an analysis of process
environment that are to be addressed, as well as                                        area deficiencies. However, this analysis does not
obstacles to addressing these deficiencies, plans for                                   include the business case(s) for addressing the
addressing them, and a business case for addressing                                     deficiencies.
them. An example is an analysis of the quality of
existing data to determine their completeness and
accuracy.
A description of organizational accountability for                  X
execution of current business policies, procedures,
and standards.
Information/Data
A description of the data management policies,                      X
processes, procedures, and tools (e.g., CRUD
Matrixa) for analyzing, designing, building, and
maintaining existing databases.
A description of the business and operational rulesb                X
for data standardization to ensure data consistency,
integrity, and accuracy, such as business and security
rules that govern access, maintenance, and use of
data.
A data dictionary, which is a repository of standard                X
data definitions for applications.




                                                Page 33                                  GAO-03-1018 DOD Business Enterprise Architecture
(Continued From Previous Page)
                                                                 Element satisfied?
Key architectural element                                  Yes       No      Partially   Explanation of partially satisfied
A conceptual data model that describes the                           X
fundamental things/objects (e.g., invoices, financial
statements, inventory) that make up the business and
how they are used, but without regard for how they
will be physically stored. It represents the
consolidated structure of business objects to be used
by business applications.
A logical database model that provides the data                      X
structures that support information flows, and that
provides the basis for developing the schemas for
designing, building, and maintaining the existing
physical databases.
A metadatac model that specifies the rules and                       X
standards for access to information.
A description of information flows and relationships                 X
between organizational units, business operations,
and applications.
Services/Applications
A stable listing of business application systems and                         X           There is a list of systems, but the respective
system components and their interfaces.                                                  interfaces have not been described. Further, this
                                                                                         list continues to change.
A description of the common technical approach,                      X
policies, and procedures for developing/acquiring
business application systems throughout their life
cycle, including requirements management, design,
implementation, testing, deployment, operations, and
maintenance. The common technical approach
should also describe the process for integrating
legacy systems with the systems to be
developed/acquired.
Technical
Descriptions of the enterprise infrastructure servicesd              X
to include specific details regarding the functionality
and capabilities that these services provide to enable
systems applications.
Identification of the technical standardse implemented               X
for each enterprise service.
A description of the physical IT infrastructure needed               X
to support the current and any newly developed
and/or acquired systems outside the scope of the
architecture, including the relationships among
hardware, software, and communications devices.




                                                 Page 34                                  GAO-03-1018 DOD Business Enterprise Architecture
(Continued From Previous Page)
                                                                   Element satisfied?
Key architectural element                                    Yes       No       Partially    Explanation of partially satisfied
Common policies and procedures for                                     X
developing/acquiring infrastructure systems
throughout their life cycle, including requirements
management, design, implementation, testing,
deployment, operations, and maintenance.
Security
A description of the policies, procedures, goals,                      X
strategies, and requirements relevant to information
assurance and security.
A listing of security and information assurance                        X
related terms.
A listing of accountable organizations and their                       X
respective responsibilities for implementing
enterprise security services.
A description of operational security rules that are                   X
derived from security policies.
A description of enterprise security infrastructure                    X
services (e.g., identification and authentication)
needed to protect the department’s assets, and the
means for implementing such a service (e.g.,
firewalls and intrusion detection software).
A description of the security standardsf implemented                   X
for each enterprise service to secure assets. These
standards should be derived from security
requirements.
A description of the protection mechanisms                             X
implemented to secure the department’s assets, such
as firewalls and intrusion detection software,
including a description of the relationships among
these protection mechanisms.
Performance
A description of the performance management                            X
process, including how the organization measures,
tracks, evaluates, and predicts business performance
with respect to business functions, baseline data, and
service levels.
A description of customer-focused, measurable goals                    X
and outcomes for business products and services.
A description of measurable goals and outcomes for                     X
managing technology to enable the achievement of
business goals and outcomes.
Source: GAO analysis of DOD data.
                                                   a
                                                    A CRUD (create, read, update, and/or delete) matrix shows the specific business functions and
                                                   applications that create, read, update, and/or delete specific data elements, which enables the
                                                   organization to develop applications.




                                                   Page 35                                    GAO-03-1018 DOD Business Enterprise Architecture
b
 Business and operational rules define specific constraints for the data, such as security needs (e.g.,
confidentiality and access of data), and actions that should or should not occur such as updating or
deleting data.
c
  Metadata are “data about data” that enable automation and consistent management and use of
information, such as rules and standards.
d
 Examples of enterprise services include application services, such as web services, and collaboration
services, such as instant messaging or voice conferencing.
e
 Technical standards are strict rules and protocols governing how a given enterprise service is to be
implemented.
f
Security standards cover such services as identification and authentication, audit trail creation, access
controls, virus prevention, and intrusion prevention and detection.


“To Be” Architecture: This architecture is intended to capture the vision of
future business operations and supporting technology. It should describe
the desired capabilities and structures at a specified point(s) in the future.
The “To Be” architecture should show, for example, future business
processes, information needs, and supporting infrastructure, and it should
be fiscally and technologically achievable. According to relevant
guidance,35 the “To Be” architecture should contain, among other things, a
description of (1) the future business operations that will be performed to
support the organization’s mission, including the entities/people that will
perform the functions, processes, and activities, and the locations where
the functions, processes, and activities will be performed, (2) the logical
database model that is to be used to guide the creation of the physical
databases where information will be stored, (3) the systems to be
developed or acquired to support the business operations, (4) the physical
infrastructure (e.g., hardware and software) that will be needed to support
the business systems, (5) the organizations that will be accountable for
implementing security and the tools to be used to secure and protect
systems and data, and (6) the metrics that will be used to evaluate the
effectiveness of mission operations and supporting system performance in
achieving mission goals and objectives. By including these, the
architecture would allow DOD to satisfy the act’s requirements, such as
routinely providing timely, accurate, and reliable information for
management decision making.


35
  See for example, Office of Management and Budget, Federal Enterprise Architecture
Business Reference Model, Version 1.0 (2002); Chief Information Officer Council, A
Practical Guide to Federal Enterprise Architecture, Version 1.0 (February 2001); Office of
Management and Budget, Management of Federal Information Resources, Circular No. A-
130 (Nov. 28, 2000); Cook, M.A., Building Enterprise Information Architectures:
Reengineering Information Systems (Prentice Hall Inc.: 1996); and National Institute of
Standards and Technology, Information Management Directions: The Integration
Challenge, Special Publication 500-167 (September 1989).




Page 36                                      GAO-03-1018 DOD Business Enterprise Architecture
Version 1.0 of DOD’s “To Be” architecture provides some of this descriptive
content, but not to the extent needed to meet the act’s requirements and
permit effective acquisition and implementation of system solutions and
associated operational change. On the positive side, it contains a
description of the future business operations and a logical database model.
However, the future business operations are not described in terms of the
entities/people who will perform the functions, processes, and activities,
and the locations where the functions, processes, and activities will be
performed. Further, we found no linkage between the logical database
model and the conceptual data model, which raises concerns regarding the
utility of this model in supporting information flows for business
operations and systems. In addition, it does not describe (1) the actual
systems to be developed or acquired to support future business operations,
(2) the physical infrastructure (e.g., hardware and software) that will be
needed to support the business systems, (3) the organizations that will be
accountable for implementing security and the tools to be used to secure
and protect systems and data, and (4) the metrics that will be used to
evaluate the effectiveness of mission operations and supporting system
performance in achieving mission goals and objectives. Without this
information, the organization will not have a common vision and frame of
reference for defining a transition plan to guide and constrain capital
investment, and thus will be unable to effectively leverage technology to
orchestrate logical and systematic change and optimize enterprisewide
mission performance. See table 6 for the results of our analysis of DOD’s
“To Be” architecture.




Page 37                         GAO-03-1018 DOD Business Enterprise Architecture
Table 6: Detailed Analysis of the Extent to Which DOD’s “To Be” Architecture Satisfies Key Elements

                                                             Element satisfied?
Key architectural element                              Yes       No      Partially   Explanation of partially satisfied
Business
A description of the strategic goals, which define                       X           The architecture contains a description of the strategic
what an organization wants to achieve.                                               goals, but does not address how it will support the
                                                                                     department’s warfighter goals.
A business strategy, which defines how the                               X           The architecture lists business strategies, such as
strategic goals and objectives will be achieved.                                     utilizing more commercial practices to promote private
                                                                                     sector partnerships, but does not describe how these
                                                                                     strategies will be implemented.
Common policies, procedures, and business rules                          X           The architecture does not have common policies and
for consistent implementation of architecture.                                       procedures, nor has it defined a plan for achieving this.
                                                                                     It does, however, recognize the need for such policies,
                                                                                     procedures, and business rules, and provides a
                                                                                     general time frame for when they will be developed.

                                                                                     The architecture includes high-level descriptions of
                                                                                     business rules, but does not formally define how these
                                                                                     rules will be automated and implemented.
A description of key business processes and how                          X           The architecture has a high-level description of
they support the agency’s mission, including the                                     processes, without a specific identification of
organizational units responsible for performing                                      organizations and locations.
the business processes and the locations where
the business processes are performed.
A description of the architecture governance                             X           The architecture has a draft concept for governance,
structure and processes to ensure that the                                           but does not describe, for example, the process for
department’s business transformation effort                                          ensuring compliance with the architecture and the
remains compliant with the architecture.                                             processes for managing risks and approving the
                                                                                     architecture and systems investments.
A listing of opportunities to unify and/or simplify                      X           The architecture contains a list of deficiencies for the
systems and processes across the agency.                                             operational activities, but not for systems. For example,
                                                                                     it does not identify the specific pilot projects that will be
                                                                                     conducted, nor does it identify the resources (funding
                                                                                     and staffing) needed for conducting these pilots.
A description of the organizational approach for                         X           The architecture has a notional organizational
communications and interactions among                                                structure for communications and interactions among
business lines and program areas for                                                 departmental entities for reporting and management
management reporting, operational functions,                                         purposes.
and architecture development and use.
Information/Data
Description of data management policies,                                 X           The architecture contains a high-level data
processes, procedures, and tools (e.g., CRUD                                         management strategy, including guidelines, and an
Matrix) for analyzing, designing, building, and                                      approach for managing the data in an EA environment.
maintaining databases in an enterprise                                               However, it does not identify the policies, processes,
architected environment.                                                             procedures, and tools to be used.




                                                   Page 38                                GAO-03-1018 DOD Business Enterprise Architecture
(Continued From Previous Page)
                                                            Element satisfied?
Key architectural element                             Yes       No      Partially   Explanation of partially satisfied
A description of the business and operational                           X           The architecture contains descriptions of data
rules for data standardization to ensure data                                       standards upon which business rules can later be
consistency, integrity, and accuracy, such as                                       developed. The architecture contains rules for security,
business and security rules that govern access,                                     but lacks the details needed to consistently enforce
maintenance, and use of data.                                                       these rules (e.g., the rules do not always identify the
                                                                                    event, entity name, and the action to occur). In
                                                                                    addition, the architecture does not provide any
                                                                                    evidence as to whether these business rules have
                                                                                    been verified and validated for completeness.
A data dictionary, which is a repository of                             X           The architecture contains a data dictionary comprised
standard data definitions for applications.                                         of a list of terms and their respective definitions.
                                                                                    However, the architecture does not have a complete
                                                                                    list of terms nor does it contain a list of the data
                                                                                    elementsa needed to support systems and database
                                                                                    design.
A conceptual data model that describes the                              X           The architecture contains a high-level conceptual data
fundamental things/objects (e.g., invoices,                                         model that does not specify how the business objects
financial statements, inventory) that make up the                                   are used by applications (i.e., does not show how the
business and how they are used, but without                                         information is used by the enterprise). Further, this
regard for how they will be physically stored. It                                   model does not show a consolidated view of the data
represents the consolidated structure of business                                   (business objects) to be used by applications.
objects to be used by business applications.
A logical database model that provides the data                         X           The architecture contains data structures, which
structures that support information flows, and that                                 describe, for example, data entities, attributes, and
provides the basis for developing the schemas for                                   relationships among data. However, the model has not
designing, building, and maintaining the physical                                   been verified or validated for completeness with
databases.                                                                          respect to business relevance (i.e., business scenarios
                                                                                    do not show evidence of this validation), nor are there
                                                                                    criteria for defining the number of business scenarios
                                                                                    that have to be completed. In addition, it does not show
                                                                                    the relationship among the data structures in this data
                                                                                    model nor the data structure underlying the
                                                                                    data/information flows for business operations and
                                                                                    systems. Further, the architecture does not contain a
                                                                                    unified enterprise data model that reconciles the
                                                                                    independent data models that have been developed for
                                                                                    each business process area.
A metadata model that specifies the rules and                           X           The architecture notes that an approach, strategy, and
standards for access to information.                                                plan for creating and managing metadata have not yet
                                                                                    been developed. However, it notes that these
                                                                                    documents will be created at a later time.
A description of information flows and                                  X           The architecture contains notional system-to-system
relationships between organizational units,                                         relationships, including how the system may support
business operations, and system elements.                                           business activities, which can be used to extend
                                                                                    development of the architecture, but the architecture
                                                                                    does not link organizational units to business
                                                                                    operations and system elements (e.g., hardware and
                                                                                    software).




                                                Page 39                                 GAO-03-1018 DOD Business Enterprise Architecture
(Continued From Previous Page)
                                                            Element satisfied?
Key architectural element                             Yes       No      Partially   Explanation of partially satisfied
Services/Applications
A description of the business application systems                       X           The architecture has grouped business functions into
and system components and their interfaces.                                         system entitiesb and identified the communication
                                                                                    paths between these entities; however, these entities
                                                                                    are notional.
A description of the common technical approach,                         X           The architecture does not have a common technical
policies, and procedures for developing/acquiring                                   approach and policies and procedures, nor has it
business application systems throughout their life                                  defined a plan for achieving this. It does, however,
cycle, including requirements management,                                           recognize the need for having an approach and
design, implementation, testing, deployment,                                        policies and procedures, and provides a general time
operations, and maintenance. The common                                             frame for when they will be developed.
technical approach should also describe the
process for integrating legacy systems with the
systems to be developed/acquired.
Technical
Descriptions of the enterprise infrastructure                           X           The architecture contains high-level definitions for the
services to include specific details regarding the                                  enterprise services. However, the specific enterprise
functionality and capabilities these services will                                  services for this architecture are to be developed within
provide to enable systems applications.                                             the context of the GIG’s enterprise services, and,
                                                                                    according to DOD, the GIG is not complete and is still
                                                                                    evolving.
Identification of the technical standards to be                         X           DOD has identified enterprise infrastructure services
implemented for each enterprise service.                                            for system entities. However, standards profiles that
                                                                                    support the services, and commonly apply to all
                                                                                    system entities, are not clearly identified and
                                                                                    described. DOD has not yet defined standards profiles
                                                                                    to be employed in the conduct of business processes.
A description of the physical IT infrastructure                 X
needed to support the developed and/or acquired
systems, including the relationships among
hardware, software, and communications devices.
Common policies and procedures for                                      X           The architecture does not have common policies and
developing/acquiring infrastructure systems                                         procedures, nor has it defined a plan for achieving this.
throughout their life cycle including requirements                                  It does, however, recognize the need for having these
management, design, implementation, testing,                                        policies and procedures, and provides a general time
deployment, operations, and maintenance. These                                      frame for when they will be developed.
policies and procedures should also address the
integration of applications, including legacy
systems.




                                                  Page 40                               GAO-03-1018 DOD Business Enterprise Architecture
(Continued From Previous Page)
                                                             Element satisfied?
Key architectural element                              Yes       No      Partially   Explanation of partially satisfied
Security
A description of the policies, procedures, goals,                        X           The architecture refers to policies, but application of
strategies, and requirements relevant to                                             the policies is inconsistent within the architecture. It
information assurance and security.                                                  does not contain procedures; but recognizes the need
                                                                                     for them and provides a general time frame for when
                                                                                     they will be developed. The architecture contains
                                                                                     hypothetical security goals for such attributes as risk
                                                                                     and impact assessments. It also contains a high-level
                                                                                     strategy that explains where information assurance
                                                                                     should be addressed in the architecture and the target
                                                                                     capabilities needed for information assurance (e.g.,
                                                                                     threat/vulnerability assessments). In addition, the
                                                                                     architecture lists relevant security requirements.

                                                                                     However, the goals, strategies, and requirements have
                                                                                     not been mapped to specific physical security systems
                                                                                     solutions. It is also unclear how information assurance
                                                                                     activities will support the department’s warfighter
                                                                                     goals.
A listing of security and information assurance                          X           The data dictionary does list some security-related
related terms.                                                                       terms (e.g., availability, integrity, and authentication);
                                                                                     however, the definitions for these terms are
                                                                                     inconsistent with the definitions contained in the
                                                                                     existing policy.

                                                                                     In addition, some of the terms that are not listed (e.g.,
                                                                                     need-to-know and nonrepudiation) are critical to
                                                                                     implementing effective information assurance controls.

A listing of accountable organizations and their                 X
respective responsibilities for implementing
enterprise security services.
A description of operational security rules that are             X
derived from security policies.
A description of enterprise security infrastructure                      X           The architecture contains generic descriptions of
services (e.g., identification and authentication)                                   enterprise security services, but does not specify the
that will be needed to protect the department’s                                      means for implementation.
assets, and the means for implementing such
services (e.g., firewalls and intrusion detection
software).
A description of the security standards to be                            X           The architecture describes the enterprise services and
implemented for each enterprise service to                                           associated standards that apply to individual system
secure assets. These standards should be                                             entities. However, it does not link requirements to
derived from security requirements.                                                  standards and vice versa.




                                                   Page 41                                GAO-03-1018 DOD Business Enterprise Architecture
(Continued From Previous Page)
                                                           Element satisfied?
Key architectural element                            Yes       No         Partially     Explanation of partially satisfied
A description of the protection mechanisms that                X
will be implemented to secure the department’s
assets, such as firewalls and intrusion detection
software, including a description of the
relationships among these protection
mechanisms.
Performance
A description of the performance management                               X             The architecture contains a high-level proposal to
process, including how the organization will                                            develop this process; however, buy-in has not been
measure, track, evaluate, and predict business                                          achieved. Until buy-in is obtained, the development of
performance with respect to business functions,                                         such a process will not be an architectural
baseline data, and service levels.                                                      requirement.
A description of customer-focused measurable                              X             The architecture contains performance metrics for
goals and outcomes for business products and                                            operational activities and notional systems; however,
services.                                                                               these metrics are not linked to measurable goals
                                                                                        associated with business products and services.
A description of measurable goals and outcomes                            X             The architecture contains plans to establish baseline
for managing technology to enable the                                                   measures that can be used to establish technical
achievement of business goals and outcomes.                                             performance measures, but it does not yet recognize
                                                                                        the need to tie these measures to the business
                                                                                        goals/outcomes.
Source: GAO analysis of DOD data.
                                                a
                                                 Data elements are basic units of information that cannot be further subdivided. For example, you may
                                                create a data structure called ‘Address,’ which contains the data elements ‘Street Address, City, State,
                                                and Zip Code.’
                                                b
                                                 System entities are logical groups of system functions (e.g., general ledger, payroll) representing “To
                                                Be” capabilities and requirements.




                                                Page 42                                      GAO-03-1018 DOD Business Enterprise Architecture
Transition Plan: According to relevant guidance and best practices,36 the
transition plan should provide a temporal roadmap for moving from the “As
Is” to the “To Be” environment. An important step in the development of a
well-defined transition plan is a gap analysis that compares the “As Is” and
“To Be” architectures to identify differences. Other important steps include
analyses of technology opportunities and market place trends as well as
assessments of fiscal and budgetary realities and institutional acquisition
and development capabilities. Using such analyses and assessments,
options are explored and decisions are made regarding which legacy
systems to retain, modify, or retire, and which new systems to introduce on
either a tactical (temporary) basis or to pursue as strategic solutions.
Accordingly, transition plans identify legacy, migration, and new systems,
and sequence them to show, for example, the phasing out and termination
of systems and capabilities, and the timing of the introduction of new
systems and capabilities. Furthermore, they do so in light of resource
constraints, such as budget, people, acquisition/development process
maturity, and associated time frames. Recognizing the importance of a
well-defined transition plan, the act37 also required DOD to identify (1) all
mission-critical or mission-essential operational and developmental
financial and nonfinancial systems, (2) the actual costs to operate and
maintain these systems during fiscal year 2002, and (3) the estimated costs
for fiscal year 2003.

DOD’s transition plan generally does not possess these attributes, and is
basically a plan to develop a transition plan. Specifically, it does not
(1) provide a gap analysis identifying the needed changes to current
business processes and systems, (2) identify all of the systems that will not
become part of the “To Be” architecture as well as the time frames for
phasing out these systems, (3) show a time-based strategy for replacing
legacy systems, including identification of intermediate (i.e., migration)
systems that may be temporarily needed, and (4) define the resources (e.g.,
funding and staff) needed to transition to the target environment. Further,
while the transition plan contained system cost information for fiscal years
2002 and 2003, it did not associate this information, as specified in the act,


36
  See for example, Office of Management and Budget, Federal Enterprise Architecture
Business Reference Model, Version 1.0 (2002); Chief Information Officer Council, A
Practical Guide to Federal Enterprise Architecture, Version 1.0 (February 2001); and Office
of Management and Budget, Management of Federal Information Resources, Circular No.
A-130 (Nov. 28, 2000).
37
     Section 1004 of Public Law 107-314.




Page 43                                    GAO-03-1018 DOD Business Enterprise Architecture
with mission-critical or mission-essential operational and developmental
financial and nonfinancial systems.

DOD attributed the state of its transition plan to attempting to develop this
plan concurrently with developing its “As Is” and “To Be” architectures,
which it found was not feasible. As a result, DOD does not yet have a
meaningful and reliable basis for managing the disposition of its existing
inventory of about 2,300 systems or for sequencing the introduction of
modernized business operations and supporting systems. See table 7 for
the detailed results of our analysis of DOD’s transition plan.




Page 44                          GAO-03-1018 DOD Business Enterprise Architecture
Table 7: Detailed Analysis of the Extent to Which DOD’s Transition Plan Satisfies Key Architectural Elements

                                                                 Element satisfied?
Key architectural element                                  Yes       No      Partially   Explanation of partially satisfied
Transition Plan
Analysis of the gaps between the baseline and target                         X           The transition plan does not contain a detailed gap
architecture for business processes,                                                     analysis that shows how and when operational and
information/data, and services/application systems to                                    system deficiencies will be addressed.
define missing and needed capabilities.
                                                                                         The transition plan does, however, provide general
                                                                                         time frames for when some potential needed
                                                                                         capabilities will be developed, such as incentives
                                                                                         and training plans.
A high-level strategya for implementing the enterprise
architecture, including specific time-phased
milestones for acquiring and deploying systems,
performance metrics, and financial and nonfinancial
resource needs.

This strategy should include:

• A listing of the legacy systems that will not be part                      X           Of the about 2,300 existing systems in DOD’s
  of the “To Be” environment and the schedule for                                        current inventory, DOD has identified 558 legacy
  terminating these systems.                                                             systems and provided retirement dates for 68 (31
                                                                                         have specific termination dates and 37 have only
                                                                                         fiscal year). In other cases, DOD has not specified
                                                                                         any time frames or schedules for termination.


• A strategy that recognizes the need to train staff                         X           The transition plan recognizes that training will be
  relevant to changes to policies, procedures,                                           needed, but does not contain specific information
  business processes, and systems to enable                                              as to when training will occur, the types of training
  operational efficiency and effectiveness. This                                         that will be needed to address changes, and the
  strategy should contain milestone dates for training                                   anticipated costs.
  staff and associated costs.

• A list of the systems to be developed/acquired to                          X           A description of future systems has been provided;
  achieve business needs.                                                                however, the systems described are notional.

• A strategy for employing enterprise application                            X           The transition plan contains a high-level strategy
  integration (EAI) plans, methods, and tools to, for                                    that could be employed for EAI. However, it does
  example, provide for efficiently reusing applications                                  not address the plans, methods, and tools to be
  that already exist concurrent with adding new                                          used.
  applications and databases.




                                                 Page 45                                  GAO-03-1018 DOD Business Enterprise Architecture
(Continued From Previous Page)
                                                                    Element satisfied?
Key architectural element                                     Yes       No        Partially     Explanation of partially satisfied
A technical (systems, infrastructure, and data)
migration plan that shows:

• the transition from legacy to replacement systems                     X
  with explicit sunset dates and intermediate systems
  that may be temporarily needed to sustain existing
  functionality during the transition period;

• an analysis of system interdependencies, including                    X
  the level of effort required to implement related
  systems in a sequenced portfolio of projects that
  includes milestones, time lines, costs, and
  capabilities; and

• a cost estimate for the initial phase(s) of the                       X
  transition, and high-level cost projection for
  transition to the target architecture.
Source: GAO analysis of DOD data.
                                                    a
                                                     An acquisition/business strategy is a plan of action for achieving a specific goal or result through
                                                    contracting for software products and services.




Contractor Review of                                DOD’s verification and validation contractor assessed Version 1.0 of its
Version 1.0 of the                                  architecture against relevant best practices to determine its quality. In June
                                                    2003,38 consistent with our assessment, this contractor reported that while
Architecture Also Identified
                                                    DOD’s architecture contained significant content, it lacked the depth and
Weaknesses                                          detail needed to begin building and implementing modernized systems and
                                                    making operational changes. Further, the contractor reported that the
                                                    architecture was not easily understandable and that its utility to
                                                    stakeholders in system acquisition planning was limited. According to the
                                                    contractor, these conclusions were based on the following findings.

                                                    • Linkages among architecture products had not been defined, making it
                                                      difficult to navigate through the architecture.

                                                    • Architecture products did not adequately describe the “As Is”
                                                      environment, including business processes and existing business
                                                      application systems and supporting technology, which would make it



                                                    38
                                                      MITRE Technical Report: Review of Financial Management Enterprise Architecture
                                                    (FMEA), Version 1.0 (June 2003).




                                                    Page 46                                       GAO-03-1018 DOD Business Enterprise Architecture
                                 difficult for DOD to perform a gap analysis to support development of a
                                 transition plan.

                              • Architecture products did not adequately describe the “To Be”
                                environment, including (1) business rules governing how data are to be
                                accessed and used within the automated environment, (2) migration and
                                target systems and applications, (3) enterprise infrastructure services
                                and the technical standards relevant to each service, (4) security needs,
                                including standards and protection mechanisms (e.g., firewalls), and
                                (5) performance metrics.

                              • The transition plan was merely a plan to develop a transition plan.

                              As a result, the contractor recommended, among other things, that the
                              department discontinue further development of the “To Be” architecture
                              until it addressed identified deficiencies. Program officials stated that they
                              will address these comments in subsequent versions of the architecture.
                              However, they could not provide us with any written plans governing the
                              scope of comments to be addressed, and how they will be addressed and
                              validated.



DOD Has Yet to Establish an   Using the architecture as an integral investment management frame of
Effective Investment          reference is essential to effectively selecting and controlling business
                              system investments and to moving the organization toward the target
Management Process for
                              architecture. Such use of an architecture is provided for in legislation,
Selecting and Controlling     federal guidance, and best practices. In addition, subsection 1004(d) of the
Business System               act stipulates that any amount in excess of $1 million may be obligated for
Investments                   system improvements only if the DOD Comptroller makes a determination
                              that the improvement is necessary for (1) critical national security
                              capability or critical safety and security requirements or (2) prevention of
                              significant adverse effect on a project that is needed to achieve an essential
                              capability. The act further states that once the architecture is approved,
                              the DOD Comptroller must determine that expenditures for system
                              improvements are consistent with the enterprise architecture and the
                              transition plan. These legislative requirements are consistent with our
                              open recommendations to DOD for selecting and controlling business
                              systems investments. Specifically, we recommended that DOD gain control
                              over business system investments by establishing a hierarchy of investment
                              review boards from across the department, establishing a standard set of
                              criteria to ensure alignment and consistency with the architecture, and
                              directing the boards to perform a comprehensive review of all ongoing



                              Page 47                           GAO-03-1018 DOD Business Enterprise Architecture
business system investments. (App. III contains details on the status of
DOD’s efforts to address our open recommendations.)

To comply with the legislative requirement and address our
recommendations, the DOD Comptroller issued a memorandum on March
7, 2003, to DOD’s component organizations stating that the BMSI office—
which is responsible for overseeing the development and implementation
of the architecture—must review all system initiatives with expenditures in
excess of $1 million. In addition, the memorandum directs the DOD
components, as an integral part of the review and approval process, to
present information to DOD Comptroller officials and relevant domain
owners that demonstrates that each investment (1) complies with the
architecture, and (2) is economically justified.

DOD has not yet defined and implemented an effective investment
management process to proactively identify and control system
investments exceeding $1 million while the architecture is being developed and
after it is completed. Based on DOD data, as of June 6, 2003, the DOD
Comptroller had approved one system initiative with expenditures
exceeding $1 million since enactment of the act, and was reviewing four
others. The one system approval for $10 million was an enhancement to
the Mechanization of Contract Administrative Services (MOCAS) system—
which is DOD’s primary contractor pay system and is used to maintain data
on the majority of DOD’s weapons systems as well as service contracts
administered by the Defense Contract Management Agency. According to
DOD, the enhancements to MOCAS are essential because the system
intended to replace MOCAS—Defense Procurement Payment System—was
terminated in December 2002 by the DOD Comptroller after 7 years of
effort and an investment of $126 million because of poor program
performance and increasing costs. In approving the enhancements to
MOCAS, the DOD Comptroller determined that it was needed to assure
continued system operations and that the failure of MOCAS would
jeopardize DOD’s ability to pay contractors on time, which is one of the
criteria in the act.

BMSI officials stated that the department’s current process for selecting
and controlling business system investments depends on the system
owners coming forward with the request for approval, and that it has not
established the means to determine which systems should be submitted for
review. In response to our prior open recommendations, the DOD
Comptroller states that the department is currently establishing a
governance structure that includes an investment review board and making



Page 48                          GAO-03-1018 DOD Business Enterprise Architecture
the domain owners an integral part of the review and approval process for
selecting and controlling business system investments. According to DOD
officials, the board is to utilize a portfolio management approach based on
established approval thresholds to address investment decisions across the
department. Further, DOD officials state that the department is developing
standard criteria to be used by the investment boards to assess business
system investments, including consistency with the architecture. However,
this proposed governance concept has not yet been adopted. We discuss
this process in more detail later in this report.

Our analysis of DOD’s fiscal years 2003 and 2004 IT budget requests shows
that over 200 systems in each year’s budget, totaling about $4 billion per
year, could involve obligations of funds that meet the $1 million threshold.
This indicates that the majority of the billions of dollars that DOD invests in
business system improvements annually have not been subjected to the
scrutiny of the DOD Comptroller as now called for in the act. The act
places limitations on the legal authority of individual program and
government contracting officials to obligate funds in support of the
systems for which they are responsible, but DOD has yet to proactively
manage investments to avoid violations of the limitations and to review
investments in any meaningful way to enforce these statutory limitations.
Program officials acknowledge that the department, at a minimum, could
use DOD’s IT budget documentation to proactively fulfill the act’s
requirements. Until DOD strengthens its process for selecting and
controlling business system investments and adopts an effective
governance concept, it remains exposed to the risk of spending billions of
dollars on duplicative, stove-piped, nonintegrated systems that do not
optimize mission performance and accountability and, therefore, do not
support the department’s transformation goals.




Page 49                           GAO-03-1018 DOD Business Enterprise Architecture
DOD’s Plans for             According to DOD officials, it intends to (1) further develop, evolve, and
                            extend the architecture, including the transition plan, and issue a revised
Evolving and                version, and (2) improve processes for selecting and controlling business
Extending Its               systems investments. However, DOD’s plans for this next phase have not
                            been explicitly defined. Until they are clearly and completely defined and
Enterprise                  effectively implemented, the department risks perpetuating past business
Architecture and for        system investment practices and spending tens of billions of dollars on
Improving Business          incompatible, duplicative, and nonintegrated systems.
System Investment
Decision Making Are
Unclear

DOD’s Plans for Issuing     According to DOD, it intends to issue its next significant version of the
Next Version of             architecture in May 2004 and this update is to extend and evolve the
                            architecture. To accomplish this, program documentation states that DOD
Architecture Products Are
                            will, among other things,
Unclear
                            • determine the contractor resources needed to evolve and extend the
                              architecture;

                            • develop a methodology for integrating the architecture with DOD’s GIG
                              and OMB’s Federal Enterprise Architecture;39

                            • establish an approach for maintaining its existing systems inventory;
                              and

                            • evaluate the architecture for completeness, accuracy, and integration of
                              end-to-end business processes and system functions.

                            However, how DOD will accomplish these and other activities associated
                            with effectively updating its architecture has not been defined, nor have
                            such things as roles and responsibilities for executing activities,
                            dependencies among activities, and measures of activity progress. Rather,
                            the department basically has plans to develop a strategy that will define


                            39
                             See the background section of this report for a description of OMB’s Federal Enterprise
                            Architecture.




                            Page 50                                GAO-03-1018 DOD Business Enterprise Architecture
                            this next phase of activities. By following this approach, DOD will again be
                            setting unrealistic expectations; and without clearly defined plans for
                            evolving and extending the architecture, the department is at risk of falling
                            short of its intended goals to centrally guide and direct its architecture
                            efforts.



DOD’s Plans for Improving   As previously described, DOD has a proposed governance concept that
Controls over Ongoing and   describes how and by whom business transformation requirements
                            identified by the architecture will be implemented in the department. This
Planned Business System     proposal vests the business line representatives or domain owners with the
Investments Are Unclear     authority, responsibility, and accountability for business transformation,
                            implementation of the architecture, development and execution of the
                            transition plan, and portfolio management within their domains. This
                            proposal also designates the domain owners of the business process areas
                            and provides them a high-level description of their roles and
                            responsibilities. In addition, the proposal allocates the current inventory of
                            about 2,300 systems to these domain owners as portfolios of investments to
                            manage.

                            However, it is not clear how the proposed approach will be implemented,
                            and how it will satisfy the act’s investment selection and control
                            requirements. Further, it is also not clear how the proposed approach will
                            address our recommendations for establishing a hierarchy of investment
                            review boards and an explicit set of standard criteria for selecting,
                            controlling, and evaluating IT investments as a portfolio of options, with
                            one criterion to ensure consistency and compliance with ongoing
                            architecture development efforts.

                            According to DOD officials, as a first step, the domain owners will validate
                            cost and other functional information associated with the existing
                            inventory of about 2,300 systems and identify those inventoried systems
                            that will not become part of the “To Be” architecture. According to DOD,
                            these efforts will evolve over time and, therefore, its plans do not include a
                            completion date.

                            Moreover, DOD program documentation provides for initiating pilot
                            projects in the near term that are to demonstrate and implement a portion
                            of the architecture and be usable across the department. In contrast, DOD
                            officials stated that the pilot projects are intended to validate
                            departmentwide business processes and not to implement production
                            systems. Thus, the purpose and scope of these projects remain unclear and



                            Page 51                           GAO-03-1018 DOD Business Enterprise Architecture
              specific projects have yet to be selected. If DOD intends for these projects
              to demonstrate or validate an enterprisewide business process to address a
              current deficiency in DOD’s business operations and systems, such as the
              lack of common data standards, these projects could help DOD improve its
              architecture and thus could be reasonable investments. However, if the
              pilot projects are to be used to acquire and implement system solutions and
              place them into production to achieve an operational capability, it is
              unclear how DOD will ensure architecture alignment and manage the risk
              associated with investing in more systems before it has a well-defined
              blueprint and an effective investment management process to guide and
              control them.



Conclusions   Recent legislation and our past recommendations to DOD recognize that it
              is absolutely essential to have and use a well-defined enterprise
              architecture to guide and constrain DOD’s business systems modernization
              program. DOD’s efforts to date to develop such an architecture, and satisfy
              its legislative mandate, are good first steps to this end, but more steps are
              needed before it will have an adequate basis for acquiring and
              implementing its desired systems environment. In our view, DOD’s BEA
              (version 1.0) provides a foundation for it to move forward in adding
              missing architectural scope and detail, and ultimately validating that the
              architecture is sufficiently complete and correct.

              DOD has not, however, made similar strides in its efforts to control its
              ongoing and planned systems investments. In effect, nothing significant
              has changed since our prior review in the way that DOD is investing billions
              of dollars annually in existing and new systems. This means that the
              department has yet to implement our prior recommendations for
              controlling systems funding, and it has not yet defined and implemented an
              effective approach to satisfy legislative requirements for approving systems
              investments over $1 million. As a result, billions of dollars continue to be at
              risk of being spent on more systems that are duplicative, are not
              interoperable, cost more to maintain than necessary, and do not optimize
              mission performance and accountability.

              The future of DOD’s architecture development and implementation
              activities is difficult to understand because DOD’s near-term plans are
              unclear. As a result, DOD’s business systems modernization efforts remain
              exposed to considerable risk. It is critical for DOD to effectively expand
              and extend its architecture to the point that it provides a sound basis for
              departmentwide investment decision making, and that in doing so, it



              Page 52                           GAO-03-1018 DOD Business Enterprise Architecture
                  continue to centrally guide and direct its architecture development efforts
                  and not allow DOD domain owners to proceed independently. Similarly, it
                  is critical for DOD to immediately gain control over near-term investments
                  pending the architecture’s completion. This includes justifying further
                  investment in each ongoing system project beyond fiscal year 2003 and not
                  starting any new projects that are intended to be put into production and
                  provide operational capabilities, pilot or otherwise, until the
                  (1) architecture has been sufficiently completed and (2) DOD has
                  established an effective institutional approach to make informed systems
                  investment decisions, including ensuring that each investment is
                  architecturally aligned. To do less continues to put billions of dollars at
                  unnecessary risk of perpetuating today’s legacy systems environment.



Recommendations   Because our open recommendations to DOD for managing the
                  development, maintenance, and implementation of its BEA, including
                  effectively controlling ongoing investment in business systems, are critical
                  to the success of its modernization and transformation efforts, we reiterate
                  the recommendations that we made in our May 200140 and February 200341
                  reports. To further assist the department in effectively implementing these
                  recommendations, we are augmenting them by providing the following
                  more specific implementation steps. Specifically, we recommend to the
                  Secretary of Defense that he or his appropriate designee,

                  • define and implement an effective investment management process to
                    proactively identify, control, and obtain DOD Comptroller review and
                    approval of expenditures for new and ongoing business system
                    investments exceeding $1 million while the architecture is being
                    developed and after it is completed, and which includes clearly defined
                    domain owners’ roles and responsibilities for selecting and controlling
                    ongoing and planned system investments;

                  • implement the core elements in our EA Framework for Assessing and
                    Improving Enterprise Architecture Management that we identify in
                    this report as not satisfied, including ensuring that minutes of the
                    executive body charged with directing, overseeing, and approving the
                    architecture are prepared and maintained;


                  40
                       GAO-01-525.
                  41
                       GAO-03-458.




                  Page 53                          GAO-03-1018 DOD Business Enterprise Architecture
• update version 1.0 of the architecture to include the 340 Joint Financial
  Management Improvement Program requirements that our report
  identified as omitted or not fully addressed;

• update version 1.0 of the architecture to include the 29 key elements
  governing the “As Is” architectural content that our report identified as
  not being fully satisfied;

• update version 1.0 of the BEA to include the 30 key elements governing
  the “To Be” architectural content that our report identified as not being
  fully satisfied;

• update version 1.0 to ensure that “To Be” architecture artifacts are
  internally consistent, to include addressing the inconsistencies
  described in this report, as well as including user instructions or
  guidance for easier architecture navigation and use;

• update version 1.0 of the architecture to include (1) the three key
  elements governing the transition plan content that our report identified
  as not being fully satisfied and (2) those system investments that will
  not become part of the “To Be” architecture, including time frames for
  phasing out those systems;

• update version 1.0 of the architecture to address comments made by the
  verification and validation contractor;

• develop a well-defined near-term plan for extending and evolving the
  architecture and ensure that this plan includes addressing our
  recommendations, defining roles and responsibilities of all stakeholders
  involved in extending and evolving the architecture, explaining
  dependencies among planned activities, and defining measures of
  activity progress; and

• limit the pilot projects to small, low-cost, low-risk prototype
  investments that are intended to provide knowledge needed to extend
  and evolve the architecture, and are not to acquire and implement
  production version system solutions or to deploy an operational system
  capability.




Page 54                          GAO-03-1018 DOD Business Enterprise Architecture
Agency Comments and   In written comments on a draft of this report (reprinted in app. IV), the
                      department concurred with 9 of our 10 recommendations, partially
Our Evaluation        concurred with the remaining one, and described recently completed,
                      ongoing, or planned efforts to address them. We will evaluate whether
                      DOD’s efforts fully address our recommendations in future BEA reviews.

                      DOD partially concurred with our recommendation regarding the
                      architectural content of the “As Is” environment stating that because the
                      current operating environment is dynamic, complete satisfaction of the 29
                      key elements that our report identified is not realistically achievable. DOD
                      stated that such data, even if they were possible to obtain, would be
                      obsolete upon arrival and, therefore, the department does not deem the
                      data collection effort to be cost effective. DOD stated that it is currently
                      analyzing the 29 key elements and that as part of its incremental
                      development approach, it will collect relevant “As Is” documentation,
                      where appropriate, and will include the data in future releases of the BEA.

                      We agree that architectural information that does not provide value
                      commensurate with cost should not be captured in the BEA. However,
                      DOD’s comments concerning the missing 29 “As Is” key elements do not
                      contain sufficient context, detail, and explanation to understand which key
                      elements DOD proposes to satisfy and which it does not. Further, its
                      comments do not adequately explain and justify why key elements should
                      be waived. As noted in our report, DOD’s “As Is” architecture products
                      provide little descriptive content and do not satisfy 90 percent of the
                      architectural elements required by relevant guidance needed to permit
                      development of a meaningful and useful transition plan. Further, as noted
                      in our March 2003 report,42 while further development of the “As Is”
                      environment can coincide with the development of the transition plan, not
                      having defined the “As Is” operations and technology at this juncture is
                      risky because it defers until too late in the architecture development cycle
                      creation of sufficient descriptive content and context to develop an
                      effective transition plan.

                      We are sending copies of this report to interested congressional
                      committees; the Director, Office of Management and Budget; the Secretary
                      of Defense; the Under Secretary of Defense (Comptroller); the Assistant
                      Secretary of Defense (Networks and Information Integration)/Chief


                      42
                           GAO-03-571R.




                      Page 55                          GAO-03-1018 DOD Business Enterprise Architecture
Information Officer; the Under Secretary of Defense (Acquisition,
Technology, and Logistics); the Under Secretary of Defense (Personnel and
Readiness); and the Director, Defense Finance and Accounting Service.
This report will also be available at no charge on our Web site at
http://www.gao.gov.

If you or your staff have any questions on matters discussed in this report,
please contact Gregory D. Kutz at (202) 512-9095 or kutzg@gao.gov, or
Randolph C. Hite at (202) 512-3439 or hiter@gao.gov. Major contributors to
this report are acknowledged in appendix V.




Gregory D. Kutz
Director
Financial Management and Assurance




Randolph C. Hite
Director
Information Technology Architecture and System Issues




Page 56                          GAO-03-1018 DOD Business Enterprise Architecture
List of Committees

The Honorable John W. Warner
Chairman
The Honorable Carl Levin
Ranking Minority Member
Committee on Armed Services
United States Senate

The Honorable Ted Stevens
Chairman
The Honorable Daniel K. Inouye
Ranking Minority Member
Subcommittee on Defense
Committee on Appropriations
United States Senate

The Honorable Duncan Hunter
Chairman
The Honorable Ike Skelton
Ranking Minority Member
Committee on Armed Services
House of Representatives

The Honorable Jerry Lewis
Chairman
The Honorable John P. Murtha
Ranking Minority Member
Subcommittee on Defense
Committee on Appropriations
House of Representatives




Page 57                          GAO-03-1018 DOD Business Enterprise Architecture
Appendix I

SEC. 1004. [of Public Law 107-314]                                        Appendx
                                                                                ies




Development and Implementation of Financial
Management Enterprise Architecture                                         Append
                                                                                x
                                                                                iI




              Page 58      GAO-03-1018 DOD Business Enterprise Architecture
Appendix I
SEC. 1004. [of Public Law 107-314]
Development and Implementation of
Financial Management Enterprise
Architecture




Page 59                              GAO-03-1018 DOD Business Enterprise Architecture
Appendix II

Scope and Methodology                                                                               Appendx
                                                                                                          Ii




              To accomplish our objectives for determining (1) the extent to which
              DOD’s actions complied with the requirements of section 1004 of Public
              Law 107-314 and (2) DOD’s plans for further development and
              implementation of the architecture, we assessed DOD’s initial architecture,
              which, according to DOD, was approved by the DOD Comptroller and
              transmitted to the Comptroller General on May 8, 2003. This report
              provides specific details on our assessment results. Our overall
              assessment of DOD’s initial architecture was issued on July 7, 2003,1 which
              satisfied the legislative requirement that we submit a report to
              congressional defense committees within 60 days of the architecture’s
              approval.

              Consistent with the act and as agreed with congressional defense
              committees’ staffs, this assessment focused on (1) compliance with all
              federal accounting, financial management, and reporting requirements,
              (2) the content of the “As Is” and “To Be” environments, (3) the content of
              the transition plan to include time-phased milestones for phasing out
              existing systems, resource needs for implementing the “To Be”
              environment, and information on the systems inventory, and (4) the extent
              to which DOD is controlling its business system investments. In addition,
              we used our Enterprise Architecture Management Maturity Framework2
              that describes the five stages of management maturity to determine the
              extent to which DOD has adopted key elements of architecture
              management best practices. To make this determination, we reviewed
              program documentation, such as program policies and procedures,
              steering and executive committee charters, and architecture products, and
              compared them to the elements in the framework. We did not validate the
              cost and budget information provided by the program’s budget analyst.

              Specific to our review of federal requirements, we could not determine
              whether the architecture contained all federal accounting, financial
              management, and reporting requirements because a central repository of
              all such requirements does not exist. Nevertheless, to assess the
              completeness of the federal requirements, we compared the about 4,000
              external requirements3 contained in the architecture to those listed in


              1
                  GAO-03-877R.
              2
                  GAO-03-584G.
              3
                External requirements are those that are obtained from authoritative sources and
              constrain various aspects of the architecture.




              Page 60                                GAO-03-1018 DOD Business Enterprise Architecture
Appendix II
Scope and Methodology




selected JFMIP federal systems requirements publications.4 Of the 4,000
external requirements incorporated in the initial architecture, we
performed a detailed review of 1,767 (about 45 percent), all of which were
JFMIP requirements. Specifically, we identified whether these
requirements were incorporated in the initial architecture, relevant to
DOD’s business operations, or were current. To supplement our
documentation review, we held interviews with government and contractor
officials from the Office of the Under Secretary of Defense (Comptroller)
and IBM.

For our review of the architecture,5 our internal team of architecture
experts identified relevant criteria to be used to assess the architecture
products, including best practices and federal guidance.6

In reviewing the criteria, these experts categorized the key requirements
according to their relevance to the three primary component parts of the
architecture specified in the act and recognized in best practices and
federal guidance: the “As Is” architecture, the “To Be” architecture, and the
transition plan. For ease of reporting, they further divided the “As Is” and
“To Be” architectures into five architectural components similar to OMB’s
architecture reference models: business, information/data,
services/applications, technical, and performance. We added security as a
sixth component because of its recognized importance in the various

4
  We used nine JFMIP systems requirements documents: JFMIP-SR-03-01, Revenue System
Requirements (January 2003); JFMIP-SR-02-02, Acquisition/Financial Systems Interface
Requirements (June 2002); JFMIP-SR-02-01, Core Financial System Requirements
(November 2001); JFMIP-SR-99-5, Human Resources & Payroll Systems Requirements
(April 1999); JFMIP-FFMSR-8, System Requirements for Managerial Cost Accounting
(February 1998); JFMIP-FFMSR-7, Inventory System Requirements (June 1995); JFMIP-SR-
99-9, Travel System Requirements (July 1999); JFMIP-SR-00-4, Property Management
Systems Requirements (October 2000); JFMIP-SR-01-01, Benefit System Requirements
(September 2001).
5
  We reviewed version 1.0 of DOD’s BEA including the transition plan, which was completed
on April 30, 2003, and according to program officials, approved on May 8, 2003, by the
department’s Comptroller.
6
  See, for example, Office of Management and Budget, Federal Enterprise Architecture
Business Reference Model, Version 1.0 (2002); Chief Information Officers Council, A
Practical Guide to Federal Enterprise Architecture, Version 1.0 (February 2001); Office of
Management and Budget, Management of Federal Information Resources, Circular No. A-
130 (Nov. 28, 2000); M.A. Cook, Building Enterprise Information Architectures:
Reengineering Information Systems (Upper Saddle River, N.J., Prentice Hall: 1996); and
National Institute of Standards and Technology, Information Management Directions: The
Integration Challenge, Special Publication 500-167 (September 1989).




Page 61                                GAO-03-1018 DOD Business Enterprise Architecture
Appendix II
Scope and Methodology




architecture frameworks and relevance to the other five architectural
components. For each of these six architectural components, we identified
the key architectural requirements that would need to be addressed for the
“As Is” and “To Be” environments for the department to create an
architecture that would be effective in facilitating its business
modernization efforts and documented this information in detailed
matrices. These experts also identified the key architectural requirements
for the transition plan component of the architecture, which were also
documented in a detailed matrix. We then compared the architecture
products including the transition plan against the identified criteria
governing their content and documented the results of our analysis in the
matrices.

We interviewed program officials, including the program director, the Chief
Architect, and contractor staff (IBM and MITRE) to discuss our preliminary
findings and to clarify the intended scope and purpose of this version of the
architecture. We also participated in a 2-day architecture walkthrough in
which DOD officials provided a progress update on the department’s
development of the architecture and future plans for further evolution and
implementation of the architecture. In addition, we reviewed the program’s
verification and validation contractor’s (MITRE) report7 documenting its
assessment of version 1.0 of the architecture including the transition plan.
We also interviewed program officials as to the department’s plans for
addressing MITRE’s comments.

To review DOD’s actions to comply with the conditions for obligations in
excess of $1 million for financial system improvements, we obtained and
reviewed memorandums and other documentation regarding the approval
of expenditures for system investments in excess of $1 million. We also
reviewed and analyzed the DOD IT budget requests for fiscal years 2003
and 2004 to identify systems that met the $1 million threshold and
compared this to the total number of systems DOD reviewed and approved
to measure DOD’s progress in reviewing those systems that meet the
legislative threshold. To augment our document reviews and analyses, we
interviewed officials from various DOD organizations, including the Office
of the Under Secretary of Defense (Comptroller); Office of the Under
Secretary of Defense (Acquisition, Technology, and Logistics); and the
Office of the Under Secretary of Defense (Personnel and Readiness).


7
  MITRE Technical Report: Review of Financial Management Enterprise Architecture
(FMEA), Version 1.0 (June 2003).




Page 62                             GAO-03-1018 DOD Business Enterprise Architecture
Appendix II
Scope and Methodology




To determine DOD’s plans for further development and implementation of
the architecture, we reviewed the initial transition plan and IBM’s
statement of work, DOD’s proposed governance concept, and program
documentation pertaining to plans for implementing pilot projects. We also
reviewed DOD’s response to the recommendations we made in our
February 2003 report8 pertaining to controlling ongoing and planned IT
systems investments. To augment our document reviews and analyses, we
interviewed government and contractor officials from the Office of the
Under Secretary of Defense (Comptroller) and IBM.

We conducted our work primarily at DOD headquarters offices in
Washington, D.C., and Arlington, Virginia, and we performed our work from
March 2003 through June 2003 in accordance with U.S. generally accepted
government auditing standards. We requested comments on a draft of this
report from the Secretary of Defense or his designee. Written comments
from the Under Secretary of Defense (Comptroller) are addressed in the
“Agency Comments and Our Evaluation” section of this report and are
reprinted in appendix IV.




8
    GAO-03-458.




Page 63                         GAO-03-1018 DOD Business Enterprise Architecture
Appendix III

Status of Prior Recommendations on DOD’s
Business Enterprise Architecture                                                                                                               Appendx
                                                                                                                                                     iI




                                                                    Status of recommendation
                                                                                       Not               DOD comments and our
Recommendations                                                Implemented   Partial   implemented       assessment
GAO-01-525: Information Technology: Architecture Needed to Guide Modernization of DOD’s Financial Operations. May 17, 2001.
1) The Secretary of Defense immediately designate              X
DOD financial management modernization a
departmental priority and accordingly direct the Deputy
Secretary of Defense to lead an integrated program
across the department for modernizing and optimizing
financial management operations and systems.
2) The Secretary immediately issue a DOD policy that                                   X                 As discussed in this report, DOD
directs the development, implementation, and                                                             has a policy for developing the
maintenance of a financial management enterprise                                                         Global Information Grid (GIG),
architecture.                                                                                            which requires that all
                                                                                                         departmental architectures be in
                                                                                                         alignment with the GIG. While this
                                                                                                         policy outlines the roles and
                                                                                                         responsibilities for development,
                                                                                                         maintenance, and implementation
                                                                                                         of the GIG, it does not do so for
                                                                                                         other departmental architectures,
                                                                                                         including the BEA.
3) The Secretary immediately modify the Senior                               X                           The department has established
Financial Management Oversight Council’s charter to                                                      the Financial Management
                                                                                                         Executive and Steering
• designate the Deputy Secretary of Defense as the                                                       Committees to serve as advisory
  Council chair and the Under Secretary of Defense                                                       bodies to the Under Secretary of
  (Comptroller) as the Council vice-chair;                                                               Defense (Comptroller) for financial
• empower the Council to serve as DOD’s financial                                                        management modernization.
  management enterprise architecture steering                                                            However, as discussed in this
  committee, giving it the responsibility and authority to                                               report, DOD has not assigned
  ensure that a DOD financial management enterprise                                                      responsibility for directing,
  architecture is developed and maintained in                                                            overseeing, and approving the
  accordance with the DOD C4ISR Architecture                                                             BEA to a committee or group
  Framework;                                                                                             comprised of representatives from
• empower the Council to serve as DOD’s financial                                                        across the department.
  management investment review board, giving it the
  responsibility and authority to (1) select and control all
  DOD financial management investments and                                                               In addition, DOD has not yet
  (2) ensure that its investment decisions treat                                                         established a hierarchy of
  compliance with the financial management enterprise                                                    investment review boards, each
  architecture as an explicit condition for investment                                                   responsible and accountable for
  approval that can be waived only if justified by a                                                     selecting and controlling
  compelling written analysis; and                                                                       investments that meet defined
• expand the role of the Council’s System Compliance                                                     criteria, including compliance with
  Working Group to include supporting the Council in                                                     the enterprise architecture.
  determining the compliance of each system investment
  with the enterprise architecture at key decision points
  in the system’s development or acquisition life cycle.



                                                  Page 64                                  GAO-03-1018 DOD Business Enterprise Architecture
                                                  Appendix III
                                                  Status of Prior Recommendations on DOD’s
                                                  Business Enterprise Architecture




(Continued From Previous Page)
                                                                    Status of recommendation
                                                                                       Not            DOD comments and our
Recommendations                                                Implemented   Partial   implemented    assessment
4) The Secretary immediately make the Assistant                              X                        The Assistant Secretary of
Secretary of Defense (Command, Control,                                                               Defense (Networks and
Communications & Intelligence), in collaboration with the                                             Information Integration)/ Chief
Under Secretary of Defense (Comptroller), accountable                                                 Information Officer, is a member of
to the Senior Financial Management Oversight Council                                                  the Executive and Steering
for developing and maintaining a DOD financial                                                        Committees; however, as
management enterprise architecture.                                                                   discussed previously, members’
                                                                                                      roles and responsibilities are
In fulfilling this responsibility, the Assistant Secretary                                            advisory in nature.
appoint a Chief Architect for DOD financial management
modernization and establish and adequately staff and                                                  DOD established a Financial
fund an enterprise architecture program office that is                                                Management Modernization
responsible for developing and maintaining a DOD-wide                                                 Program Office in July 2001. Also,
financial management enterprise architecture in a                                                     DOD has appointed a chief
manner that is consistent with the framework defined in                                               architect and, according to DOD, it
the CIO Council’s published guide for managing                                                        has adequate program funding and
enterprise architectures. In particular, the Assistant                                                staff for developing and
Secretary should take appropriate steps to ensure that                                                maintaining its BEA. However, the
the Chief Architect                                                                                   chief architect’s roles and
                                                                                                      responsibilities have not yet been
• obtains executive buy-in and support;                                                               defined.
• establishes architecture management structure and
  controls;
• defines the architecture process and approach;
• develops the baseline architecture, the target
  architecture, and the sequencing plan;
• facilitates the use of the architecture to guide financial
  management modernization projects and investments;
  and
• maintains the architecture.
5) The Assistant Secretary of Defense (Command,                              X                        The program office, which the chief
Control, Communications & Intelligence) report at least                                               architect is part of, briefs the
quarterly to the Senior Financial Management Oversight                                                Steering Committee monthly on
Council on the chief architect’s progress in developing a                                             the status of DOD’s efforts for
financial management enterprise architecture, including                                               developing its BEA; however
the chief architect’s adherence to enterprise architecture                                            meeting minutes are not prepared
policy and guidance from OMB, the CIO Council, and                                                    and maintained. Also, as noted
DOD.                                                                                                  previously, DOD has not issued a
                                                                                                      policy on the development,
                                                                                                      implementation, and maintenance
                                                                                                      of the BEA.




                                                  Page 65                               GAO-03-1018 DOD Business Enterprise Architecture
                                                Appendix III
                                                Status of Prior Recommendations on DOD’s
                                                Business Enterprise Architecture




(Continued From Previous Page)
                                                                  Status of recommendation
                                                                                     Not               DOD comments and our
Recommendations                                              Implemented   Partial   implemented       assessment
6) The Senior Financial Management Oversight Council                       X                           Senate Report 107-213, enacted
report to the Secretary of Defense every 6 months on                                                   on July 18, 2002, directs that the
progress in developing and implementing a financial                                                    department report every 6 months
management enterprise architecture.                                                                    to congressional defense
                                                                                                       committees on the status of the
                                                                                                       architecture effort. DOD submitted
                                                                                                       the first report on January 31,
                                                                                                       2003. In doing so, the Under
                                                                                                       Secretary of Defense
                                                                                                       (Comptroller), who is the chair of
                                                                                                       the Executive and Steering
                                                                                                       Committees, approved the
                                                                                                       semiannual report.
7) The Secretary report every 6 months to the                              X                           Senate Report 107-213 directs
congressional defense authorizing and appropriating                                                    that the department report every 6
committees on progress in developing and implementing                                                  months on the status of the
a financial management enterprise architecture.                                                        architecture effort. DOD submitted
                                                                                                       the first report on January 31,
                                                                                                       2003.
8) Until a financial management enterprise architecture                              X                 DOD has not yet defined and
is developed and the Council is positioned to serve as                                                 implemented an effective approach
DOD’s financial management investment review board                                                     for selecting and controlling
as recommended above, the Secretary of Defense limit                                                   business systems investments.
DOD components’ financial management investments to                                                    DOD has stated that the
                                                                                                       department plans to establish a
• deployment of systems that have already been fully                                                   governance structure that includes
  tested and involve no additional development or                                                      investment review boards to help
  acquisition cost;                                                                                    control investment in business
• stay-in-business maintenance needed to keep existing                                                 systems and help ensure
  systems operational;                                                                                 consistency with the architecture.
• management controls needed to effectively invest in
  modernized systems; and
• new systems or existing system changes that are
  congressionally directed or are relatively small, cost
  effective, and low risk and can be delivered in a
  relatively short time frame.
GAO-03-458: DOD Business Systems Modernization: Improvements to Enterprise Architecture Development and Implementation Efforts
Needed. February 28, 2003.
1) The Secretary of Defense ensure that the enterprise                               X                 As discussed in this report, DOD
architecture executive committee members are                                                           has not assigned responsibility for
singularly and collectively made explicitly accountable to                                             directing, overseeing, and
the Secretary for the delivery of the enterprise                                                       approving the EA to a committee
architecture including approval of each version of the                                                 or group comprised of
architecture.                                                                                          representatives from across the
                                                                                                       department.




                                                Page 66                                  GAO-03-1018 DOD Business Enterprise Architecture
                                                Appendix III
                                                Status of Prior Recommendations on DOD’s
                                                Business Enterprise Architecture




(Continued From Previous Page)
                                                                  Status of recommendation
                                                                                     Not               DOD comments and our
Recommendations                                              Implemented   Partial   implemented       assessment
2) The Secretary of Defense ensure that the enterprise                     X                           DOD has a communications and
architecture program is supported by a proactive                                                       change management plan, but
marketing and communication program.                                                                   steps have not yet been taken to
                                                                                                       implement the plan.

3) The Secretary of Defense ensure that the quality                        X                           DOD’s quality assurance function
assurance function                                                                                     does not yet address process
• includes the review of adherence to process standards                                                standards and program
  and reliability of reported program performance,                                                     performance nor is it yet an
• is made independent of the program management                                                        independent function. However,
  function, and                                                                                        DOD stated that it intends to make
• is not performed by subject matter experts involved in                                               this function independent. Further,
  the development of key architecture products.                                                        DOD subject matter experts
                                                                                                       continue to be involved in the
                                                                                                       quality assurance function.
4) The Secretary gain control over ongoing IT                                        X                 As discussed in this report, DOD
investments by establishing a hierarchy of investment                                                  has not yet established investment
review boards, each responsible and accountable for                                                    review boards to control its IT
selecting and controlling investments that meet defined                                                investments. DOD has stated that
threshold criteria, and each composed of the appropriate                                               it is in the process of establishing a
level of executive representatives, depending on the                                                   review board and defining the roles
threshold criteria, from across the department.                                                        and responsibilities.
5) The Secretary gain control over ongoing IT                                        X                 As discussed in this report, DOD
investments by establishing a standard set of criteria to                                              has not yet established an
include (1) alignment and consistency with the DOD                                                     investment review board nor has it
enterprise architecture and (2) our open                                                               established standard criteria to be
recommendations governing limitations in business                                                      used in assessing ongoing IT
system investments pending development of the                                                          investments, including alignment
architecture.                                                                                          and consistency with the BEA.
6) The Secretary gain control over ongoing IT                                        X                 As discussed above, the
investments by directing these boards to immediately                                                   investment review boards and
apply these criteria in completing reviews of all ongoing                                              standard criteria have not yet been
IT investments, and to not fund investments that do not                                                established. DOD states that it is
meet these criteria unless they are otherwise justified by                                             working with the domain owners to
explicit criteria waivers.                                                                             develop a governance structure
                                                                                                       that identifies critical processes
                                                                                                       and enterprise requirements to
                                                                                                       improve control over its IT
                                                                                                       investments.
Source: GAO analysis of DOD data.




                                                Page 67                                  GAO-03-1018 DOD Business Enterprise Architecture
Appendix IV

Comments from the Department of Defense                                 Appendx
                                                                              iIV




              Page 68    GAO-03-1018 DOD Business Enterprise Architecture
Appendix IV
Comments from the Department of Defense




Page 69                             GAO-03-1018 DOD Business Enterprise Architecture
Appendix IV
Comments from the Department of Defense




Page 70                             GAO-03-1018 DOD Business Enterprise Architecture
Appendix IV
Comments from the Department of Defense




Page 71                             GAO-03-1018 DOD Business Enterprise Architecture
Appendix IV
Comments from the Department of Defense




Page 72                             GAO-03-1018 DOD Business Enterprise Architecture
Appendix V

GAO Contacts and Staff Acknowledgments                                                              Append
                                                                                                         x
                                                                                                         i
                                                                                                         V




GAO Contacts      Jenniffer Wilson, (202) 512-9192
                  Cynthia Jackson, (202) 512-5086



Acknowledgments   In addition to the individuals named above, key contributors to this report
                  included Beatrice Alff, Nabajyoti Barkakati, Justin Booth, Francine
                  Delvecchio, Francis Dymond, Jason Kelly, Neelaxi Lakhmani, Anh Le,
                  Evelyn Logue, Mai Nguyen, Darby Smith, Stacey Smith, Alan Steiner,
                  Randolph Tekeley, William Wadsworth, and James Weidner.




(192100)          Page 73                            GAO-03-1018 DOD Business Enterprise Architecture
GAO’s Mission            The General Accounting Office, the audit, evaluation and investigative arm of
                         Congress, exists to support Congress in meeting its constitutional responsibilities
                         and to help improve the performance and accountability of the federal government
                         for the American people. GAO examines the use of public funds; evaluates federal
                         programs and policies; and provides analyses, recommendations, and other
                         assistance to help Congress make informed oversight, policy, and funding
                         decisions. GAO’s commitment to good government is reflected in its core values of
                         accountability, integrity, and reliability.


Obtaining Copies of      The fastest and easiest way to obtain copies of GAO documents at no cost is
                         through the Internet. GAO’s Web site (www.gao.gov) contains abstracts and full-
GAO Reports and          text files of current reports and testimony and an expanding archive of older
                         products. The Web site features a search engine to help you locate documents
Testimony                using key words and phrases. You can print these documents in their entirety,
                         including charts and other graphics.
                         Each day, GAO issues a list of newly released reports, testimony, and
                         correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site
                         daily. The list contains links to the full-text document files. To have GAO e-mail this
                         list to you every afternoon, go to www.gao.gov and select “Subscribe to
                         e-mail alerts” under the “Order GAO Products” heading.


Order by Mail or Phone   The first copy of each printed report is free. Additional copies are $2 each. A check
                         or money order should be made out to the Superintendent of Documents. GAO
                         also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a single
                         address are discounted 25 percent. Orders should be sent to:
                         U.S. General Accounting Office
                         441 G Street NW, Room LM
                         Washington, D.C. 20548
                         To order by Phone:     Voice: (202) 512-6000
                                                TDD: (202) 512-2537
                                                Fax: (202) 512-6061


To Report Fraud,         Contact:
                         Web site: www.gao.gov/fraudnet/fraudnet.htm
Waste, and Abuse in      E-mail: fraudnet@gao.gov
Federal Programs         Automated answering system: (800) 424-5454 or (202) 512-7470



Public Affairs           Jeff Nelligan, Managing Director, NelliganJ@gao.gov (202) 512-4800
                         U.S. General Accounting Office, 441 G Street NW, Room 7149
                         Washington, D.C. 20548
United States                  Presorted Standard
General Accounting Office      Postage & Fees Paid
Washington, D.C. 20548-0001           GAO
                                 Permit No. GI00
Official Business
Penalty for Private Use $300
Address Service Requested