oversight

Organizational Transformation: Enterprise Architecture Value Needs to Be Measured and Reported

Published by the Government Accountability Office on 2012-09-26.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                 United States Government Accountability Office

GAO              Report to Congressional Committees




September 2012
                 ORGANIZATIONAL
                 TRANSFORMATION
                 Enterprise
                 Architecture Value
                 Needs to Be Measured
                 and Reported




GAO-12-791
                                                 September 2012

                                                 ORGANIZATIONAL TRANSFORMATION
                                                 Enterprise Architecture Value Needs to Be
                                                 Measured and Reported
Highlights of GAO-12-791, a report to
congressional committees




Why GAO Did This Study                           What GAO Found
According to OMB, the federal                    Among the 27 agencies that GAO studied, all have fully or partially defined goals or
executive branch plans to spend at               purposes for their architectures, 11 have fully or partially established a method or
least $75 billion on information                 metrics for measuring outcomes resulting from the use of their architectures, while
technology (IT) investments in fiscal            5 have fully or partially measured and reported outcomes and benefits
year 2012. In response to a statute              (see table).
which mandates that GAO identify
duplicative activities within federal                                                                                    Outcomes and
                                                                                             Goals or   Metrics and    benefits periodically
agencies, GAO previously identified                                                          purpose      method         measured and
enterprise architecture as a                      Agency                                     defined    established          reported
mechanism for reducing duplication                Agriculture                                   ●            ○                   ○
and overlap in investments. An                    Air Force                                     ●            ○                  ○
architecture is a “blueprint” that
                                                  Army                                          ●            ◐                  ○
describes how an organization
                                                  Commerce                                      ●            ◐                  ○
operates in terms of business
processes and technology, how it                  Defense—Business Enterprise Architecture      ●            ○                  ○
intends to operate in the future, and             Defense—Enterprise Architecture               ●            ○                  ○
how it plans to transition to the future          Education                                     ●            ◐                  ◐
state. Knowing whether architecture               Energy                                        ●            ○                  ○
outcomes are being achieved requires              Health and Human Services                     ●            ●                  ◐
defining the architecture’s goals,
                                                  Homeland Security                             ●            ○                  ○
establishing a method and metrics to
                                                  Housing and Urban Development                 ●            ●                  ◐
measure architecture outcomes, and
periodically measuring and reporting              Interior                                      ●            ○                  ○
these outcomes. To assess agencies’               Justice                                       ●            ○                  ○
use of architecture as a mechanism for            Labor                                         ●            ○                  ○
reducing duplication and overlap, GAO             Navy                                          ●            ○                  ○
committed to determine the extent to              State                                         ●            ○                  ○
which agencies are measuring and                  Transportation                                ●            ◐                  ○
reporting architecture outcomes and
                                                  Treasury                                      ●            ◐                  ◐
benefits. To do this, GAO reviewed
                                                  Veterans Affairs                              ◐            ○                  ○
relevant documentation from 27 major
federal agencies, reviewed the results            Environmental Protection Agency               ●            ○                  ○
of a GAO survey on the benefits of                General Services Administration               ●            ◐                  ○
using architecture, and interviewed               National Aeronautics and Space
                                                                                                ●            ○                  ○
agency officials.                                 Administration
                                                  National Science Foundation                   ●            ○                  ○
                                                  Nuclear Regulatory Commission                 ●            ◐                  ○
What GAO Recommends
                                                  Officer of Personnel Management               ●            ◐                  ○
GAO is making recommendations to                  Small Business Administration                 ●            ○                  ○
the agencies and OMB to improve                   Social Security Administration                ●            ○                  ○
measurement and reporting of                      United States Agency for International
architecture outcomes. In commenting                                                            ●            ●                  ●
                                                  Development
on a draft of this report, OMB and most          Source: GAO analysis of agency data.
of the agencies generally agreed with
the findings and recommendations.                Agencies cited a lack of guidance as a key reason why they have not established
                                                 methods and metrics for measuring outcomes and benefits. Although the Office
                                                 of Management and Budget (OMB) has issued recent enterprise architecture
View GAO-12-791. For more information,           guidance to agencies, OMB has not yet provided sufficient details on the method
contact Valerie C. Melvin at (202) 512-6304 or   and metrics that could be used to measure architecture program outcomes.
melvinv@gao.gov.

                                                                                              United States Government Accountability Office
Contents


Letter                                                                                      1
                Background                                                                  3
                Almost All Agencies Had Defined the Purpose of Their
                  Architectures, but Had Yet to Fully Measure and Report
                  Outcomes and Benefits                                                   12
                Conclusions                                                               21
                Recommendations for Executive Action                                      22
                Agency Comments and Our Evaluation                                        23

Appendix I      Objective, Scope, and Methodology                                         29



Appendix II     Detailed Assessments of Individual Departments and
                Agencies against Relevant Elements of Our Enterprise
                Architecture Management Maturity Framework                                32



Appendix III    Comments from the Department of Labor                                     63



Appendix IV     Comments from the Department of the Treasury                              64



Appendix V      Comments from the Department of Agriculture                               65



Appendix VI     Comments from the Department of Commerce                                  67



Appendix VII    Comments from the Department of Defense                                   68



Appendix VIII   Comments from the Department of Education                                 70




                Page i                                GAO-12-791 Organizational Transformation
Appendix IX      Comments from the Department of Homeland Security                        72



Appendix X       Comments from the Department of the Interior                             74



Appendix XI      Comments from the Department of State                                    75



Appendix XII     Comments from the Department of Veterans Affairs                         79



Appendix XIII    Comments from the National Aeronautics and Space Administration          82



Appendix XIV     Comments from the Social Security Administration                         84



Appendix XV      Comments from the Environmental Protection Agency                        86



Appendix XVI     Comments from the Department of Health and Human Services                87



Appendix XVII    Comments from the Department of Energy                                   89



Appendix XVIII   Comments from the Department of Housing and Urban Development            93



Appendix XIX     GAO Contact and Staff Acknowledgments                                    96




                 Page ii                              GAO-12-791 Organizational Transformation
Tables
         Table 1: Summary of Agencies’ Progress in Measuring and
                  Reporting Enterprise Architecture Outcomes and Benefits
                  Relative to Elements of GAO’s EAMMF (Version 2.0)                13
         Table 2: Agencies Included in our Study                                   29
         Table 3: Department of Agriculture Satisfaction of EAMMF
                  Elements                                                         33
         Table 4: Department of the Air Force Satisfaction of EAMMF
                  Elements                                                         34
         Table 5: Department of the Army Satisfaction of EAMMF Elements            35
         Table 6: Department of Commerce Satisfaction of EAMMF
                  Elements                                                         37
         Table 7: DOD Business Enterprise Architecture Satisfaction of
                  EAMMF Elements                                                   38
         Table 8: Department of Defense Satisfaction of EAMMF Elements             39
         Table 9: Department of Education Satisfaction of GAO EAMMF
                  Elements                                                         40
         Table 10: Department of Energy Satisfaction of EAMMF Elements             41
         Table 11: Department of Health and Human Services Satisfaction of
                  EAMMF Elements                                                   42
         Table 12: Department of Homeland Security Satisfaction of
                  EAMMF Elements                                                   43
         Table 13: Department of Housing and Urban Development
                  Satisfaction of EAMMF Elements                                   44
         Table 14: Department of the Interior Satisfaction of EAMMF
                  Elements                                                         46
         Table 15: Department of Justice Satisfaction of EAMMF Elements            47
         Table 16: Department of Labor Satisfaction of EAMMF Elements              48
         Table 17: Department of the Navy Satisfaction of EAMMF Elements           49
         Table 18: Department of State Satisfaction of EAMMF Elements              50
         Table 19: Department of Transportation Satisfaction of EAMMF
                  Elements                                                         51
         Table 20: Department of the Treasury Satisfaction of EAMMF
                  Elements                                                         52
         Table 21: Department of Veterans Affairs Satisfaction of EAMMF
                  Elements                                                         53
         Table 22: Environmental Protection Agency Satisfaction of EAMMF
                  Elements                                                         54
         Table 23: General Services Administration Satisfaction of EAMMF
                  Elements                                                         55
         Table 24: National Aeronautics and Space Administration
                  Satisfaction of EAMMF Elements                                   56


         Page iii                              GAO-12-791 Organizational Transformation
Table 25: The National Science Foundation Satisfaction of EAMMF
        Elements                                                          57
Table 26: Nuclear Regulatory Commission Satisfaction of EAMMF
        Elements                                                          58
Table 27: Office of Personnel Management Satisfaction of EAMMF
        Elements                                                          59
Table 28: Small Business Administration Satisfaction of EAMMF
        Elements                                                          60
Table 29: Social Security Administration Satisfaction of EAMMF
        Elements                                                          61
Table 30: U.S. Agency for International Development Satisfaction
        of EAMMF Elements                                                 62




Page iv                               GAO-12-791 Organizational Transformation
Abbreviations
CIO                  chief information officer
Commerce             Department of Commerce
DHS                  Department of Homeland Security
DOD                  Department of Defense
EAMMF                Enterprise Architecture Management Maturity
                     Framework
Education            Department of Education
Energy               Department of Energy
EPA                  Environmental Protection Agency
GSA                  General Services Administration
HHS                  Department of Health and Human Services
HUD                  Department of Housing and Urban Development
Interior             Department of the Interior
IT                   information technology
Justice              Department of Justice
Labor                Department of Labor
NASA                 National Aeronautics and Space Administration
NRC                  Nuclear Regulatory Commission
NSF                  National Science Foundation
OMB                  Office of Management and Budget
OPM                  Office of Personnel Management
SBA                  Small Business Administration
SSA                  Social Security Administration
State                Department of State
Transportation       Department of Transportation
Treasury             Department of the Treasury
USAID                United States Agency for International Development
USDA                 United States Department of Agriculture
VA                   Department of Veterans Affairs

This is a work of the U.S. government and is not subject to copyright protection in the
United States. The published product may be reproduced and distributed in its entirety
without further permission from GAO. However, because this work may contain
copyrighted images or other material, permission from the copyright holder may be
necessary if you wish to reproduce this material separately.




Page v                                           GAO-12-791 Organizational Transformation
United States Government Accountability Office
Washington, DC 20548




                                   September 26, 2012

                                   The Honorable Joseph Lieberman
                                   Chairman
                                   The Honorable Susan Collins
                                   Ranking Member
                                   Committee on Homeland Security and Governmental Affairs
                                   United States Senate

                                   The Honorable Darrell Issa
                                   Chairman
                                   The Honorable Elijah Cummings
                                   Ranking Member
                                   Committee on Oversight and Government Reform
                                   House of Representatives

                                   Billions of taxpayer dollars are spent on information technology (IT)
                                   investments each year; according to the Office of Management and
                                   Budget (OMB), the executive branch plans to spend at least $75 billion in
                                   fiscal year 2012. We have previously reported that federal expenditures
                                   on IT could be reduced by, among other things, using enterprise
                                   architecture as a tool for organizational transformation. 1

                                   An enterprise architecture is a blueprint for organizational change defined
                                   in models that describe (in both business and technology terms) how the
                                   entity operates today and how it intends to operate in the future; it also
                                   includes a plan for transitioning to this future state. Effective use of an
                                   enterprise architecture is a hallmark of successful organizations and can
                                   be important to achieving operations and technology environments that
                                   maximize institutional mission performance and outcomes. Among other
                                   things, this includes realizing cost savings through consolidation and
                                   reuse of shared services and elimination of antiquated and redundant
                                   mission operations, enhancing information sharing through data
                                   standardization and system integration, and optimizing service delivery
                                   through streamlining and normalization of business processes and


                                   1
                                    GAO, Opportunities to Reduce Potential Duplication in Government Programs, Save Tax
                                   Dollars, and Enhance Revenue, GAO-11-318SP (Washington, D.C.: Mar. 1, 2011). An
                                   interactive, web-based version of the report is available at
                                   http://www.gao.gov/ereport/gao-11-318SP.




                                   Page 1                                       GAO-12-791 Organizational Transformation
mission operations. Moreover, the use of architectures is required by the
Clinger-Cohen Act of 1996 and by OMB. 2

In our March 2011 report on opportunities to reduce potential duplication
in government programs, 3 we identified enterprise architecture as a
mechanism for identifying potential overlap and duplication. We noted
that realizing this potential and knowing whether benefits are in fact being
achieved from the use of an architecture requires associated measures
and metrics. Accordingly, under the statutory requirement which
mandates that GAO identify federal programs, agencies, offices, and
initiatives with duplicative goals and activities within departments and
government-wide, 4 we committed to study the extent to which federal
departments and agencies are measuring and reporting enterprise
architecture outcomes and benefits.

To address our objective, we analyzed 27 major departments’ and
agencies’ 5 documents describing their enterprise architecture goals and
purposes and their approaches to measuring and reporting architecture
outcomes and benefits, such as their IT Strategic Plan, Enterprise
Architecture Program Management Plan, and Enterprise Architecture
Value Measurement Plan. We compared the agencies’ approaches to
relevant elements of our Enterprise Architecture Management Maturity
Framework (EAMMF). 6 Further, we reviewed outcomes reported to
agency enterprise architecture oversight officials and analyzed responses
to a 2011 GAO survey about the benefits associated with agencies’
architecture programs. We also discussed our analyses with and obtained
testimonial evidence from cognizant agency officials. A more detailed


2
 40 U.S.C. § 11315; The E-Government Act of 2002 also provided a more detailed
definition of the concept and elements of enterprise architecture. See 44 U.S.C. §§
3601(4) and 3602; OMB Circular A-130 (Nov. 30, 2000); and Chief Information Officers
Council, A Practical Guide to Federal Enterprise Architecture, Version 1.0
(February 2001).
3
GAO-11-318SP.
4
Pub. L. No. 111-139, § 21, 124 Stat. 29 (2010), 31 U.S.C. § 712 Note.
5
 These 27 major departments and agencies are the 24 Chief Financial Officer Act entities
identified in 31 U.S.C. § 901(b), as well as the Departments of the Air Force, Army, and
Navy.
6
 GAO, Organizational Transformation: A Framework for Assessing and Improving
Enterprise Architecture Management (Version 2.0), GAO-10-846G (Washington, D.C.:
August 2010).




Page 2                                         GAO-12-791 Organizational Transformation
             discussion of our objective, scope, and methodology is provided in
             appendix I.

             We conducted this performance audit from November 2011 to September
             2012 in accordance with generally accepted government auditing
             standards. Those standards require that we plan and perform the audit to
             obtain sufficient, appropriate evidence to provide a reasonable basis for
             our findings and conclusions based on our audit objective. We believe
             that the evidence obtained provides a reasonable basis for our findings
             and conclusions based on our audit objective.


             An enterprise architecture is a blueprint that describes the current and
Background   desired states of an organization or functional area in both logical and
             technical terms, as well as a plan for transitioning between the two states.
             An enterprise can be viewed as either a single organization or a
             functional area that transcends more than one organization. An
             architecture can be viewed as the structure (or structural description) of
             any activity. Thus, enterprise architectures are systematically derived and
             captured descriptions depicted in models, diagrams, and narratives. More
             specifically, an architecture describes the enterprise in logical terms (such
             as interrelated business processes and business rules, information needs
             and flows, and work locations and users) as well as in technical terms
             (such as hardware, software, data, communications, security attributes,
             and performance standards). It provides these perspectives both for the
             enterprise’s current environment and for its target environment, and it
             provides a transition plan for moving from the current to the target
             environment. Enterprise architectures are a recognized tenet of
             organizational transformation and IT management in public and private
             organizations.

             When employed in concert with other institutional management
             disciplines, such as strategic planning, portfolio-based capital planning
             and investment control, and human capital management, an enterprise
             architecture can greatly increase the chances of configuring an
             organization to promote agility and responsiveness, optimize mission
             performance and strategic outcomes, and address new federal initiatives
             like promoting open and participatory government and leveraging cloud
             computing.




             Page 3                                   GAO-12-791 Organizational Transformation
Federal Legislation and   The Clinger-Cohen Act of 1996, among other things, requires federal
OMB Guidance Pertaining   agency chief information officers (CIO) to develop, maintain, and facilitate
to Establishment of an    the implementation of IT architectures. 7 Subsequent OMB guidance more
                          broadly interpreted IT architecture as an enterprise architecture. 8 In
Enterprise Architecture
                          September 1999, the federal CIO Council published the Federal
                          Enterprise Architecture Framework, 9 which provided federal agencies
                          with a common construct for their architectures to facilitate the
                          coordination of common business processes, technology insertion,
                          information flows, and system investments among federal agencies. The
                          framework defined a collection of interrelated models for describing
                          multiorganizational functional segments of the federal government. 10
                          Further, in 2000 and 2001, the federal CIO Council developed enterprise
                          architecture guidance focused on assessing an IT investment’s
                          compliance with an architecture 11 as well as guidance that addressed the
                          end-to-end steps associated with developing, maintaining, and
                          implementing an architecture program. 12

                          OMB is responsible for overseeing the development of enterprise
                          architectures within and across federal agencies. 13 In February 2002, it


                          7
                           40 U.S.C. § 11315. According to GAO’s EAMMF, such architectures provide an important
                          means of integrating business processes and agency goals with IT.
                          8
                           See for example OMB, Information Technology Architectures, Memorandum M-97-16
                          (June 18, 1997), rescinded with the update of OMB Circular A-130 (Nov. 30, 2000) , which
                          requires that agencies document and submit their enterprise architecture to OMB. Chief
                          Information Officers Council, Architecture Alignment and Assessment Guide (October
                          2000). Chief Information Officers Council, A Practical Guide to Federal Enterprise
                          Architecture, Version 1.0 (February 2001).
                          9
                          Federal Enterprise Architecture Framework, Version 1.1 (September 1999).
                          10
                             The most recent revision to the Federal Enterprise Architecture Framework (Version 2.0)
                          is included in OMB’s Common Approach to Federal Enterprise Architecture, which is
                          discussed subsequently in this report.
                          11
                            Chief Information Officers Council, Architecture Alignment and Assessment Guide
                          (October 2000).
                          12
                            Chief Information Officers Council, A Practical Guide to Federal Enterprise Architecture,
                          Version 1.0 (February 2001).
                          13
                            The E-Government Act of 2002 provided a more detailed definition of the concept and
                          elements of enterprise architecture and established the OMB Office of Electronic
                          Government and assigned it, among other things, responsibilities for overseeing the
                          development of enterprise architectures within and across federal agencies. See 44
                          U.S.C. § 3601(4) and 44 U.S.C § 3602(f)(14).




                          Page 4                                           GAO-12-791 Organizational Transformation
established the Federal Enterprise Architecture program. According to
OMB, the program is intended to facilitate government-wide improvement
through cross-agency analysis and identification of duplicative
investments, gaps, and opportunities for collaboration, interoperability,
and integration within and across agency programs. Federal enterprise
architecture reference models are intended to inform agency efforts to
develop their agency-specific enterprise architectures and enable
agencies to ensure that their proposed investments are not duplicative
with those of other agencies and to pursue, where appropriate, joint
projects. In 2007, OMB issued the Federal Enterprise Architecture
Practice Guidance 14 to provide high-level overviews of architecture
concepts, descriptions of the content included in architecture work
products, and direction on developing and using architectures, including
measuring enterprise architecture program value.

According to the latest version of OMB’s Enterprise Architecture
Assessment Framework (version 3.1, dated June 2009), 15 its purpose is
to provide the measurement areas and criteria for federal agencies to use
in realizing architecture-driven performance improvements and outcomes
(e.g., improving mission performance; saving money and avoiding costs;
enhancing the quality of agency investment portfolios; improving the
quality, availability, and sharing of data and information; and increasing
the transparency of government operations). To accomplish this, the
framework uses key performance indicators to assess architecture
maturity or effectiveness relative to three capability areas—completion,
use, and results. Each capability area contains a set of key performance
indicators and associated outcomes, as well as criteria for gauging
progress in meeting the outcomes. In particular, according to the
framework, as part of the results capability area, agencies should
measure actual results attributed to the architecture, and therefore the
effectiveness and value of architecture activities. However, to reduce the
reporting burden on agencies, in August 2009, OMB issued a
memorandum that stated that agencies were no longer required to
provide self-assessments of enterprise architecture completion, use, and
results to OMB.




14
 OMB, Federal Enterprise Architecture Practice Guidance (November 2007).
15
  OMB, Improving Agency Performance Using Information and Information Technology
(Enterprise Architecture Assessment Framework v3.1) (June 2009).




Page 5                                      GAO-12-791 Organizational Transformation
In May 2012, OMB released the Common Approach to Federal Enterprise
Architecture 16 to promote increased levels of mission effectiveness by
standardizing the development and use of architectures within and
between federal agencies. The approach stresses that enterprise
architecture can enable service delivery, functional integration, and
resource optimization, and can be an authoritative reference for the
design and documentation of systems and services. According to a
memorandum accompanying the Common Approach, each agency is to
submit to OMB by August 31, 2012, an enterprise roadmap that covers
fiscal years 2012 to 2015, to serve as an authoritative reference for IT
portfolio reviews. 17 The roadmap is to map the organization’s strategic
goals to business services and integrate technology solutions across the
agency’s lines of business. It is to discuss the overall architecture and
identify performance gaps, resource requirements, planned solutions,
transition plans, and a summary of the current and future architectures. It
is also to describe the enterprise architecture governance process,
implementation methodology, and documentation framework.

As one of the elements intended to ensure that agency enterprise
architecture programs can be effective in developing solutions that
support planning and decision making, the guidance begins to lay out a
Collaborative Planning Methodology. The methodology entails defining
what benefits will be achieved, when those benefits will be achieved, and
how those benefits will be measured, as well as measuring performance
outcomes against identified metrics. The guidance emphasizes the
importance of measuring the attainment of outcomes, so that the positive
effects (added value) of the architecture program can be identified.
Specifically, each agency’s roadmap is to document how the
effectiveness and efficiency of the program will be measured. The
guidance discusses the difference between outcome and output
measures, and notes that while output measures are important for
indicating an initiative’s progress, outcome measures are needed to
indicate the attainment of goals. According to the Federal Chief




16
 OMB, The Common Approach To Federal Enterprise Architecture (May 2012).
17
  OMB, Memorandum for Federal Agency Chief Information Officers, Increasing Shared
Approaches to Information Technology Services (Washington, D.C., May 2, 2012).
Agencies will be required to submit an updated enterprise roadmap to OMB by April 1st
each year, beginning April 1, 2013.




Page 6                                        GAO-12-791 Organizational Transformation
                          Enterprise Architect, OMB plans to provide agencies more detailed
                          guidance on measuring enterprise architecture value by December 2012.

                          To assist in developing this guidance, the Architecture Subcommittee of
                          the CIO Council’s Strategy and Planning Committee has established a
                          working group to develop an approach for measuring enterprise
                          architecture value through identifying best practices from the public and
                          private sectors. 18 The group plans to draw upon research to create a
                          value measurement program that aligns with the Collaborative Planning
                          Methodology discussed in the Common Approach, and deliver a white
                          paper to OMB on value measurement by the end of fiscal year 2012.
                          According to the Federal Chief Enterprise Architect, the working group’s
                          recommendations will be considered for incorporation into the Common
                          Approach.


GAO’s Enterprise          In August 2010, we issued an Enterprise Architecture Management
Architecture Management   Maturity Framework that provides federal agencies with a common
Maturity Framework        benchmarking tool for assessing the management of their enterprise
                          architecture efforts and developing improvement plans. 19 The framework
                          includes 59 core elements, or building blocks, of enterprise architecture
                          management. The core elements represent practices, structures,
                          activities, and conditions that, when properly employed based on the
                          unique facts and circumstances of each organization and the stated
                          purpose of its enterprise architecture program, can permit that
                          organization to maximize its chances of realizing an architecture’s
                          institutional value. The core elements are categorized into seven
                          hierarchical stages of management maturity and four critical success
                          attribute representations.

                          In particular, core element 41 describes the practice of measuring and
                          reporting enterprise architecture outcomes. The architecture is a strategic
                          asset that represents an investment in the organization’s future and is
                          intended to produce strategic mission value (results and outcomes).
                          Measuring the extent to which this expected value is actually being
                          realized is important to identifying what, if any, enterprise architecture


                          18
                            The CIO Council includes CIOs and Deputy CIOs from 28 federal agencies and is
                          chaired by the Office of Management and Budget Deputy Director for Management.
                          19
                           GAO-10-846G.




                          Page 7                                       GAO-12-791 Organizational Transformation
                          program changes are warranted. Such value can be derived from
                          realizing cost savings through consolidation and reuse of shared services
                          and elimination of antiquated and redundant mission operations,
                          enhancing information sharing through data standardization and system
                          integration, and optimizing service delivery through streamlining and
                          normalization of business processes and mission operations.

                          In addition, core element 58 specifies that enterprise architecture quality-
                          and outcomes-measurement methods should be continuously improved.
                          Organizations should periodically reevaluate their methods for assessing
                          corporate and subordinate architecture quality and program outcomes
                          and address the extent to which program measures and metrics are
                          sufficiently measurable, meaningful, repeatable, consistent, actionable,
                          and aligned with the architecture program’s strategic goals and intended
                          purpose.


Prior GAO Work Has        In 2002 and 2003, we reported on the status of enterprise architectures
Highlighted Federal       government-wide. 20 We found that some federal agencies had begun to
Agency Enterprise         establish the management foundation needed to successfully develop,
                          implement, and maintain an enterprise architecture, but that executive
Architecture Challenges
                          leadership was key to addressing management challenges identified by
                          enterprise architecture programs: (1) overcoming limited executive
                          understanding, (2) inadequate funding, (3) insufficient number of skilled
                          staff, and (4) organizational parochialism. Accordingly, we made
                          recommendations to OMB to improve enterprise architecture leadership
                          and oversight. OMB responded to these recommendations by
                          establishing its Chief Architects Forum to, among other things, share
                          enterprise architecture best practices among federal agencies, and by
                          developing an assessment tool, which it used to periodically evaluate
                          enterprise architecture programs at federal agencies.




                          20
                            GAO, Information Technology: Enterprise Architecture Use across the Federal
                          Government Can Be Improved, GAO-02-6 (Washington, D.C.: Feb. 19, 2002); Information
                          Technology: Leadership Remains Key to Agencies Making Progress on Enterprise
                          Architecture Efforts, GAO-04-40 (Washington, D.C.: Nov. 17, 2003).




                          Page 8                                      GAO-12-791 Organizational Transformation
In 2006, we reviewed enterprise architecture management at 27 major
federal departments and agencies. 21 Our work showed that the state of
architecture development and implementation varied considerably across
departments and agencies, with some having more mature programs
than others. Overall, most agencies had not reached a sufficient level of
maturity in their enterprise architecture development, particularly with
regard to their approaches to assessing each investment’s alignment with
the architecture and measuring and reporting on architecture results and
outcomes. Our 2006 report also noted that challenges we identified in our
earlier reviews continued to present hurdles to effective implementation of
enterprise architecture.

We have also reported on enterprise architecture management and
development at several individual departments and agencies, including
agencies that have demonstrated improvements to their architectures:

•    In 2009, we reported that recent versions of the Department of
     Homeland Security’s (DHS) enterprise architecture had largely
     addressed our prior recommendations aimed at adding needed
     architectural depth and breadth. 22 Nonetheless, we concluded that
     important content, such as prioritized segments and information
     exchanges between critical business processes, was still missing from
     its architecture.

•    Between 2009 and 2012, we conducted several reviews of the
     Department of Housing and Urban Development’s (HUD) enterprise
     architecture and made a number of recommendations for




21
  GAO, Enterprise Architecture: Leadership Remains Key to Establishing and Leveraging
Architectures for Organizational Transformation, GAO-06-831 (Washington, D.C.: Aug. 14,
2006).
22
 GAO, Homeland Security: Despite Progress, DHS Continues to Be Challenged in
Managing Its Multi-Billion Dollar Annual Investment in Large-Scale Information
Technology Systems, GAO-09-1002T (Washington, D.C.: Sept. 15, 2009).




Page 9                                        GAO-12-791 Organizational Transformation
     improvement. 23 Over the course of these reviews, we found that HUD
     had made progress in establishing its architecture, although as of
     September 2012, the department had not yet finalized its updated
     architecture policy, as we had recommended.

•    In September 2011, we reported on the status of the three military
     departments’ (Air Force, Army, and Navy) architecture programs. 24
     We reported that while each of the military departments had long-
     standing efforts to develop and use enterprise architectures, they had
     much to do before their efforts could be considered mature.
     Accordingly, we recommended that the military departments each
     develop a plan for fully satisfying the elements of our framework. The
     Department of Defense (DOD) and the Army concurred with these
     recommendations, but the Air Force and Navy did not. In this regard,
     DOD stated that the Air Force and Navy did not have a valid business
     case that would justify the implementation of all the elements.
     However, we maintained that the recommendation was warranted. To
     date, none of the military departments have addressed our
     recommendations.

•    In April 2012, we reported that the Social Security Administration
     (SSA) had developed an enterprise architecture for years 2011
     through 2016 that captured certain foundational information about the
     current and target environments to assist in evolving existing
     information systems and developing new systems; however, the
     architecture lacked important content that would allow the agency to
     more effectively plan its investments to reach its vision of modernized
     systems and operations. 25 We recommended that SSA develop an


23
  GAO, Information Technology: HUD Needs to Strengthen Its Capacity to Manage and
Modernize Its Environment, GAO-09-675 (Washington, D.C.: July 31, 2009); Information
Technology: HUD Needs to Better Define Commitments and Disclose Risk for
Modernization Projects in Future Expenditure Plans, GAO-11-72 (Washington, D.C.: Nov.
23, 2010); Information Technology: HUD’s Expenditure Plan Satisfies Statutory
Conditions, and Implementation of Management Controls Is Under Way, GAO-11-762
(Washington, D.C.: Sept. 7, 2011); and HUD Information Technology: More Work
Remains to Implement Necessary Management Controls, GAO-12-580T (Washington,
D.C.: Mar. 29, 2012).
24
  GAO, Organizational Transformation: Military Departments Can Improve their Enterprise
Architecture Programs, GAO-11-902 (Washington, D.C.: Sept. 26, 2011).
25
  GAO, Social Security Administration: Improved Planning and Performance Measures
Are Needed to Help Ensure Successful Technology Modernization, GAO-12-495
(Washington, D.C: Apr. 26, 2012).




Page 10                                       GAO-12-791 Organizational Transformation
     enterprise architecture plan that included certain key elements. The
     agency responded that it would comply with recent direction from the
     Federal Chief Architect to deliver an enterprise architecture roadmap
     that meets OMB standards.

In addition to our evaluation of agency-specific enterprise architectures, we
have reported on the need for federal agencies to measure and report
architecture outcomes. Specifically, in March 2011, we reported that while
some progress had been made in improving the content and use of
departments’ and agencies’ architectures, more time was needed for
agencies to fully realize the value of having well-defined and implemented
architectures. 26 We noted that some agencies had reported that they were
addressing the EAMMF core element associated with measuring and
reporting enterprise architecture results and outcomes and had realized
significant financial benefits. For example, we reported that the Department
of the Interior had demonstrated that it was using its enterprise architecture
to modernize agency IT operations and avoid costs through enterprise
software license agreements and hardware procurement consolidation,
which had resulted in reported financial benefits of at least $80 million.
However, we concluded that over 50 percent of the departments and
agencies had yet to fully address this element.

In February 2012, we again reviewed the extent to which major federal
agencies had reported financial benefits from the use of enterprise
architecture. We found that four agencies (in addition to the Department
of Interior) had done so. These four agencies were the Department of
Health and Human Services (HHS), which, facilitated by its architecture
program, moved to a new telecommunications contract, resulting in a
savings of about $21 million; the Nuclear Regulatory Commission (NRC),
which avoided an estimated $1.3 million cost in 2011 by eliminating
duplicative staff planning systems; DOD, which reported saving $179
million between fiscal years 2008 and 2010 by streamlining Navy
business operations, retiring legacy systems, and moving toward a real-
time paperless business environment for processing vendor payments;
and the Department of Agriculture, which reported savings of $27 million
over 5 years (2011 through 2015) by moving 120,000 e-mail users to a
cloud-based solution. 27 We also noted that 12 agencies had reported


26
 GAO-11-318SP.
27
  Cloud computing is a form of computing that relies on Internet-based services and
resources to provide computing services to customers.




Page 11                                         GAO-12-791 Organizational Transformation
                      financial benefits but had not reliably measured them (i.e., they did not
                      provide supporting documentation), and an additional 10 agencies had
                      not reported financial benefits, although 8 of these agencies reported that
                      they had established or expected to establish a process to measure
                      benefits in the future. 28


                      Most of the 27 major agencies in our current study had yet to periodically
Almost All Agencies   (i.e., regularly and repeatedly, such as monthly, quarterly or annually)
Had Defined the       measure and report enterprise architecture outcomes and benefits. Our
                      framework 29 recognizes that knowing whether architecture outcomes are
Purpose of Their      being achieved requires an approach to measuring the value of
Architectures, but    architecture activities that includes defining the architecture’s intended
Had Yet to Fully      purpose or strategic goals; establishing metrics along with a method to
                      measure architecture outcomes and benefits; and periodically measuring
Measure and Report    and reporting to the agency’s architecture executive committee
Outcomes and          (executive-level representatives from each line of business, who have the
                      authority to commit resources) these outcomes and benefits. While all
Benefits              agencies had fully or partially defined their architecture’s strategic goals
                      or intended purpose, only 3 had fully and 8 had partially established
                      metrics and a method to measure outcomes and benefits. Of the
                      agencies that fully or partially established a method and metrics, 4 had
                      measured and reported outcomes only once, and 1 had periodically (e.g.,
                      monthly) reported on outcomes and benefits. A summary of the 27
                      agencies’ progress in measuring and reporting architecture outcomes and
                      benefits is presented in table 1. For detailed assessments of individual
                      departments and agencies against relevant elements of our framework,
                      see appendix II.




                      28
                        GAO, Follow-up on 2011 Report: Status of Actions Taken to Reduce Duplication,
                      Overlap, and Fragmentation, Save Tax Dollars, and Enhance Revenue, GAO-12-453SP
                      (Washington, D.C.: Feb. 28, 2012).
                      29
                       GAO-10-846G.




                      Page 12                                    GAO-12-791 Organizational Transformation
Table 1: Summary of Agencies’ Progress in Measuring and Reporting Enterprise Architecture Outcomes and Benefits
Relative to Elements of GAO’s EAMMF (Version 2.0)

                                                                  The enterprise     Metrics and a method    Enterprise architecture
                                                                  architecture’s    have been established to outcomes and benefits
                                                                  strategic goals     measure enterprise         are periodically
                                                                    or intended      architecture strategic  measured and reported
                                                                   purpose are      mission value (outcomes    to the architecture
Agency                                                                defined             and benefits)       executive committee
Department of Agriculture                                                  ●                  ○                         ○
Department of the Air Force                                                ●                  ○                         ○
Department of the Army                                                     ●                  ◐                         ○
Department of Commerce                                                     ●                  ◐                         ○
Department of Defense – Business Enterprise Architecture                   ●                  ○                         ○
Department of Defense - Enterprise Architecture
                                               a
                                                                           ●                  ○                         ○
Department of Education                                                    ●                  ◐                         ◐
Department of Energy                                                       ●                  ○                         ○
Department of Health and Human Services                                    ●                  ●                         ◐
Department of Homeland Security                                            ●                  ○                         ○
Department of Housing and Urban Development                                ●                  ●                         ◐
Department of the Interior                                                 ●                  ○                         ○
Department of Justice                                                      ●                  ○                         ○
Department of Labor                                                        ●                  ○                         ○
Department of the Navy                                                     ●                  ○                         ○
Department of State                                                        ●                  ○                         ○
Department of Transportation                                               ●                  ◐                         ○
Department of the Treasury                                                 ●                  ◐                         ◐
Department of Veterans Affairs                                             ◐                  ○                         ○
Environmental Protection Agency                                            ●                  ○                         ○
General Services Administration                                            ●                  ◐                         ○
National Aeronautics and Space Administration                              ●                  ○                         ○
National Science Foundation                                                ●                  ○                         ○
Nuclear Regulatory Commission                                              ●                  ◐                         ○
Office of Personnel Management                                             ●                  ◐                         ○
Small Business Administration                                              ●                  ○                         ○
Social Security Administration                                             ●                  ○                         ○
US Agency for International Development                                    ●                  ●                         ●
                                          Source: GAO analysis of agency data.

                                          ● Satisfied ◐ Partially Satisfied ○ Not Satisfied




                                          Page 13                                              GAO-12-791 Organizational Transformation
                          Note: Agencies fully satisfied the relevant elements of our framework by providing sufficient
                          documentation to verify that all aspects of the elements were met. Agencies partially satisfied the
                          elements by providing documentation to verify that at least some aspect of the elements was
                          satisfied. Elements that were neither fully nor partially satisfied were judged to be not satisfied.
                          a
                           According to DOD, the DOD Enterprise Architecture is a federation of descriptions that provide
                          context and rules for accomplishing the mission of the department, and the DOD Business Enterprise
                          Architecture is the blueprint to guide and constrain investments as they relate to or impact business
                          operations.




Almost All Agencies Had   Before an agency knows what outcomes it should measure, it needs to
Defined the Goals or      define the purpose or expected value (i.e., goals) of its architecture. The
Purposes of Their         purpose can include, among other things, consolidating the organization’s
                          IT infrastructure, normalizing and integrating its data and promoting
Enterprise Architecture   information sharing, reengineering core business or mission functions and
                          processes, modernizing applications and sharing services, modernizing
                          the entire IT environment, and transforming how the organization
                          operates. Expected value from implementation of enterprise architecture
                          can include, for example, reduced operating costs, enhanced ability to
                          quickly and less expensively change to meet shifting external
                          environment and new business demands or opportunities, or improved
                          alignment between operations and strategic goals. Twenty-six of the
                          agencies we reviewed had fully defined their architecture goals or
                          purposes. The following are examples of the goals or purposes defined
                          by these agencies:

                          •    The Department of Energy’s goals include identifying, reusing, and
                               leveraging, where possible, existing and planned technology and
                               infrastructure components across the department and identifying
                               areas, through capital planning and investment control and enterprise
                               architecture integration analysis, to reduce costs, identify redundancy,
                               and increase system and process effectiveness.

                          •    HHS’s goals include enabling improved mission and business
                               outcomes by providing products to support sound decisions, business
                               processes, and effective solutions; enabling the optimized use of
                               resources; and increasing interoperability and information sharing
                               within HHS and between HHS and external stakeholders.

                          •    HUD’s goals include improving the efficiency and effectiveness of the
                               department’s programs; simplifying its IT environments by promoting
                               standards and sharing and reusing common technologies; improving
                               interoperability by establishing enterprise-wide standards; and
                               reducing system development and operation and maintenance costs




                          Page 14                                                 GAO-12-791 Organizational Transformation
    by eliminating duplicative investments, promoting sharing of common
    services, and establishing department-wide standards.

•   The Department of Justice’s purpose includes identifying redundant
    legacy programs to either retire or migrate to an enterprise solution,
    thereby reducing the complexity and cost of the IT environment.

•   The Department of Labor’s purpose is to use its enterprise
    architecture process with its capital planning and investment
    management process to ensure that investments support strategic
    goals and are not duplicative of existing business solutions. Using this
    approach, the department plans to identify duplicative resources and
    investments, gaps, and opportunities for internal and external
    collaboration resulting in operational improvements and cost-effective
    solutions to business requirements.

•   The Department of Transportation’s goal is to use its architecture as a
    decision-making tool to support business plan development and
    identify areas of duplication and inefficiencies in the department.

•   The General Services Administration’s (GSA) goals are to increase
    system interoperability and cost efficiencies, reduce duplication, and
    increase innovation.

•   The National Science Foundation’s goals include improving utilization
    of IT resources by eliminating duplicative investments and promoting
    the sharing of common services and standards.

•   The U.S. Agency for International Development’s (USAID) goals
    include facilitating analysis of the agency’s IT environment, including
    IT hardware, software, and enterprise applications, to promote the
    effective and efficient deployment of IT services.

However, one agency (Department of Veterans Affairs) had only partially
defined its architecture’s purpose. The Department of Veterans Affairs
(VA) is in the process of developing an architecture program overview
statement and guiding principles. Specifically, according to draft
documentation, the department’s architecture is to guide efficient,
effective, and interoperable implementation of the department’s vision of
providing seamless delivery of benefits and services to veterans.
According to department officials, these architecture principles are
expected to be finalized and formally released by September 30, 2012.




Page 15                                  GAO-12-791 Organizational Transformation
                            Because they had defined the goals or purposes of their architectures,
                            almost all of the agencies had taken an important first step toward
                            establishing metrics and a method for measuring architecture outcomes
                            and benefits.


More than Half of the       Measuring the extent to which the expected value is actually being
Agencies Had Not Yet        realized is important to identifying what, if any, architecture program
Established Metrics and a   changes are warranted. According to our framework, agencies should
                            establish measurable, meaningful, repeatable, consistent, and actionable
Method for Measuring        metrics that align with the architecture’s intended purpose or strategic
Enterprise Architecture     goals and document a methodology that provides the steps to be followed
Value                       to consistently and repeatedly measure architecture outcomes and
                            benefits. Further, according to OMB guidance, metrics should measure
                            outcomes (i.e., results of products and services such as benefits to
                            Congress and the American taxpayer), or expected value, rather than
                            output (i.e., direct products and services).

                            Of the federal agencies that we reviewed, three had fully established
                            metrics and a method to measure architecture outcomes and benefits,
                            while eight had partially done so. Specifically, HHS, HUD, and USAID had
                            fully established metrics and a method for measuring and reporting
                            enterprise outcomes and benefits.

                            •     HHS had established a metric to measure the extent to which it
                                  increases the number of services that are reused based on its
                                  enterprise architecture service component reference model. 30 The
                                  department had also established a method for how the metrics are to
                                  be measured, including how they are to be calculated, the data
                                  sources to be used, and targets to be achieved.

                            •     HUD had established a method and metrics to measure the extent to
                                  which the department decreases the number of technology products
                                  that duplicate existing capabilities and the extent to which it has
                                  decreased the number of obsolete systems in its IT inventory, using
                                  its enterprise architecture. The department had also established the
                                  steps to measure results and outcomes, including identifying
                                  appropriate sources, and determining baseline, target, and actual
                                  value measurements.


                            30
                                A service component reference model identifies and classifies IT service components.




                            Page 16                                          GAO-12-791 Organizational Transformation
•   USAID had established metrics and guidance for measuring
    enterprise architecture outcomes, including cost savings and
    avoidance due to process efficiency, technology standardization,
    retirement, and consolidation.

Partial steps had been taken by the other eight agencies. Specifically,
one agency (Army) had established a metric and method—for measuring
the extent to which it reduces the number of applications within data
centers— but only for one of its three segment architectures. The other
seven agencies had established metrics but not a method for measuring
and reporting architecture outcomes and benefits. The metrics for each of
the seven agencies are described below.

•   Commerce established as a metric the IT cost reduction associated
    with adopting enterprise-wide standards.

•   Education established a metric to measure spending on development,
    modernization, and enhancement relative to steady-state spending
    (i.e., the cost to maintain current systems and technologies).

•   Transportation established an expected architecture outcome of
    reduced total cost of ownership of IT investments, and planned to
    measure cost savings and/or cost avoidance identified through
    reviews of business processes, data, applications, and technology.

•   Treasury established architecture metrics associated with its data
    center consolidation initiative, including the extent to which it
    decreases the number of servers, increases the percentage of
    operating systems that are virtual, and decreases the demand for data
    center square footage.

•   GSA established as a metric the extent to which the agency is
    increasing its use of IT standards.

•   NRC established a metric to measure progress toward having
    common access controls by measuring the reduction in passwords
    and sign-ons.

•   Office of Personnel Management (OPM) has established cost savings
    as a metric to measure architecture outcomes.

The remaining 16 agencies in our study had not established metrics or a
method for measuring architecture outcomes. While some of these



Page 17                                GAO-12-791 Organizational Transformation
                             agencies had established metrics that measure output, such as the
                             percentage or number of segments and solution architectures or
                             architecture artifacts that have been reviewed and approved by the
                             enterprise architecture program, these metrics do not measure outcomes
                             (i.e., results of enterprise architecture products and services such as
                             benefits to Congress and the American taxpayer) of the program.

                             Without established metrics and a method to measure architecture
                             outcomes, agencies cannot ensure that they are able to consistently and
                             repeatedly measure outcomes.


Five Agencies Had Fully or   Using established metrics and a documented method, architecture
Partially Measured and       outcomes should be periodically measured and reported to senior
Reported Architecture        executives. We have previously found that executive leadership was key to
                             addressing management challenges identified by enterprise architecture
Value                        programs, such as overcoming limited executive understanding and
                             inadequate funding. As such, architecture outcomes and benefits should be
                             periodically reported to senior agency executives who are responsible for
                             making decisions about the architecture program and whether to invest
                             additional resources or make changes to the program.

                             Of the 27 agencies in our review, 1 had consistently and repeatedly
                             measured and reported, using established metrics, outcomes of its
                             architecture program. Specifically, USAID had reported monthly the
                             measured outcomes to its CIO and through an internal agency website
                             established for CIO staff. Outcomes reported include cost savings of
                             $12.3 million and cost avoidance of $9.5 million as a result of transitioning
                             disparate human resource systems to a human resource shared services
                             center using enterprise architecture. The agency also reported estimated
                             savings of $15.7 million from moving its e-mail service to a cloud-based
                             solution, which was recommended by the architecture team to replace
                             multiple installations of the current e-mail solution.

                             Two other agencies had measured and reported outcomes with an
                             established method, but did so only once. Specifically,

                             •   HHS determined, based on its enterprise architecture service
                                 component reference model, and reported to the CIO in November
                                 2010 that 16 percent of its services were reused. However, the
                                 department had not measured the metric again and thus did not know
                                 the extent to which it had increased its reuse of services since 2010.




                             Page 18                                  GAO-12-791 Organizational Transformation
•    HUD measured and submitted its architecture value measurement
     report for fiscal year 2011 to a department executive committee in
     August 2012, and highlighted areas, based on measurements, where
     additional focus and improvement are needed. For example, the
     report noted that HUD had not decreased the number of technology
     products in its enterprise architecture technical reference model 31 that
     duplicate existing capabilities.

While two additional agencies, as described below, had also measured
and reported architecture outcomes once, they did so without an
established method for measuring outcomes, but rather in an ad hoc
manner.

•    Education reported in its October 2011 Office of the CIO Organization
     Performance Review report that development, modernization, and
     enhancement funding in the IT portfolio increased from 10 percent of
     total IT spending in fiscal year 2011, to 13 percent of total IT spending
     in fiscal year 2012 through use of the department’s architecture
     segment modernization planning process. However, Education had
     not established a method for measuring and reporting architecture
     outcomes and benefits. As a result, it cannot ensure that it will be able
     to consistently and repeatedly measure architecture outcomes over
     time.

•    Treasury reported in its E-Government Act Report for fiscal year 2011
     that its enterprise architecture plans focused on reducing duplication
     through its data center consolidation initiative. Accordingly, it reported
     through its CIO to OMB a reduction of 1,283 in the number of servers,
     an increase from 25 percent to 36 percent of operating systems that
     were virtualized, and a reduction in data center square footage of
     15,896 between 2010 and 2011. However, Treasury had not
     established a method for measuring and reporting architecture
     outcomes and benefits. As a result, it cannot ensure that it will be able
     to consistently and repeatedly measure outcomes over time.

The remaining agencies (22) had not yet measured and reported
architecture outcomes to senior executives. Agencies generally cited two
reasons why they had not done so. Specifically, agencies had not



31
  A Technical Reference Model describes the standards, specifications, and technologies
that support the delivery of service components.




Page 19                                        GAO-12-791 Organizational Transformation
                            determined how to attribute discrete outcomes to enterprise architecture
                            when other activities, such as strategic planning, capital planning, and
                            project management may have contributed to the outcomes. In addition,
                            agencies cited an absence of guidance and best practices for how to
                            measure enterprise architecture outcomes. As discussed in the next
                            section, OMB has issued recent enterprise architecture guidance to
                            agencies, but has not yet provided sufficient details on the method and
                            metrics that could be used to measure architecture program outcomes.

                            Collectively, this means that while efforts are underway, without the use
                            of associated measures and metrics by the majority of agencies, the 27
                            major departments and agencies are not positioned to know whether
                            outcomes and benefits are in fact being achieved. Until agencies
                            establish an approach for measuring enterprise architecture outcomes,
                            including a documented method (i.e., steps to be followed) and metrics
                            that are measurable, meaningful, repeatable, consistent, actionable, and
                            aligned with the agency’s enterprise architecture’s strategic goals and
                            intended purpose; and measure and report enterprise architecture
                            outcomes and benefits to top agency officials and to OMB, agency senior
                            executives are less likely to be sufficiently informed about whether to
                            invest additional resources or make changes to the enterprise
                            architecture program.


OMB’s Guidance to           The E-Government Act of 2002 assigned OMB the responsibilities for
Agencies Lacks Sufficient   overseeing the development of enterprise architectures within and across
Details on Measuring        the federal agencies. Since then, OMB has issued guidance and
                            frameworks for developing and using architectures, including a May 2012
Enterprise Architecture     policy and guidance on establishing a common approach to developing
Value                       and using enterprise architectures within and between federal agencies.
                            The policy required each federal agency to submit by August 31, 2012, an
                            enterprise roadmap that reports, among other things, how architecture
                            program effectiveness and efficiency will be measured. However, while
                            this guidance begins to describe an approach for collaboratively
                            identifying, planning for, achieving, and measuring needed organizational
                            outcomes (called the Collaborative Planning Methodology) and discusses
                            the difference between outcome and output measures, it does not provide
                            sufficient details on the method and metrics that could be used to
                            measure architecture program outcomes.

                            As we noted earlier, according to our framework a methodology should
                            provide the steps to be followed to consistently and repeatedly measure
                            architecture outcomes and benefits. While OMB’s collaborative planning


                            Page 20                                GAO-12-791 Organizational Transformation
              methodology emphasizes the importance of measuring benefits and
              describing how they will be measured when planning for and executing
              collaborative projects, it does not call for specific metrics to be used or
              identify steps to be followed to consistently and repeatedly measure
              outcomes and benefits. Further, OMB does not call for agency roadmaps
              to include measurement methods and metrics, and reports on specific
              outcomes and benefits that an agency has achieved or plans to achieve.

              In discussing this matter, the Federal Chief Enterprise Architect agreed with
              our assessment but stated that the methodology was not intended to be
              guidance on measuring architecture value and that more detailed guidance
              was being developed. According to the Federal Chief Architect, the detailed
              guidance on measuring enterprise architecture value is expected to be
              provided to agencies by December 2012, in time to facilitate the
              development of their next roadmap submissions, due in April 2013.

              With the development of clear and sufficiently detailed guidance on
              measuring outcomes by OMB, agencies may be better positioned to
              develop methods and metrics for measuring and reporting the strategic
              value produced by their enterprise architecture programs. Moreover, with
              reports about architecture outcomes and benefits, agency executives
              could increase their understanding of the architecture programs, such that
              warranted changes could be addressed, or the need for expanded
              architecture development and use may be able to be economically
              justified. An established method and metrics to measure outcomes and
              benefits will enable agencies to repeatedly and consistently measure and
              report the extent to which they are achieving value.


              Enterprise architecture value has yet to be measured and reported across
Conclusions   the majority of the federal agencies. While most of the agencies reviewed
              have defined their architecture’s goals or purpose, the majority had yet to
              establish metrics and a method for measuring and reporting architecture
              value. This means that while efforts are underway, the majority of the
              agencies do not know the extent to which they are realizing benefits that
              they have set out to achieve, such as cost savings or avoidance through
              eliminating duplicative investments. Furthermore, most of the agencies
              had not measured and reported outcomes to stakeholders or agency
              executives. Without measurable, meaningful, repeatable, consistent, and
              actionable metrics that align with the architecture’s strategic goals or
              intended purpose and a documented methodology that provides the steps
              to be followed to consistently and repeatedly measure outcomes and
              benefits, senior agency executives may not have the information needed


              Page 21                                  GAO-12-791 Organizational Transformation
                      to determine whether to invest additional resources or make changes to
                      the program. OMB’s forthcoming guidance is an opportunity to overcome
                      the absence of detailed directions to agencies on how they can measure
                      and report enterprise architecture strategic value.


                      To enhance federal agencies’ ability to realize enterprise architecture
Recommendations for   benefits, we recommend the following actions.
Executive Action
                      We recommend that the Secretaries of the Departments of Agriculture,
                      the Air Force, the Army, Commerce, Defense, Education, Energy,
                      Homeland Security, the Interior, Labor, the Navy, State, Transportation,
                      the Treasury, and Veterans Affairs; the Attorney General; the
                      Administrators of the Environmental Protection Agency, General Services
                      Administration, National Aeronautics and Space Administration, and
                      Small Business Administration; the Commissioners of the Nuclear
                      Regulatory Commission and Social Security Administration; and the
                      Directors of the National Science Foundation and the Office of Personnel
                      Management ensure the following two actions are taken:

                      •   fully establish an approach for measuring enterprise architecture
                          outcomes, including a documented method (i.e., steps to be followed)
                          and metrics that are measurable, meaningful, repeatable, consistent,
                          actionable, and aligned with the agency’s enterprise architecture’s
                          strategic goals and intended purpose; and

                      •   periodically measure and report enterprise architecture outcomes and
                          benefits to top agency officials (i.e., executives with authority to
                          commit resources or make changes to the program) and to OMB.

                      In addition, we recommend that the Secretaries of the Departments of
                      Health and Human Services and Housing and Urban Development
                      ensure that enterprise architecture outcomes are periodically measured
                      and reported to top agency officials.

                      To assist agencies in measuring and reporting outcomes achieved
                      through enterprise architecture, we recommend that the Director of OMB
                      ensure that the planned December 2012 guidance for enterprise
                      architecture value measurement and reporting includes

                      •   sufficient details on the method and metrics that agencies could use
                          to measure their architecture program’s value and




                      Page 22                                 GAO-12-791 Organizational Transformation
                     •     a requirement for agencies to include in their April 2013 enterprise
                           roadmap submissions a measurement method (i.e., steps to be
                           followed) and metrics, and report on the outcomes and benefits
                           achieved through enterprise architecture.


                     We received comments on a draft of this report from OMB and the 24 32
Agency Comments      agencies in our study. OMB’s Federal Chief Enterprise Architect stated in
and Our Evaluation   oral comments and via e-mail that OMB agreed with the report and the
                     recommendations. Among the agencies in our study, 5 responded via e-
                     mail that they had no comments on our draft report. 33 One of these
                     agencies—USAID—provided technical comments, which we incorporated
                     as appropriate. An additional 2 agencies provided letters stating that they
                     had no comments on our draft report. Specifically, Labor’s Assistant
                     Secretary for Administration and Management stated in a written
                     response (reproduced in appendix III) that the department had no
                     comments, and Treasury’s Deputy Assistant Secretary for Information
                     Systems and Chief Information Officer stated in a written response
                     (reproduced in appendix IV) that the department had no comments on the
                     draft report but appreciated GAO’s efforts in its development.

                     Among the remaining agencies, 13 agreed with our results. These
                     comments are summarized below.

                     •     USDA’s Acting Chief Information Officer stated in written comments
                           that the department concurred with our findings and recommendations
                           and plans to develop metrics and guidance to comply with OMB
                           guidance on measuring enterprise architecture, when it is provided.
                           USDA’s written comments are reproduced in appendix V.

                     •     Commerce’s Acting Secretary stated in written comments that the
                           department agreed with the general findings and specific
                           recommendations as they relate to the department. Commerce’s
                           written comments are reproduced in appendix VI.

                     •     DOD’s Deputy CIO for Information stated in written comments that the
                           department concurred with our recommendations and is developing



                     32
                          DOD included comments from the departments of the Air Force, Army, and Navy.
                     33
                         Transportation, GSA, NSF, and NRC and USAID.




                     Page 23                                        GAO-12-791 Organizational Transformation
    an enterprise architecture management plan that provides high-level
    processes, including to measure architecture outcomes. DOD’s
    written comments are reproduced in appendix VII.

•   Education’s CIO stated in written comments that the department
    concurred with our recommendations and described steps the
    department plans to take to address the recommendations. For
    example, the department plans to develop, document, and implement
    a measurement and reporting method that will be used to periodically
    monitor its progress toward achieving goals, desired outcomes, and
    benefits. The department also provided technical comments that we
    have incorporated, as appropriate, in the report. Education’s written
    comments are reproduced in appendix VIII.

•   DHS’s Director, Departmental GAO-OIG Liaison Office, stated in
    written comments that the department concurred with our
    recommendations and described actions it plans to take to address
    them. For example, DHS stated that it plans to brief architecture
    outcomes for the goals and objectives outlined in the strategic plan to
    the CIO by October 31, 2012. DHS’s written comments are
    reproduced in appendix IX.

•   Interior’s Assistant Secretary for Policy Management and Budget
    stated in written comments that the department concurred with our
    recommendations. Interior’s written comments are reproduced in
    appendix X.

•   A Management Analyst in Justice’s audit liaison group commented via
    e-mail that the department agreed with our recommendations. The
    official also provided technical comments that we have incorporated,
    as appropriate.

•   State’s Comptroller provided written comments which noted that the
    department concurred with our conclusions and recommendations,
    and described steps being taken or planned to address the
    recommendations. For example, the department plans to implement,
    in fiscal year 2013, a metric to measure reduction in the percentage of
    information exchange elements between critical management
    systems through use of its enterprise architecture. State’s written
    comments are reproduced in appendix XI.

•   VA’s Chief of Staff stated in written comments that the department
    generally agreed with our conclusions and concurred with the
    recommendations. The department also described actions it had


Page 24                                 GAO-12-791 Organizational Transformation
    taken in fiscal year 2012 to re-establish its enterprise architecture
    program and actions it plans to take to continue to mature the
    program in fiscal year 2013 that would begin to address our
    recommendations. VA’s written comments are reproduced in
    appendix XII.

•   NASA’s CIO stated in written comments that the agency concurred
    with the recommendations, and described steps the agency plans to
    take to address them. For example, NASA plans to revise, by June
    2013, its procedural requirements to better align architecture metrics
    and methods to measure outcomes. NASA’s written comments are
    reproduced in appendix XIII.

•   SSA’s Deputy Chief of Staff stated in written comments that the
    agency agreed with the recommendations. SSA’s written comments
    are reproduced in appendix XIV.

•   EPA’s Assistant Administrator and Chief Information Officer stated in
    written comments that the agency agrees with our findings and
    described steps it plans to take to address our recommendations.
    For example, it plans to develop a performance measurement plan,
    which will identify processes to measure enterprise architecture
    outcomes. EPA’s written comments are reproduced in appendix XV.

•   HHS’s Assistant Secretary for Legislation in written comments stated
    that the department concurred with our findings and described actions
    it is taking, and plans to take, to improve architecture value
    measurement. HHS’s written comments are reproduced in appendix
    XVI.

The remaining four agencies provided comments that expressed
concerns with certain aspects of our results. These comments are
summarized below.

•   Energy’s Chief Architect provided written comments in which the
    department stated that it had established metrics and a method for
    measuring architecture value and that its efforts justify a partially-
    satisfied rating. However, our study found that, although the
    department has submitted its Enterprise Modernization Roadmap to
    OMB, the roadmap includes potential architecture program metrics
    that are still being defined and have yet to be finalized and approved.
    The department also stated that it had achieved several
    accomplishments which justified a partially-satisfied rating for
    measuring and reporting architecture outcomes and benefits. In this


Page 25                                   GAO-12-791 Organizational Transformation
    regard, the department highlighted accomplishments such as
    collecting and reporting architecture success stories to a working-sub
    group of its Information Technology Council, which includes senior-
    level IT management from the offices of the Chief Financial Officer
    and the CIO. However, we found that these accomplishments and
    success stories are not based on an established set of metrics and a
    documented, consistently applied methodology for measuring and
    reporting architecture outcomes. As a result, the department cannot
    ensure that it will be able to consistently and repeatedly measure
    outcomes over time. Thus, we stand by our findings. Energy’s written
    comments are reproduced in appendix XVII.

•   HUD’s CIO provided written comments on the report stating that the
    department has complied with our recommendation that enterprise
    architecture outcomes be periodically measured and reported to top
    agency officials. Specifically, the department stated the fiscal year
    2011 report on outcomes was submitted to an executive committee in
    August 2012. It also stated that an EA Value Measurement Plan will
    be issued annually and results of the measures in the plan will be
    documented in an annual report for the fiscal year. However, while the
    department has completed and submitted its first report (i.e., for fiscal
    year 2011) to an executive committee, it has yet to measure and
    report on the metrics again, and therefore does not know the extent to
    which it has achieved its target outcomes. In addition, the official
    commented on the statement in the background of our report that the
    department had not yet finalized its architecture policy, as we had
    previously recommended. The official commented that a policy has
    been in place since April 2002. However, as the official stated, its
    updated policy has yet to be approved. As a result, we stand by the
    statement. HUD’s written comments are reproduced in appendix
    XVIII.

•   A Senior Analyst, e-mailing on behalf of OPM’s Office of the CIO,
    provided comments in which the agency stated that savings through
    enterprise architecture are being measured and reported. However, it
    provided no evidence to support this statement and stated that more
    information will be available once the department implements a
    revised enterprise architecture roadmap, expected by the end of
    December 2012. As a result, we did not change our finding.

•   A Program Manager in SBA’s Office of Congressional and Legislative
    Affairs provided e-mail comments. Specifically, in comments on our
    finding that stated the agency had not defined its enterprise
    architecture strategic goals or intended purpose, SBA stated that its


Page 26                                  GAO-12-791 Organizational Transformation
    architecture’s purpose and goals are defined and it provided
    supporting documentation in this regard. In response, we updated the
    finding and recommendation accordingly. In comments on our finding
    that the agency had not established a method or metrics to measure
    outcomes and benefits, SBA provided its Capital Planning and
    Investment Control Policy Guide and its fiscal year 2011 Summary of
    Performance and Financial Information. However, neither of these
    documents demonstrated a method and metrics for measuring
    architecture outcomes. In comments on our finding that the agency is
    not periodically measuring architecture outcomes and benefits, the
    agency stated that outcomes are measured and reported as part of
    the integrated enterprise architecture-capital planning and investment
    control effort through the Business Technology Investment Advisory
    Committee and the Business Technology Investment Council. While
    the agency provided some documentation, it did not provide
    requested examples of reports submitted to the officials. The agency
    also added that outcomes are reported in its annual performance
    report and provided documentation. While we agree that some
    outcomes are documented, the report does not highlight architecture-
    related outcomes, and SBA did not provide documentation linking the
    outcomes to its enterprise architecture. Therefore, we did not change
    our findings relative to establishing a method and metrics and
    measuring and reporting architecture outcomes.




Page 27                                 GAO-12-791 Organizational Transformation
We are sending copies of this report to other interested congressional
committees; the Director of the Office of Management and Budget; the
Secretaries of Agriculture, the Air Force, the Army, Commerce, Defense,
Education, Energy, Health and Human Services, Homeland Security,
Housing and Urban Development, the Interior, Labor, the Navy, State,
Transportation, the Treasury, and Veterans Affairs; the Attorney General;
the Administrators of the Environmental Protection Agency, General
Services Administration, National Aeronautics and Space Administration,
Small Business Administration, and U.S. Agency for International
Development; the Commissioners of the Nuclear Regulatory Commission
and the Social Security Administration; and the Directors of the National
Science Foundation and Office of Personnel Management. In addition,
the report will be available at no charge on the GAO website at
http://www.gao.gov.

If you or your staffs have questions on matters discussed in this report,
please contact me at (202) 512-6304 or melvinv@gao.gov. Contact points
for our Offices of Congressional Relations and Public Affairs may be
found on the last page of this report. GAO staff who made contributions to
this report are listed in appendix XIX.




Valerie C. Melvin
Director
Information Management and
   Technology Resources Issues




Page 28                                GAO-12-791 Organizational Transformation
Appendix I: Objective, Scope, and
              Appendix I: Objective, Scope, and
              Methodology



Methodology

              Our objective was to determine the extent to which federal agencies are
              measuring and reporting enterprise architecture outcomes and benefits.
              To accomplish the objective, we focused on 28 enterprise architecture
              programs relating to 27 major departments and agencies. These 27
              included the 24 departments and agencies identified in the Chief
              Financial Officers Act, 1 as well as the Departments of the Air Force,
              Army, and Navy. At the Department of Defense (DOD), we reviewed two
              department-wide architecture programs—the Business Enterprise
              Architecture and the DOD Enterprise Architecture. Table 2 identifies the
              agencies included in our study. These agencies were also included in our
              2006 review of agencies’ management maturity. 2

              Table 2: Agencies Included in Our Study

               Agency
               Department of Agriculture
               Department of the Air Force
               Department of the Army
               Department of Commerce
               Department of Defense (Business Enterprise Architecture and Enterprise Architecture)
               Department of Education
               Department of Energy
               Department of Health and Human Services
               Department of Homeland Security
               Department of Housing and Urban Development
               Department of the Interior
               Department of Justice
               Department of Labor
               Department of the Navy
               Department of State
               Department of Transportation
               Department of the Treasury
               Department of Veterans Affairs



              1
               See 31 U.S.C. § 901(b).
              2
               GAO, Enterprise Architecture: Leadership Remains Key to Establishing and Leveraging
              Architectures for Organizational Transformation, GAO-06-831 (Washington, D.C.:
              Aug. 14, 2006).




              Page 29                                        GAO-12-791 Organizational Transformation
Appendix I: Objective, Scope, and
Methodology




 Agency
 Environmental Protection Agency
 General Services Administration
 National Aeronautics and Space Administration
 National Science Foundation
 Nuclear Regulatory Commission
 Office of Personnel Management
 Small Business Administration
 Social Security Administration
 United States Agency for International Development
Source: GAO.



We reviewed the responses to a survey we administered in May 2011, of
federal agencies’ efforts to measure and report enterprise architecture
results and outcomes. The purpose of the survey was to follow up with
the agencies we reviewed in 2006, about the costs and benefits
associated with their enterprise architecture programs. 3 In addition, we
requested and reviewed documents describing each agency’s enterprise
architecture program, focusing on the purpose and goals of the programs
and the methods and metrics used to measure outcomes, such as IT
strategic plans, program management plans, enterprise transition plans,
enterprise modernization roadmaps, and value measurement plans. We
analyzed the extent to which the documentation satisfied elements
related to outcomes measurement and reporting in version 2.0 of our
Enterprise Architecture Management Maturity Framework (EAMMF). 4
Specifically, we assessed agencies against elements of the framework
related to defining the architecture’s intended purpose or strategic goals,
establishing a method and metrics to measure architecture strategic
mission value (outcomes and benefits), and periodically measuring and
reporting outcomes and benefits to an architecture executive committee.
We also reviewed outcomes and benefits reported to agency architecture
oversight officials, for example, in value measurement reports or
performance measurement reports. We assessed the reliability of the
reported outcomes and benefits by discussing with agency officials the



3
 GAO-06-831.
4
 GAO, Organizational Transformation: A Framework for Assessing and Improving
Enterprise Architecture Management (Version 2.0), GAO-10-846G (Washington, D.C.:
August 2010).




Page 30                                          GAO-12-791 Organizational Transformation
Appendix I: Objective, Scope, and
Methodology




method and data used to determine them, and by reviewing relevant
documents, such as business cases and return on investment analyses.

To guide our analysis, we defined detailed evaluation criteria for
determining whether a given element was fully satisfied, partially satisfied,
or not satisfied. To fully satisfy an element, sufficient documentation had
to be provided to permit us to verify that all aspects of the element were
met. To partially satisfy an element, sufficient documentation had to be
provided to permit us to verify that at least some aspects of the element
were met. Elements that were neither fully nor partially satisfied were
judged to be not satisfied.

Our evaluation included independently analyzing the extent to which each
agency had satisfied the elements using the survey responses and
supporting documentation as a starting point. We then corroborated the
analyses with supporting documentation, sought additional information as
necessary through interviews with the agencies’ architecture officials,
obtained and reviewed additional documentation as appropriate, and
refined our determinations about the degree to which each element was
satisfied. Finally, we shared with agencies preliminary versions of the
analyses that appear in this report as appendix II, and made further
adjustments, as appropriate, based on additional discussions and
supporting documentation. We also met with the Federal Chief Enterprise
Architect to discuss current efforts and plans to guide federal agencies’
efforts to measure and report enterprise architecture outcomes and
benefits.

We conducted our work from November 2011 to September 2012 in
accordance with generally accepted government auditing standards.
Those standards require that we plan and perform the audit to obtain
sufficient, appropriate evidence to provide a reasonable basis for our
findings and conclusions based on our audit objective. We believe that
the evidence obtained provides a reasonable basis for our findings and
conclusions based on our audit objective.




Page 31                                  GAO-12-791 Organizational Transformation
Appendix II: Detailed Assessments of Individual
                 Appendix II: Detailed Assessments of
                 Individual Departments and Agencies against
                 Relevant Elements of Our Enterprise
Departments and Agencies against Relevant Elements
                 Architecture Management Maturity Framework


of Our Enterprise Architecture Management Maturity
Framework
                 The following sections summarize the extent to which each of the 27
                 departments and agencies addressed elements in GAO’s Enterprise
                 Architecture Management Maturity Framework (EAMMF) that pertain to
                 measuring and reporting enterprise architecture outcomes and benefits.

                 The assessments given for each element are defined as follows:
                 ●   The agency or department fully satisfied the element.
                 ◐ The agency or department satisfied some, but not all, aspects of the
                     element.
                 ○ The agency or department did not satisfy any aspect of the element.




                 Page 32                                       GAO-12-791 Organizational Transformation
                                           Appendix II: Detailed Assessments of
                                           Individual Departments and Agencies against
                                           Relevant Elements of Our Enterprise
                                           Architecture Management Maturity Framework




Department of Agriculture                  Table 3 shows the Department of Agriculture’s (USDA) satisfaction of
                                           relevant framework elements in version 2.0 of GAO’s EAMMF.

Table 3: Department of Agriculture Satisfaction of EAMMF Elements

Element                                        Satisfied? Summary
The enterprise architecture’s intended
purpose or strategic goals are defined.
                                                    ●           USDA has defined its architecture’s purpose and goals. Specifically,
                                                                according to the department’s IT strategic plan for 2012 through 2016,
                                                                enterprise architecture and portfolio management practices are to be used
                                                                to address mission needs in a cost-effective and efficient manner. In
                                                                addition, according to the plan, the enterprise architecture program is to be
                                                                used as a strategic enabler to drive planning activities, provide insights,
                                                                and identify improvement opportunities for consolidation and reuse.
A method and metrics have been established
to measure enterprise architecture strategic
                                                    ○           The department has not established a method and metrics for measuring
                                                                enterprise architecture outcomes; however, according to agency officials,
mission value (outcomes and benefits).                          it plans to do so. In particular, officials said the department is planning to
                                                                integrate the capital planning, budget, and enterprise architecture
                                                                processes and is working with the Federal Chief Information Officer (CIO)
                                                                in the area of shared services using architecture. In addition, according to
                                                                the department’s IT strategic plan, the department had planned to
                                                                establish and periodically report on enterprise architecture program
                                                                metrics by the end of fiscal year 2012. Officials explained that they now
                                                                expect to complete this effort by the end of fiscal year 2013 because they
                                                                are waiting for the Office of Management and Budget to provide additional
                                                                guidance.
Enterprise architecture outcomes and
benefits are periodically measured and
                                                    ○           The department has not periodically measured and reported enterprise
                                                                architecture outcomes.
reported to the agency’s enterprise
architecture executive committee.
                                           Source: GAO analysis of agency-provided data.




                                           Page 33                                                 GAO-12-791 Organizational Transformation
                                           Appendix II: Detailed Assessments of
                                           Individual Departments and Agencies against
                                           Relevant Elements of Our Enterprise
                                           Architecture Management Maturity Framework




Department of the Air                      Table 4 shows the Department of the Air Force’s satisfaction of relevant
Force                                      framework elements in version 2.0 of GAO’s EAMMF.

Table 4: Department of the Air Force Satisfaction of EAMMF Elements

Element                                        Satisfied? Summary
The enterprise architecture’s intended
purpose or strategic goals are defined.
                                                    ●           The December 2009 Air Force Architecting Concept of Operations defines
                                                                the vision and goal for the department’s architecture as follows:
                                                                • Vision: to enable the delivery of timely, relevant, unambiguous
                                                                   information to support informed decision making by Air Force leaders to
                                                                   maximize military capabilities while optimizing allocation of resources.
                                                                • Goal: to use architecture to unravel the complexity of systems,
                                                                   processes, and programs to reveal their interdependent relationships to
                                                                   decision makers, in an easily understandable format, so they may be
                                                                   adequately considered as decisions are made.
                                                                Further, according to the Concept of Operations, the architecture is to be
                                                                used as a tool to eliminate redundancy, build efficiency, and maximize
                                                                resource distribution to ultimately increase the combat effectiveness of the
                                                                Air Force.
A method and metrics have been established
to measure enterprise architecture strategic
                                                    ○           Air Force officials reported that the department has not yet established a
                                                                method or metrics to measure and report enterprise architecture outcomes
mission value (outcomes and benefits).                          and benefits. Officials stated that they have had a 60 percent architecting
                                                                division personnel turnover rate since June 2011, and have not been able
                                                                to identify industry-recognized enterprise architecture results metrics.
                                                                Nonetheless, officials stated that they anticipate documenting potential
                                                                metrics in October 2013.
Enterprise architecture outcomes and
benefits are periodically measured and
                                                    ○           The Air Force has yet to measure and report enterprise architecture
                                                                outcomes and benefits.
reported to the agency’s enterprise
architecture executive committee.
                                           Source: GAO analysis of agency-provided data.




                                           Page 34                                                GAO-12-791 Organizational Transformation
                                          Appendix II: Detailed Assessments of
                                          Individual Departments and Agencies against
                                          Relevant Elements of Our Enterprise
                                          Architecture Management Maturity Framework




Department of the Army                    Table 5 shows the Department of the Army’s satisfaction of relevant
                                          framework elements in version 2.0 of GAO’s EAMMF.

Table 5: Department of the Army Satisfaction of EAMMF Elements

Element                                     Satisfied? Summary
The enterprise architecture’s intended
purpose or strategic goals are defined.
                                                ●       Army has defined the purpose and goals for each of its three segment
                                                        architectures. According to Army officials, the collective purpose of the
                                                        segment architectures is to make performance-based and cost-informed
                                                        decisions that lead to the optimization of operations and technical
                                                        environments. The purpose and goals for each of the three segment
                                                        architectures has been defined as follows:
                                                        • Generating Force. According to Army’s 2011 Business Transformation
                                                           Plan, the purpose of this architecture is to drive integration across
                                                           functional domains, ensure integration between the Generating and
                                                           Operating Forces, and inform stakeholders on acquisition decisions
                                                           pertaining to the migration of legacy functionality to Army’s Enterprise
                                                           Resource Planning solution. Expected benefits are the streamlining of
                                                           end-to-end business processes aligned to the business enterprise
                                                           architecture and the elimination or reduction of the need to tailor
                                                           commercial-off-the-shelf systems.
                                                        • Operating Force. According to Army’s 2004 Architecture Approval and
                                                           Development memorandum, the purpose of this architecture is to assist
                                                           in managing systems that support the current and future Army and to
                                                           become a critical component in prioritizing and synchronizing Army-
                                                           wide efforts. According to Army’s 2011 Network Integration Roles,
                                                           Responsibilities, and Functions memorandum, architecture analysis will
                                                           be used to identify duplicative systems and incompatible
                                                           implementations and to integrate requirements, platforms, and network
                                                           capabilities across program offices, among other things.
                                                        • Network. According to Army’s January 2011 Network Enterprise
                                                           Architecture Foundation document, the purpose of this architecture is
                                                           to provide relevant, trusted, affordable, and timely information to
                                                           decision makers that support Army development and transformation,
                                                           and help sustain the Army’s transformation by facilitating an end-to-end
                                                           alignment of capabilities and investments in support of Army planning
                                                           and prioritization documents. Also, according to Army’s 2010 Global
                                                           Network Enterprise Construct Implementation Plan, the network
                                                           architecture program’s strategic initiatives include federating and
                                                           integrating networks, enforcing standards, and aligning Army and
                                                           federal data center consolidation initiative goals and objectives.




                                          Page 35                                        GAO-12-791 Organizational Transformation
                                           Appendix II: Detailed Assessments of
                                           Individual Departments and Agencies against
                                           Relevant Elements of Our Enterprise
                                           Architecture Management Maturity Framework




Element                                        Satisfied? Summary
A method and metrics have been established
to measure enterprise architecture strategic
                                                    ◐           Army has established metrics to measure outcomes and benefits of its
                                                                Network segment architecture related to its data center consolidation
mission value (outcomes and benefits).                          initiative. The metrics measure the extent to which the number of data
                                                                centers are closed each year, and the extent to which the number of
                                                                servers, the amount of floor space, and energy usage and its associated
                                                                costs are reduced. In addition, the Army has established a metric to
                                                                measure the extent to which it reduces the number of applications on data
                                                                servers and within data centers. It has also established, in its January
                                                                2012 Performance Plan for Reducing the Resources Required for Data
                                                                Servers and Centers, a method for measuring the reduction in
                                                                applications, which includes using an automated tool to collect, rationalize,
                                                                and track the migration of its applications.
                                                                However, Army has not established a method and metrics to measure
                                                                outcomes and benefits for its Generating Force and Operating Force
                                                                segment architectures. Although officials reported that they are tracking
                                                                the status of architecture artifact development, artifacts are architecture
                                                                program outputs rather than outcomes resulting from the use of an
                                                                architecture.
                                                                Army officials reported that the department faces a challenge that directly
                                                                relates to the lack of a centralized enterprise architecture office that can
                                                                provide oversight and guidance for architecture activities. A regulation
                                                                intended to address this challenge, with measures for assessing whether
                                                                the Army architecture is meeting the department’s needs, has been
                                                                drafted but has not been approved.
Enterprise architecture outcomes and
benefits are periodically measured and
                                                    ○           Although the Network segment has established architecture method and
                                                                metrics related to data center consolidation, it has yet to measure and
reported to the agency’s enterprise                             report the outcomes and benefits. According to the Army’s performance
architecture executive committee.                               plan, an annual application reduction report will be provided to the DOD
                                                                CIO starting in fiscal year 2013. With regard to the Generating Force and
                                                                Operating Force segment architectures, the Army has yet to measure and
                                                                report enterprise architecture outcomes and benefits.
                                           Source: GAO analysis of agency-provided data.




                                           Page 36                                                GAO-12-791 Organizational Transformation
                                           Appendix II: Detailed Assessments of
                                           Individual Departments and Agencies against
                                           Relevant Elements of Our Enterprise
                                           Architecture Management Maturity Framework




Department of Commerce                     Table 6 shows the Department of Commerce’s satisfaction of relevant
                                           framework elements in version 2.0 of GAO’s EAMMF.

Table 6: Department of Commerce Satisfaction of EAMMF Elements

Element                                        Satisfied? Summary
The enterprise architecture’s intended
purpose or strategic goals are defined.
                                                    ●           Commerce defined goals for its enterprise architecture program in its
                                                                September 2010 Strategic Information Technology Plan for 2011-2015.
                                                                These include using the department’s enterprise architecture to continually
                                                                improve its business processes, align resources with Commerce’s top-
                                                                level strategic goals, and identify and support key IT management
                                                                decisions. According to the plan, Commerce plans to leverage its
                                                                architecture to reduce redundancy in its IT portfolio, combine capabilities,
                                                                utilize already-existing resources, and ensure that available IT resources
                                                                are documented and visible for all potential users.
                                                                According to the department’s Chief Enterprise Architect, the department
                                                                is reevaluating the goals and objectives of the enterprise architecture
                                                                program to make it more responsive to management requirements and to
                                                                place less emphasis on report and document generation. The department
                                                                established an enterprise architecture objective to adopt enterprise-wide
                                                                standards for enterprise architecture, purchasing, and cost savings in its
                                                                balanced scorecarda for the first quarter of fiscal year 2012.
A method and metrics have been established
to measure enterprise architecture strategic
                                                    ◐           Commerce has established metrics and associated targets to measure
                                                                achievement of the objective to adopt enterprise-wide standards for
mission value (outcomes and benefits).                          enterprise architecture, purchasing, and cost savings. These include the
                                                                number of IT product standards adopted (target is two) and IT cost
                                                                reduction (target is $50,000 in annual savings).
                                                                However, the department has yet to establish a methodology that provides
                                                                the steps to be followed to measure enterprise architecture strategic
                                                                mission value. According to the Chief Enterprise Architect, the department
                                                                has documented a methodology to be used to demonstrate potential cost
                                                                savings. However, officials did not provide supporting documentation.
                                                                According to the Chief Enterprise Architect, measuring and reporting
                                                                enterprise architecture outcomes is a challenge because it is difficult to
                                                                attribute outcomes directly to architecture since outcomes are achieved
                                                                through a larger process that includes strategic and capital planning.
Enterprise architecture outcomes and
benefits are periodically measured and
                                                    ○           According to the Chief Enterprise Architect, the department achieved
                                                                savings in the first quarter of 2012 by switching from a decentralized
reported to the agency’s enterprise                             approach to procuring computers, software, and computer services to a
architecture executive committee.                               single, department-wide vehicle. However, Commerce did not provide
                                                                documentation to support the measurement and reporting of these cost
                                                                savings.
                                           Source: GAO analysis of agency-provided data.
                                           a
                                            The balanced scorecard is a private-sector concept introduced by Robert Kaplan and David Norton
                                           in 1992 to assess organizational performance and is used by several government agencies. The
                                           balanced scorecard is a form of performance plan that is used to help measure performance, make
                                           improvements, and assess how well organizations are positioned to perform in the future.




                                           Page 37                                                GAO-12-791 Organizational Transformation
                                          Appendix II: Detailed Assessments of
                                          Individual Departments and Agencies against
                                          Relevant Elements of Our Enterprise
                                          Architecture Management Maturity Framework




Department of Defense–                    Table 7 shows the Department of Defense (DOD) Business Enterprise
Business Enterprise                       Architecture’s (BEA) satisfaction of relevant framework elements in
Architecture                              version 2.0 of GAO’s EAMMF.

Table 7: DOD Business Enterprise Architecture Satisfaction of EAMMF Elements

Element                                Satisfied? Summary
The enterprise architecture’s
intended purpose or strategic goals
                                          ●          According to DOD’s March 2012 BEA Overview and Summary Information, the
                                                     purpose of the architecture is to (1) serve as a blueprint for business
are defined.                                         transformation that helps to ensure that the right capabilities, resources, and
                                                     materiel are rapidly delivered to warfighters; (2) guide and constrain
                                                     implementation of interoperable defense business system solutions; (3) guide
                                                     information technology investments to align with strategic business capabilities;
                                                     and (4) support portfolio management during the investment review process.
                                                     DOD’s September 2011 Strategic Management Plan for fiscal years 2012-2013,
                                                     which establishes management goals for business operations, includes the goal
                                                     to reengineer/use end-to-end business processes to reduce transaction times,
                                                     drive down costs, and improve service. Associated with the goal is an initiative to
                                                     improve business operations through optimal use of defense business systems
                                                     and the BEA.
A method and metrics have been
established to measure enterprise
                                          ○          While the department has established metrics for measuring achievement of its
                                                     goal to improve business operations through optimal use of defense business
architecture strategic mission value                 systems and the BEA, the metrics do not measure BEA outcomes and benefits.
(outcomes and benefits).                             Specifically, DOD established metrics in its Strategic Management Plan for fiscal
                                                     years 2012-2013 which include percentage of defense business
                                                     systems/services represented in both the Defense Information Technology
                                                     Portfolio Repository (the department’s authoritative business systems inventory)
                                                     and the BEA, percentage of defense business systems/services represented in
                                                     both the Select and Native Programming Data Input System—Information
                                                     Technology (the department’s system used to prepare its budget submission)
                                                     and the BEA, and percentage of defense business systems/services reporting to
                                                     OMB through the BEA. However, these metrics measure output (i.e., direct
                                                     products and services), rather than outcomes (i.e., results of enterprise
                                                     architecture products and services such as benefits to Congress and the
                                                     American taxpayer) of the enterprise architecture program.
                                                     In addition, department officials stated that the department’s process to measure
                                                     and report architecture outcomes includes requiring components to submit
                                                     examples of business system improvement for inclusion in the department’s
                                                     annual report to Congress on Defense Business Operations. These are to be
                                                     substantiated with quantifiable measures that demonstrate desired business
                                                     outcomes and benefits. However, the guidance provided to program offices for
                                                     submitting these examples does not include the steps to be followed and metrics
                                                     for measuring BEA outcomes.
Enterprise architecture outcomes and
benefits are periodically measured
                                          ○          DOD has not periodically measured and reported enterprise architecture
                                                     outcomes and benefits. Specifically, while the March 2012 Congressional Report
and reported to the agency’s                         on Defense Business Systems describes enhancements to the BEA related to
enterprise architecture executive                    business process modeling and standardizing business data, and reports the
committee.                                           number of legacy systems that are not part of the target architecture (based on
                                                     DOD IT Portfolio Repository data), the report does not include any additional
                                                     examples.
                                          Source: GAO analysis of agency-provided data.




                                          Page 38                                             GAO-12-791 Organizational Transformation
                                           Appendix II: Detailed Assessments of
                                           Individual Departments and Agencies against
                                           Relevant Elements of Our Enterprise
                                           Architecture Management Maturity Framework




Department of Defense–                     Table 8 shows DOD’s Enterprise Architecture satisfaction of relevant
Enterprise Architecture                    framework elements in version 2.0 of GAO’s EAMMF.

Table 8: Department of Defense Satisfaction of EAMMF Elements

Element                                        Satisfied? Summary
The enterprise architecture’s intended
purpose or strategic goals are defined.
                                                    ●           DOD has defined goals for its DOD enterprise architecture. Specifically,
                                                                according to the department’s February 2009 Directive on Management of
                                                                the DOD Information Enterprise, the DOD enterprise architecture, which is
                                                                composed of DOD enterprise and component levels, is to be maintained
                                                                and applied to guide investment portfolio strategies and decisions, define
                                                                capability and interoperability requirements, establish and enforce
                                                                standards, guide security and information assurance requirements across
                                                                DOD, and provide a sound basis for transition from the existing
                                                                environment to the future.
A method and metrics have been established
to measure enterprise architecture strategic
                                                    ○           DOD has yet to establish a method and metrics for measuring DOD
                                                                enterprise architecture outcomes and benefits. According to officials,
mission value (outcomes and benefits).                          DOD’s approach to establishing a method and metrics for measuring DOD
                                                                enterprise architecture strategic mission value (outcomes and benefits) will
                                                                be accomplished through the development and publication of a DOD
                                                                instruction and an enterprise architecture management plan. In particular,
                                                                the draft instruction on enterprise architecture calls for establishing metrics
                                                                for assessing the effectiveness of the enterprise architecture to provide
                                                                information that contributes to mission effectiveness and efficiency. In
                                                                addition, the draft Enterprise Architecture Management Plan calls for the
                                                                development of metrics to assess the use of enterprise architecture,
                                                                provides examples of potential metrics, including reduction in
                                                                redundancies in DOD’s portfolio, and calls for the development of baseline
                                                                and target threshold values for each selected metric. The plan also states
                                                                that the DOD CIO and architecture organization are to determine the final
                                                                set of metrics and threshold values based on the resources available to
                                                                assess such metrics.
                                                                However, the department has not yet issued the instruction or the plan or
                                                                determined the specific method or final set of metrics to be used in
                                                                measuring enterprise architecture outcomes and benefits. According to
                                                                officials, the department expects the plan to be approved in December
                                                                2012 and the instruction to be approved in April 2013.
Enterprise architecture outcomes and
benefits are periodically measured and
                                                    ○           DOD has yet to measure and report DOD enterprise architecture
                                                                outcomes and benefits. According to DOD officials, the implementation of
reported to the agency’s enterprise                             the instruction on enterprise architecture and the enterprise architecture
architecture executive committee.                               management plan will allow the benefits of architecture to be measured
                                                                and reported. However, the department has not yet issued the instruction
                                                                or the plan.
                                           Source: GAO analysis of agency-provided data.




                                           Page 39                                                 GAO-12-791 Organizational Transformation
                                          Appendix II: Detailed Assessments of
                                          Individual Departments and Agencies against
                                          Relevant Elements of Our Enterprise
                                          Architecture Management Maturity Framework




Department of Education                   Table 9 shows the Department of Education’s satisfaction of relevant
                                          framework elements in version 2.0 of GAO’s EAMMF.

Table 9: Department of Education Satisfaction of GAO EAMMF Elements

Element                                   Satisfied? Summary
The enterprise architecture’s intended
purpose or strategic goals are defined.
                                              ●           Education has defined the purpose of its architecture program. Specifically,
                                                          according to its July 2011 Enterprise Transition Plan, enterprise architecture
                                                          provides for
                                                          • a priority-driven approach to planning and executing the activities needed
                                                             to transition from the baseline architecture to the target architecture,
                                                          • improved strategic decision-making and communication to achieve the
                                                             enterprise vision for technology at the department,
                                                          • increased control mechanisms for technology planning and investment,
                                                          • improved responsiveness to the enterprise technology needs of the
                                                             department’s business, and
                                                          • the ability to leverage technology to create a more effective and efficient
                                                             department.
A method and metrics have been
established to measure enterprise
                                              ◐           The department has established a metric to measure enterprise architecture
                                                          outcomes, but has yet to establish a method. Specifically, according to the
architecture strategic mission value                      October 2011 Office of the CIO Organization Performance Review report, a
(outcomes and benefits).                                  key performance indicator for the Office of the CIO is to increase
                                                          development, modernization, and enhancement (DME) spending through use
                                                          of its enterprise architecture segment modernization planning process. To that
                                                          end, the department has established a metric to measure the extent to which
                                                          the ratio of the increase in spending on DME is increased relative to steady-
                                                          state spending. According to the department’s June 2010 IT Portfolio
                                                          Analysis, increasing spending in DME leads to a decrease in spending to
                                                          maintain current systems and technologies (i.e., steady-state spending). The
                                                          department explained that its architecture program works with line-of-business
                                                          segment owners to develop modernization plans that include achieving
                                                          operational efficiencies.
                                                          In June 2012, the department finalized an IT Portfolio Management Value
                                                          Measurement methodology that describes the process for determining the
                                                          value of an investment relative to the department’s IT portfolio. The
                                                          information is to be used to set priorities for funding decisions or selecting
                                                          investments to be included in the department’s IT portfolio. However, the
                                                          process is not a method for measuring architecture strategic mission value.
Enterprise architecture outcomes and
benefits are periodically measured and
                                              ◐           In October 2011, the department reported the ratio of DME versus steady-
                                                          state spending increased as a result of its enterprise architecture activities.
reported to the agency’s enterprise                       Specifically, according to the department’s October 2011 Office of the CIO
architecture executive committee.                         Organization Performance Review report, DME funding in the IT Portfolio
                                                          increased from 10 percent of total IT spending in fiscal year 2011, to 13
                                                          percent of total IT spending in fiscal year 2012 through use of the
                                                          department’s architecture segment modernization planning process. However,
                                                          this metric has yet to be periodically measured and reported as an
                                                          architecture outcome.
                                          Source: GAO analysis of agency-provided data.




                                          Page 40                                              GAO-12-791 Organizational Transformation
                                           Appendix II: Detailed Assessments of
                                           Individual Departments and Agencies against
                                           Relevant Elements of Our Enterprise
                                           Architecture Management Maturity Framework




Department of Energy                       Table 10 shows the Department of Energy’s satisfaction of relevant
                                           framework elements in version 2.0 of GAO’s EAMMF.

Table 10: Department of Energy Satisfaction of EAMMF Elements

Element                                        Satisfied? Summary
The enterprise architecture’s intended
purpose or strategic goals are defined.
                                                    ●           Energy has defined its enterprise architecture goals, which include:
                                                                • Maintain alignment between technology solutions and department
                                                                  mission and goals.
                                                                • Provide enterprise architecture training and outreach opportunities,
                                                                  thereby promoting enterprise architecture value and transparency to
                                                                  support better business decisions department-wide.
                                                                • Work in conjunction with program and staff/support offices to further
                                                                  define, elaborate, and identify areas for additional development in line
                                                                  with the department’s mission.
                                                                • Continue to identify, reuse, and leverage, where possible, existing and
                                                                  planned technology and infrastructure components across the
                                                                  department.
                                                                • Identify areas, through capital planning and investment
                                                                  control/enterprise architecture integration analysis, to reduce costs,
                                                                  identify redundancy, and increase system and process effectiveness.
                                                                • Foster the organization and presentation of enterprise architecture to
                                                                  support decision making, program analysis, and efficient achievement
                                                                  of mission goals, utilizing an upgraded enterprise architecture data
                                                                  repository.
A method and metrics have been established
to measure enterprise architecture strategic
                                                    ○           Energy has not established metrics and a method for measuring
                                                                enterprise architecture strategic mission value. Specifically, although the
mission value (outcomes and benefits).                          department’s August 2012 Enterprise Modernization Roadmap includes
                                                                potential enterprise architecture program metrics (e.g., cost savings
                                                                through retiring legacy systems and cost avoidance by leveraging existing
                                                                solutions over procuring new ones through the use of enterprise
                                                                architecture), the metrics are still being defined and have yet to be
                                                                finalized and approved. Regarding a methodology, the roadmap states
                                                                that appropriate processes will be developed once the metrics are
                                                                developed and approved.
Enterprise architecture outcomes and
benefits are periodically measured and
                                                    ○           The department has yet to measure and report enterprise architecture
                                                                outcomes and benefits.
reported to the agency’s enterprise
architecture executive committee.
                                           Source: GAO analysis of agency-provided data.




                                           Page 41                                                GAO-12-791 Organizational Transformation
                                         Appendix II: Detailed Assessments of
                                         Individual Departments and Agencies against
                                         Relevant Elements of Our Enterprise
                                         Architecture Management Maturity Framework




Department of Health and                 Table 11 shows the Department of Health and Human Services’ (HHS)
Human Services                           satisfaction of relevant framework elements in version 2.0 of GAO’s
                                         EAMMF.

Table 11: Department of Health and Human Services Satisfaction of EAMMF Elements

Element                    Satisfied? Summary
The enterprise
architecture’s intended
                              ●     HHS has established the following enterprise architecture goals and objectives:
                                    • Enable improved mission and business outcomes by providing products to support sound
purpose or strategic                  decisions, business processes, and effective solutions; providing structured methods and
goals are defined.                    guidance; supporting the development of transformation plans for addressing business
                                      needs and priorities; enabling the optimized use of resources; and increasing
                                      interoperability and information sharing within HHS and between HHS and external
                                      stakeholders.
                                    • Provide a consolidated view of HHS’s enterprise by providing a consolidated view of HHS’s
                                      current and future business, information, and technologies; providing relevant, reliable, and
                                      timely information analytics capabilities to support sound business decisions; increasing the
                                      level of enterprise program integration and enterprise data sharing; facilitating the federated
                                      management and maintenance of enterprise architecture information through the use of a
                                      common framework; and facilitating the development of a consolidated view of information
                                      about systems and investments.
                                    • Strengthen the enterprise architecture program foundation by demonstrating the utility of
                                      enterprise architecture to support program and business needs and priorities, aligning the
                                      architecture program to HHS and federal enterprise business needs and priorities, and
                                      fostering effective enterprise architecture practices at the operating division level.
A method and metrics
have been established to
                              ●     HHS has established a method and metrics to measure enterprise architecture outcomes. The
                                    department developed an Enterprise Architecture Value Measurement Plan in December 2009
measure enterprise                  which includes measuring the extent to which the department increases the number of service
architecture strategic              components that are reused. Specifically, it includes measuring the extent to which the
mission value (outcomes             department increases the percentage of applicable service components in its service
and benefits).                      component reference model that are provided by one IT system and used by another. The
                                    Enterprise Architecture Value Measurement Plan also identifies a method for how the metrics
                                    are to be measured, including how they are to be calculated, the data sources to be used, and
                                    targets to be achieved.
                                    In addition, the Enterprise Architecture Value Measurement Plan includes measuring potential
                                    cost avoidance based on recommendations made by the enterprise architecture program,
                                    such as for business process reengineering; elimination of redundant IT systems and services;
                                    and consolidation and reuse of IT systems, services, and data. However, agency officials said
                                    they have yet to develop a methodology for measuring cost avoidance. The Chief Enterprise
                                    Architect stated that it is a challenge to capture cost information, which is important to
                                    establishing a baseline, because investments cut across a number of systems.
Enterprise architecture
outcomes and benefits
                              ◐     Enterprise architecture results were measured and reported to the Chief Information Officer in
                                    November 2010. Among other measures, the briefing reported that 16 percent of service
are periodically                    components were reused. However, the department did not measure the metric again and
measured and reported               therefore, does not know the extent to which it increased its reuse of service components.
to the agency’s                     According to the Chief Enterprise Architect, the department is in the process of establishing
enterprise architecture             new IT and enterprise architecture priorities and intends to establish a new enterprise
executive committee.                architecture results measurement and reporting approach by the end of fiscal year 2012.
                                         Source: GAO analysis of agency-provided data.




                                         Page 42                                           GAO-12-791 Organizational Transformation
                                           Appendix II: Detailed Assessments of
                                           Individual Departments and Agencies against
                                           Relevant Elements of Our Enterprise
                                           Architecture Management Maturity Framework




Department of Homeland                     Table 12 shows the Department of Homeland Security’s (DHS)
Security                                   satisfaction of relevant framework elements in version 2.0 of GAO’s
                                           EAMMF.

Table 12: Department of Homeland Security Satisfaction of EAMMF Elements

Element                                        Satisfied? Summary
The enterprise architecture’s intended
purpose or strategic goals are defined.
                                                    ●           The DHS Enterprise Architecture Strategic Plan for Fiscal Years 2012-
                                                                2016 identifies the vision, mission, goals, and objectives of the
                                                                department’s enterprise architecture:
                                                                • Vision: Improving mission and performance, optimizing resources, and
                                                                  unifying DHS.
                                                                • Mission: To optimize DHS resources and provide a framework for
                                                                  strategic improvement and investment decisions.
                                                                • Goals: Plan and execute segment architecture, enhance operating
                                                                  effectiveness, mature enterprise architecture practices, and strengthen
                                                                  enterprise architecture program management.
                                                                • Objectives: To achieve each of the four goals, the plan outlines five
                                                                  corresponding objectives, including establishing performance metrics to
                                                                  drive standardization and accountability and identifying cost savings
                                                                  and avoidance through efficient and effective use of resources.
A method and metrics have been established
to measure enterprise architecture strategic
                                                    ○           DHS has not established metrics and a method for measuring enterprise
                                                                architecture outcomes and benefits. The department identified examples
mission value (outcomes and benefits).                          of enterprise architecture benefits (e.g., streamlined processes, increased
                                                                tool reuse, cost avoidance, increased sharing, increased process
                                                                improvements, and increased information sharing) and categorized them
                                                                (direct user/customer benefits, operational/mission performance benefits,
                                                                financial benefits, strategic/political benefits, and non-user/public benefits),
                                                                and according to DHS officials, the examples and categories are being
                                                                used to define enterprise architecture metrics. In addition, the department
                                                                has developed a tool for documenting and reporting enterprise
                                                                architecture outcomes. However, it has not finalized metrics and a method
                                                                with detailed steps to ensure that outcomes are consistently and
                                                                repeatedly measured.
Enterprise architecture outcomes and
benefits are periodically measured and
                                                    ○           The department has yet to measure and report enterprise architecture
                                                                outcomes and benefits. DHS officials stated that they expect to report
reported to the agency’s enterprise                             architecture outcomes to a department executive body by October 1,
architecture executive committee.                               2012.
                                           Source: GAO analysis of agency-provided data.




                                           Page 43                                                  GAO-12-791 Organizational Transformation
                                           Appendix II: Detailed Assessments of
                                           Individual Departments and Agencies against
                                           Relevant Elements of Our Enterprise
                                           Architecture Management Maturity Framework




Department of Housing                      Table 13 shows the Department of Housing and Urban Development’s
and Urban Development                      (HUD) satisfaction of relevant framework elements in version 2.0 of
                                           GAO’s EAMMF.

Table 13: Department of Housing and Urban Development Satisfaction of EAMMF Elements

Element                                        Satisfied? Summary
The enterprise architecture’s intended
purpose or strategic goals are defined.
                                                  ●      According to agency documentation, the primary purpose of the enterprise
                                                         architecture is to capture the information required to effectively plan a
                                                         course for achieving HUD’s strategic vision and goals. It is to be one
                                                         element of interrelated planning activities that are to enable HUD
                                                         managers and staff to define a vision, develop strategies and plans for
                                                         achieving the vision, make resource decisions, implement strategies, and
                                                         evaluate performance.
                                                         HUD’s enterprise architecture goals are to improve the efficiency and
                                                         effectiveness of the department’s programs; simplify HUD’s IT
                                                         environment by promoting standards and sharing and reusing common
                                                         technologies; improve interoperability by establishing enterprise-wide
                                                         standards; and reduce system development and operation and
                                                         maintenance costs by eliminating duplicative investments, promoting
                                                         sharing of common services, and establishing department-wide standards.
                                                         The department also defined enterprise architecture goals in its draft 2011
                                                         Enterprise Architecture Value Measurement Plan, including the goal of
                                                         enabling the use of enterprise IT technologies for reuse and to reduce
                                                         infrastructure complexity. Associated with this goal are objectives to
                                                         leverage existing IT technology products to meet business and functional
                                                         requirements, standardize enterprise technologies where it is cost
                                                         effective, and decommission obsolete systems that are no longer in use.
A method and metrics have been established
to measure enterprise architecture strategic
                                                  ●      HUD has established a method and metrics to measure its enterprise
                                                         architecture outcomes and benefits. Specifically, the department’s fiscal
mission value (outcomes and benefits).                   year 2011 draft Enterprise Architecture Value Measurement Plan includes
                                                         measuring the extent to which the department had decreased the number
                                                         of technology products added to its Technical Reference Model (TRM) that
                                                         duplicate existing capabilities, the extent to which it had increased the
                                                         number of standardized enterprise technologies across the department
                                                         that replace legacy products and do not duplicate existing capabilities, and
                                                         the extent to which it had decreased the number of obsolete systems in its
                                                         IT inventory. The plan also included steps to measure results and
                                                         outcomes, including identifying appropriate sources, and determining
                                                         baseline, target, and actual value measures.




                                           Page 44                                         GAO-12-791 Organizational Transformation
                                         Appendix II: Detailed Assessments of
                                         Individual Departments and Agencies against
                                         Relevant Elements of Our Enterprise
                                         Architecture Management Maturity Framework




Element                                     Satisfied? Summary
Enterprise architecture outcomes and
benefits are periodically measured and
                                                  ◐           The department measured and reported enterprise architecture outcomes
                                                              for fiscal year 2011 to a department executive committee in August 2012.
reported to the agency’s enterprise                           The December 2011 Enterprise Architecture Value Measurement Report
architecture executive committee.                             highlights areas, based on measurements, where additional focus and
                                                              improvement are needed. For example, according to the report, 12
                                                              technology products were added to its TRM that duplicate existing
                                                              capabilities versus a target of 6; 0 enterprise-licensed technologies
                                                              replaced legacy products versus a target of 3; and 0 obsolete systems
                                                              were decommissioned versus a target of 13. According to officials, the
                                                              department had not been able to retire these systems because their
                                                              maintenance costs were included in fixed-price contracts that included
                                                              systems that were currently being used. However, the department has yet
                                                              to measure and report the metrics again, and therefore, does not know the
                                                              extent to which it met its targets. According to the department, an
                                                              architecture Value Measurement Plan will be issued annually and results
                                                              of the measures in the plan will be documented in an annual report for the
                                                              fiscal year.
                                         Source: GAO analysis of agency-provided data.




                                         Page 45                                               GAO-12-791 Organizational Transformation
                                           Appendix II: Detailed Assessments of
                                           Individual Departments and Agencies against
                                           Relevant Elements of Our Enterprise
                                           Architecture Management Maturity Framework




Department of the Interior                 Table 14 shows the Department of the Interior’s (DOI) satisfaction of
                                           relevant framework elements in version 2.0 of GAO’s EAMMF.

Table 14: Department of the Interior Satisfaction of EAMMF Elements

Element                                        Satisfied? Summary
The enterprise architecture’s intended
purpose or strategic goals are defined.
                                                    ●           The Department of the Interior established a purpose and goals for its
                                                                enterprise architecture program in 2009. Specifically, the purpose is to
                                                                develop, maintain, and oversee the implementation of an enterprise
                                                                architecture that helps the department achieve its strategic goals. Goals
                                                                for the enterprise architecture program are to (1) improve the
                                                                implementation of architectural plans and (2) increase the portion of the
                                                                enterprise architected. Department officials stated that an Enterprise
                                                                Modernization Roadmap with an updated enterprise architecture purpose
                                                                and goals is expected to be completed by September 2012.
A method and metrics have been established
to measure enterprise architecture strategic
                                                    ○           While the department has established metrics for measuring achievement
                                                                of its goals, the metrics do not measure enterprise architecture outcomes
mission value (outcomes and benefits).                          and benefits. According to the department’s 2009 Enterprise Architecture
                                                                Program Management Plan, the key performance measures for the
                                                                program are the percentage of segments with completed architectures and
                                                                the percentage of development/modernization/enhancement funding
                                                                associated with completed and in-progress segment architectures.
                                                                However, these are not measures of enterprise architecture program
                                                                outcomes, but rather, measures of enterprise architecture development
                                                                and implementation. Officials reported that an assessment of the
                                                                department’s enterprise architecture program was recently conducted and
                                                                a new enterprise architecture program management plan is being
                                                                developed; however, they have not established a time frame for when the
                                                                plan will be completed.
Enterprise architecture outcomes and
benefits are periodically measured and
                                                    ○           The department is not periodically measuring enterprise architecture
                                                                outcomes and benefits.
reported to the agency’s enterprise
architecture executive committee.
                                           Source: GAO analysis of agency-provided data.




                                           Page 46                                               GAO-12-791 Organizational Transformation
                                           Appendix II: Detailed Assessments of
                                           Individual Departments and Agencies against
                                           Relevant Elements of Our Enterprise
                                           Architecture Management Maturity Framework




Department of Justice                      Table 15 shows the Department of Justice’s satisfaction of relevant
                                           framework elements in version 2.0 of GAO’s EAMMF.

Table 15: Department of Justice Satisfaction of EAMMF Elements

Element                                        Satisfied? Summary
The enterprise architecture’s intended
purpose or strategic goals are defined.
                                                    ●           Justice has defined its architecture’s intended purpose. Specifically,
                                                                according to its IT Strategic Plan for 2010 through 2015, the Enterprise
                                                                Architecture Program Management Office is to review all IT investments to
                                                                identify enterprise solutions that address the needs of a core mission area
                                                                or multiple components. According to the strategy, enterprise solutions
                                                                help to eliminate redundant IT investments, increase information sharing,
                                                                and make use of shared infrastructure services, thus reducing the cost
                                                                and complexity of managing the department’s IT environment. Also,
                                                                according to the strategy, enterprise architecture analysis is to support
                                                                identifying redundant legacy programs to either retire or migrate to an
                                                                enterprise solution, thereby further reducing the complexity and the cost of
                                                                the IT environment.
A method and metrics have been established
to measure enterprise architecture strategic
                                                    ○           The department has not established a method or metrics to measure
                                                                enterprise architecture outcomes and benefits. According to the
mission value (outcomes and benefits).                          department’s IT strategy, the enterprise architecture program is to help
                                                                identify and eliminate redundant programs, thus reducing costs. However,
                                                                the department stated that it is difficult to associate these cost savings
                                                                specifically within the department-level enterprise architecture because a
                                                                number of factors and groups contribute to the results.
Enterprise architecture outcomes and
benefits are periodically measured and
                                                    ○           The department does not measure and report enterprise architecture
                                                                outcomes and benefits.
reported to the agency’s enterprise
architecture executive committee.
                                           Source: GAO analysis of agency-provided data.




                                           Page 47                                                GAO-12-791 Organizational Transformation
                                           Appendix II: Detailed Assessments of
                                           Individual Departments and Agencies against
                                           Relevant Elements of Our Enterprise
                                           Architecture Management Maturity Framework




Department of Labor                        Table 16 shows the Department of Labor’s satisfaction of relevant
                                           framework elements in version 2.0 of GAO’s EAMMF.

Table 16: Department of Labor Satisfaction of EAMMF Elements

Element                                        Satisfied? Summary
The enterprise architecture’s intended
purpose or strategic goals are defined.
                                                    ●           Labor has defined the purpose of its enterprise architecture program.
                                                                Specifically, according to its April 2011 Enterprise Transition Plan, the
                                                                department uses its enterprise architecture process with its capital
                                                                planning and investment management process to ensure that investments
                                                                support strategic goals and are not duplicative of existing business
                                                                solutions. Through use of this approach, according to the plan, the
                                                                department is able to identify duplicative resources/investments, gaps, and
                                                                opportunities for internal and external collaboration, resulting in
                                                                operational improvements and cost-effective solutions to business
                                                                requirements. In addition, the plan states that the department’s enterprise
                                                                architecture framework promotes interoperability and information sharing
                                                                and provides benefits such as
                                                                • enterprise target architecture definitions that support the department’s
                                                                   mission objectives and strategic business plans,
                                                                • identification of redundancy and consolidation opportunities, and
                                                                • realization of cost savings and cost avoidance through improved
                                                                   performance.
A method and metrics have been established
to measure enterprise architecture strategic
                                                    ○           The department has not established a method or metrics to measure its
                                                                enterprise architecture outcomes and benefits. According to the April 2011
mission value (outcomes and benefits).                          Enterprise Transition Plan, Labor will establish enterprise architecture
                                                                program metrics to evaluate outcomes of the use of enterprise architecture
                                                                in investment decision making. However, department officials reported that
                                                                they have general measures related to capital planning that they use
                                                                across the Office of the Chief Information Officer and they do not
                                                                associate the measures specifically with enterprise architecture because a
                                                                number of factors contribute to the results.
Enterprise architecture outcomes and
benefits are periodically measured and
                                                    ○           The department does not measure and report enterprise architecture
                                                                outcomes and benefits.
reported to the agency’s enterprise
architecture executive committee.
                                           Source: GAO analysis of agency-provided data.




                                           Page 48                                               GAO-12-791 Organizational Transformation
                                           Appendix II: Detailed Assessments of
                                           Individual Departments and Agencies against
                                           Relevant Elements of Our Enterprise
                                           Architecture Management Maturity Framework




Department of the Navy                     Table 17 shows the Department of the Navy’s (DON) satisfaction of
                                           relevant framework elements in version 2.0 of GAO’s EAMMF.

Table 17: Department of the Navy Satisfaction of EAMMF Elements

Element                                        Satisfied? Summary
The enterprise architecture’s intended
purpose or strategic goals are defined.
                                                    ●           DON has defined the purpose of its enterprise architecture program.
                                                                According to the program’s 2010 All-View document, which provides an
                                                                overview and summary information of the DON enterprise architecture, the
                                                                purpose is to:
                                                                • Guide the department’s investments towards achieving departmental
                                                                   goals and objectives.
                                                                • Assist DON program managers in the development of their “solution
                                                                   architectures”–as mandated by the Joint Capabilities Integration
                                                                   Development System and Acquisition processes.
                                                                More specifically, the enterprise architecture is to
                                                                • promote interoperability;
                                                                • delineate existing and future programs and projects;
                                                                • establish uniform and standard models for business processes and IT
                                                                   systems that are common across DON;
                                                                • document all aspects of the enterprise including the functional
                                                                   activities, business processes, information, participants, systems,
                                                                   applications, and supporting technology infrastructure;
                                                                • support oversight and governance of IT investments;
                                                                • enable and align business and IT investments through improved
                                                                   portfolio management, capital planning and investment control, and
                                                                   other acquisition and budgeting processes;
                                                                • enable decision makers to identify capability gaps and overlaps; and
                                                                • provide insight into Doctrine, Organization, Training, Materiel,
                                                                   Leadership, Personnel, and Facilities domains as they relate to the
                                                                   business, information, systems, applications, and information
                                                                   technology required for decision support at all levels of DON.
A method and metrics have been established
to measure enterprise architecture strategic
                                                    ○           DON has not established a method or metrics to measure enterprise
                                                                architecture outcomes and benefits. Officials stated that they anticipate
mission value (outcomes and benefits).                          establishing a method in the second half of 2013. Officials reported that a
                                                                lack of best practices for measuring enterprise architecture value
                                                                continues to inhibit their ability to demonstrate enterprise architecture
                                                                return on investment.
Enterprise architecture outcomes and
benefits are periodically measured and
                                                    ○           DON has yet to measure and report enterprise architecture outcomes and
                                                                benefits.
reported to the agency’s enterprise
architecture executive committee.
                                           Source: GAO analysis of agency-provided data.




                                           Page 49                                                GAO-12-791 Organizational Transformation
                                           Appendix II: Detailed Assessments of
                                           Individual Departments and Agencies against
                                           Relevant Elements of Our Enterprise
                                           Architecture Management Maturity Framework




Department of State                        Table 18 shows the Department of State’s satisfaction of relevant
                                           framework elements in version 2.0 of GAO’s EAMMF.

Table 18: Department of State Satisfaction of EAMMF Elements

Element                                        Satisfied? Summary
The enterprise architecture’s intended
purpose or strategic goals are defined.
                                                    ●           According to State’s IT Strategic and Tactical Plans for fiscal years 2011
                                                                to 2013, the purpose of the enterprise architecture is to focus on
                                                                interoperability and application services.
A method and metrics have been established
to measure enterprise architecture strategic
                                                    ○           The department has yet to establish a method or metrics for measuring
                                                                enterprise architecture outcomes and benefits. Agency officials stated that
mission value (outcomes and benefits).                          while their objective is to have good IT investments, it is difficult to
                                                                measure enterprise architecture’s contribution because IT investment
                                                                results are due to many factors, including good project management, and
                                                                adequate funding, as well as enterprise architecture. While, according to
                                                                the agency’s IT Tactical Plan, a key performance indicator for its
                                                                enterprise architecture program is evidence of increased use and value of
                                                                enterprise architecture products and services in providing consistent and
                                                                effective IT solutions, promoting interoperability, information sharing, and
                                                                collaboration, State has yet to establish metrics and a method for
                                                                measuring the value of its enterprise architecture products. Department
                                                                officials reported that they expect to create metrics by December 2012.
Enterprise architecture outcomes and
benefits are periodically measured and
                                                    ○           The department has yet to periodically measure and report enterprise
                                                                architecture outcomes and benefits.
reported to the agency’s enterprise
architecture executive committee.
                                           Source: GAO analysis of agency-provided data.




                                           Page 50                                                GAO-12-791 Organizational Transformation
                                           Appendix II: Detailed Assessments of
                                           Individual Departments and Agencies against
                                           Relevant Elements of Our Enterprise
                                           Architecture Management Maturity Framework




Department of                              Table 19 shows Department of Transportation’s satisfaction of relevant
Transportation                             framework elements in version 2.0 of GAO’s EAMMF.

Table 19: Department of Transportation Satisfaction of EAMMF Elements

Element                                        Satisfied? Summary
The enterprise architecture’s intended
purpose or strategic goals are defined.
                                                    ●           The department’s Information Resources Management Strategic Plan for
                                                                fiscal years 2007-2012 includes the goal to establish its enterprise
                                                                architecture as the authoritative decision tool for IT investments. According
                                                                to the plan, enterprise architecture is to be used as a decision-making tool
                                                                to support business plan development, identify areas of duplication and
                                                                inefficiencies in the department, and select top priorities for department-
                                                                wide implementation and management.
A method and metrics have been established
to measure enterprise architecture strategic
                                                    ◐           The department has established a metric but not a method to measure
                                                                enterprise architecture strategic mission value. Specifically, according to
mission value (outcomes and benefits).                          Transportation’s Information Resources Management Strategic Plan, an
                                                                expected enterprise architecture outcome is reduced total cost of
                                                                ownership, indicated by cost savings and/or cost avoidance identified
                                                                through review of business processes, data, applications, and technology,
                                                                and by the number/percentage of eliminated duplicative systems.
                                                                In addition, in December 2008, the department developed an enterprise
                                                                architecture Performance Measurement Guide, which included an
                                                                enterprise architecture performance objective to support investment
                                                                decisions for approved segment and solution architectures with an
                                                                outcome of solutions that foster transparency, increase mission
                                                                effectiveness, reduce redundancies, and minimize costs. However, a
                                                                method to measure the objective was not established. Specifically, the
                                                                guide included steps to finalize enterprise architecture performance
                                                                measures and indicators, such as conducting outreach with stakeholders
                                                                to disseminate information about performance measures and indicators,
                                                                and capturing baselines and defining targets. In addition, the guide
                                                                included a plan to collect and analyze data and to measure results on a
                                                                quarterly basis, starting the first quarter of calendar year 2009. However,
                                                                performance measures and indicators were never finalized and the guide
                                                                did not include the steps to be followed to collect and analyze the data.
                                                                According to the department’s Chief Architect, enterprise architecture was
                                                                a high priority in 2008 and 2009, but in mid-2010 the department shifted
                                                                priorities and limited resources, which has constrained enterprise
                                                                architecture efforts.
Enterprise architecture outcomes and
benefits are periodically measured and
                                                    ○           The department has not measured and reported enterprise architecture
                                                                outcomes and benefits. Though the department’s 2008 Enterprise
reported to the agency’s enterprise                             Architecture Performance Measurement Guide states that the department
architecture executive committee.                               planned to collect and analyze data and to measure and report results on
                                                                a quarterly basis starting the first quarter of calendar year 2009, it has yet
                                                                to do so. In addition, while the department reported in its response to our
                                                                survey on enterprise architecture results and outcomes that its
                                                                architecture program contributed to an estimated $83 million in cost
                                                                savings in fiscal year 2009, it did not provide documentation to support the
                                                                cost savings estimate or evidence that the outcome was reported to
                                                                agency executives.
                                           Source: GAO analysis of agency-provided data.




                                           Page 51                                                 GAO-12-791 Organizational Transformation
                                           Appendix II: Detailed Assessments of
                                           Individual Departments and Agencies against
                                           Relevant Elements of Our Enterprise
                                           Architecture Management Maturity Framework




Department of the                          Table 20 shows the Department of the Treasury’s satisfaction of relevant
Treasury                                   framework elements in version 2.0 of GAO’s EAMMF.

Table 20: Department of the Treasury Satisfaction of EAMMF Elements

Element                                        Satisfied? Summary
The enterprise architecture’s intended
purpose or strategic goals are defined.
                                                    ●           Treasury has defined its enterprise architecture goals. Specifically,
                                                                according to Treasury’s E-Government Act Report for fiscal year 2011, its
                                                                enterprise architecture plans have focused on reducing duplication
                                                                through its data center consolidation initiative because infrastructure
                                                                reflects the majority of the department’s IT spending. Accordingly,
                                                                Treasury has defined goals related to data center consolidation.
                                                                Specifically, its goals include increased cost efficiency through
                                                                consolidation of facilities and infrastructure, increased economies of scale
                                                                and associated buying power, and reduced overhead associated with
                                                                operating multiple instances of common facilities and services. According
                                                                to officials, the department is in the process of developing broader goals
                                                                that will integrate its capital planning and investment control and
                                                                architecture processes.
A method and metrics have been established
to measure enterprise architecture strategic
                                                    ◐           Treasury has established metrics for measuring enterprise architecture
                                                                outcomes and benefits but has yet to document a method. Specifically, the
mission value (outcomes and benefits).                          department plans to measure a decrease in the number of servers, an
                                                                increase in the percentage of operating systems that are virtual, and a
                                                                decrease in demand for data center square footage. According to the
                                                                department’s September 2011 Data Center Consolidation Plan, reductions
                                                                in physical assets should produce increases in capacity and cost
                                                                efficiencies for management of space and IT services. However, the
                                                                department has not established a methodology for measuring its
                                                                architecture outcomes with detailed steps to be followed (including
                                                                sources of information). Further, the Chief Enterprise Architect stated that
                                                                metrics corresponding to broader goals that integrate capital planning and
                                                                investment control and enterprise architecture are under development.
Enterprise architecture outcomes and
benefits are periodically measured and
                                                    ◐           The department has measured and reported architecture outcomes
                                                                associated with its data center consolidation. Specifically, its September
reported to the agency’s enterprise                             2011 Data Center Consolidation Plan, which was approved by the
architecture executive committee.                               department’s Deputy Assistant Secretary for Information Systems and
                                                                Chief Information Officer, reported a reduction in the number of servers,
                                                                an increase in the percentage of operating systems that were virtualized,
                                                                and a reduction in data center square footage between 2010 and 2011.
                                                                However, this metric has yet to be periodically measured and reported as
                                                                an architecture outcome. As noted above, going forward, Treasury is
                                                                developing new goals and metrics.
                                           Source: GAO analysis of agency-provided data.




                                           Page 52                                                GAO-12-791 Organizational Transformation
                                           Appendix II: Detailed Assessments of
                                           Individual Departments and Agencies against
                                           Relevant Elements of Our Enterprise
                                           Architecture Management Maturity Framework




Department of Veterans                     Table 21 shows the Department of Veterans Affairs’ (VA) satisfaction of
Affairs                                    relevant framework elements in version 2.0 of GAO’s EAMMF.

Table 21: Department of Veterans Affairs Satisfaction of EAMMF Elements

Element                                        Satisfied? Summary
The enterprise architecture’s intended
purpose or strategic goals are defined.
                                                    ◐           According to the Chief Architect, VA is in the process of developing an
                                                                enterprise architecture program overview statement and guiding enterprise
                                                                architecture principles. Specifically, according to draft documentation, VA’s
                                                                enterprise architecture is to provide tools, rules, principles, and standards
                                                                to guide efficient, effective, and interoperable implementation of the
                                                                department’s vision of providing seamless delivery of benefits and
                                                                services to veterans. Global Enterprise Architecture principles include that
                                                                all VA solutions are to utilize enterprise-wide standards, services, and
                                                                approaches to deliver seamless capabilities to veterans, facilitate IT
                                                                consolidations through reuse, and simplify the use of VA functions.
                                                                According to department officials, enterprise architecture principles have
                                                                been finalized, but are not planned to be formally released until September
                                                                30, 2012.
A method and metrics have been established
to measure enterprise architecture strategic
                                                    ○           According to the department’s Chief Architect, VA has not established a
                                                                method and metrics for measuring enterprise architecture outcomes and
mission value (outcomes and benefits).                          benefits because the department’s approach to enterprise architecture and
                                                                enterprise architecture governance is being revised. The Chief Architect
                                                                said that once the approach is updated, an enterprise architecture value
                                                                measurement plan will be developed. However, the official noted that it is
                                                                a challenge to know how much to attribute outcomes to enterprise
                                                                architecture relative to other factors in the decision-making process.
Enterprise architecture outcomes and
benefits are periodically measured and
                                                    ○           VA has not periodically measured and reported enterprise architecture
                                                                outcomes and benefits.
reported to the agency’s enterprise
architecture executive committee.
                                           Source: GAO analysis of agency-provided data.




                                           Page 53                                                GAO-12-791 Organizational Transformation
                                           Appendix II: Detailed Assessments of
                                           Individual Departments and Agencies against
                                           Relevant Elements of Our Enterprise
                                           Architecture Management Maturity Framework




Environmental Protection                   Table 22 shows the Environmental Protection Agency’s (EPA) satisfaction
Agency                                     of relevant framework elements in version 2.0 of GAO’s EAMMF.

Table 22: Environmental Protection Agency Satisfaction of EAMMF Elements

Element                                        Satisfied? Summary
The enterprise architecture’s intended
purpose or strategic goals are defined.
                                                    ●           EPA has defined its architecture goals and objectives. Specifically,
                                                                according to the agency’s February 2011 Modernization Blueprint, a major
                                                                goal is to use the architecture to identify segments within the organization
                                                                that could serve as candidates for service sharing and reuse. Additional
                                                                goals and objectives are described in the agency’s Office of Technology
                                                                Operation and Planning Mission Investments Solution Division April 2011
                                                                draft strategic plan for fiscal years 2011 to 2016. These include, among
                                                                other things, providing architecture services to enable stakeholders to
                                                                mature and increase value from their architectures, and developing a
                                                                common standard to be used in evaluating segments and solutions across
                                                                the enterprise.
A method and metrics have been established
to measure enterprise architecture strategic
                                                    ○           The agency has not established a method and metrics for measuring
                                                                enterprise architecture outcomes and benefits. Specifically, agency officials
mission value (outcomes and benefits).                          reported that the results from an enterprise architecture management
                                                                maturity self-assessment, completed in April 2012, will be analyzed to
                                                                determine areas for improvement and reflected in the agency’s enterprise
                                                                architecture performance measurement plan. According to agency officials,
                                                                the enterprise architecture program performance measurement plan is
                                                                expected to be completed in fiscal year 2013.
                                                                The agency has included metrics in its Office of Technology Operation and
                                                                Planning Mission Investments Solution Division draft strategic plan and
                                                                Enterprise Architecture Value Measures project charter approved in May
                                                                2012. These metrics include the percentage of segments and solution
                                                                architectures that have been reviewed by the enterprise architecture
                                                                program, the percentage of investments that identify future use of
                                                                enterprise services, and the percentage of complete mandatory data fields
                                                                in the agency’s enterprise architecture repository. However, these metrics
                                                                measure outputs (i.e., direct products and services) of the program rather
                                                                than outcomes (i.e., results of enterprise architecture products and
                                                                services such as benefits to Congress and the American taxpayer).
                                                                According to officials, the metrics to measure cost savings and efficiencies
                                                                from enterprise architecture will be identified in fiscal years 2013 and
                                                                2014. Officials also added that it has been challenging to measure
                                                                performance because baselines have changed from year to year due to
                                                                changing OMB reporting requirements.
Enterprise architecture outcomes and
benefits are periodically measured and
                                                    ○           The agency is not measuring and reporting architecture outcomes and
                                                                benefits.
reported to the agency’s enterprise
architecture executive committee.
                                           Source: GAO analysis of agency-provided data.




                                           Page 54                                                GAO-12-791 Organizational Transformation
                                           Appendix II: Detailed Assessments of
                                           Individual Departments and Agencies against
                                           Relevant Elements of Our Enterprise
                                           Architecture Management Maturity Framework




General Services                           Table 23 shows the General Services Administration’s (GSA) satisfaction
Administration                             of relevant framework elements in version 2.0 of GAO’s EAMMF.

Table 23: General Services Administration Satisfaction of EAMMF Elements

Element                                        Satisfied? Summary
The enterprise architecture’s intended
purpose or strategic goals are defined.
                                                    ●           According to GSA’s March 2011 briefing to OMB on its enterprise
                                                                architecture modernization plan, its enterprise architecture provides
                                                                services to increase interoperability between systems; increase reuse of
                                                                systems, information, and services; increase agility and flexibility in
                                                                building and operating systems; and facilitate achievement of agency
                                                                goals. Further, according to GSA’s March 2012 Enterprise Modernization
                                                                Roadmap, enterprise architecture is to, among other things, increase
                                                                system interoperability and cost efficiencies, reduce duplication, and
                                                                increase innovation.
A method and metrics have been established
to measure enterprise architecture strategic
                                                    ◐           GSA has established metrics but not a method for measuring enterprise
                                                                architecture outcomes and benefits. Specifically, GSA’s March 2012
mission value (outcomes and benefits).                          Enterprise Modernization Roadmap includes enterprise architecture
                                                                program work plans and corresponding output and outcome metrics. For
                                                                example, GSA has identified the percentage of applications complying
                                                                with IT standards as a metric for measuring the extent to which the agency
                                                                is increasing its use of IT standards, which is one of its desired outcomes.
                                                                The roadmap also includes a desired outcome of increasing development,
                                                                modernization, and enhancement spending by 25 percent per year
                                                                beginning in fiscal year 2014. According to the agency, the ratio of
                                                                development, modernization, and enhancement to steady-state spending
                                                                allows GSA’s enterprise architecture program to highlight the allocation of
                                                                IT spending and opportunities to reduce operating costs.
                                                                However, GSA has not established a method for measuring and reporting
                                                                enterprise architecture outcomes and benefits. According to GSA officials,
                                                                the benefits of architecture are achieved early in system development and
                                                                are difficult to relate to future return on investment measured later in a
                                                                system’s life cycle. Nonetheless, officials said they were beginning to
                                                                develop a method for measuring enterprise architecture outcomes, but did
                                                                not expect the plan to be completed for 2 to 5 years.
Enterprise architecture outcomes and
benefits are periodically measured and
                                                    ○           The agency has not measured and reported enterprise architecture
                                                                outcomes and benefits.
reported to the agency’s enterprise
architecture executive committee.
                                           Source: GAO analysis of agency-provided data.




                                           Page 55                                                GAO-12-791 Organizational Transformation
                                          Appendix II: Detailed Assessments of
                                          Individual Departments and Agencies against
                                          Relevant Elements of Our Enterprise
                                          Architecture Management Maturity Framework




National Aeronautics and                  Table 24 shows the National Aeronautics and Space Administration’s
Space Administration                      (NASA) satisfaction of relevant framework elements in version 2.0 of
                                          GAO’s EAMMF.

Table 24: National Aeronautics and Space Administration Satisfaction of EAMMF Elements

Element                  Satisfied? Summary
The enterprise
architecture’s
                            ●     NASA’s November 2011 Enterprise Architecture Policy identifies the purposes of the agency’s
                                  enterprise architecture, which include
intended purpose or               • being a composition of architectures and set of integrated reference models that map all IT
strategic goals are                  initiatives, capabilities, and services to agency needs;
defined.                          • serving to guide executive decision making, establishing a clear linkage between present
                                     capabilities and future NASA mission needs, including identifying potential shortfalls and
                                     redundancies in IT capabilities, the time frame in which the shortfall or redundancy exists, and
                                     an analysis of industry alternatives and remedial solutions/approaches;
                                  • providing a foundation for further development, modernization or modification, and
                                     enhancements of integrated architectures;
                                  • identifying mission IT dependencies;
                                  • being used as a tool to integrate strategic planning efforts and to select, guide, manage,
                                     rationalize, and prioritize agency investments;
                                  • establishing the framework for agency interoperability by providing the standard, rigorous
                                     construct for horizontal and vertical integration of mission needs and business processes
                                     through architecture;
                                  • being integral to the budget life cycle, enabling informed and timely procurement decision
                                     making to influence capital and strategic sourcing investments; and
                                  • promoting transparency and accountability by aligning functions/capabilities, services,
                                     systems, components, and related standards to agency strategy.
A method and metrics
have been established
                            ○     NASA has yet to establish a method or metrics for measuring enterprise architecture outcomes.
                                  However, according to NASA’s Chief Enterprise Architect, an approach for measuring enterprise
to measure enterprise             architecture performance is being developed. Specifically, draft enterprise architecture procedural
architecture strategic            requirements include metrics to measure the number of approved architecture artifacts. However,
mission value                     these metrics measure output (i.e., direct products and services) of the enterprise architecture
(outcomes and                     program, rather than outcomes (i.e., results of enterprise architecture products and services such
benefits).                        as benefits to Congress and the American taxpayer). Moreover, according to the Chief Architect,
                                  the agency does not yet have a mature enterprise architecture program and establishing one is a
                                  challenge because the agency’s IT environment is not structured in a way that readily accepts an
                                  enterprise-wide architecture. Specifically, the official stated that much of NASA’s funding is
                                  provided to the agency’s centers, which invest the money to meet their specific needs with little
                                  regard for the agency’s overall needs or existing capabilities. In July 2012, the NASA Chief
                                  Enterprise Architect stated that a NASA policy is expected to be issued by 2013, requiring that a
                                  method and metrics for measuring enterprise architecture value be established.
Enterprise
architecture outcomes
                            ○     NASA has yet to measure and report its architecture outcomes and benefits.

and benefits are
periodically measured
and reported to the
agency’s enterprise
architecture executive
committee.
                                          Source: GAO analysis of agency-provided data.




                                          Page 56                                          GAO-12-791 Organizational Transformation
                                           Appendix II: Detailed Assessments of
                                           Individual Departments and Agencies against
                                           Relevant Elements of Our Enterprise
                                           Architecture Management Maturity Framework




National Science                           Table 25 shows the National Science Foundation’s (NSF) satisfaction of
Foundation                                 relevant framework elements in version 2.0 of GAO’s EAMMF.

Table 25: National Science Foundation Satisfaction of EAMMF Elements

Element                                        Satisfied? Summary
The enterprise architecture’s intended
purpose or strategic goals are defined.
                                                    ●           NSF has identified enterprise architecture goals in its September 2008
                                                                Information Resource Management Plan. These goals are:
                                                                • Improve utilization of IT resources by eliminating duplicative
                                                                   investments, and promoting sharing of common services and
                                                                   standards.
                                                                • Improve program performance by ensuring business functions support
                                                                   strategic goals and priorities, data are optimized in support of the
                                                                   business, and applications and technology solutions are driven by
                                                                   business needs.
                                                                • Simplify IT investment decisions by providing a line of sight from
                                                                   strategy to business function to technology, which enables decision
                                                                   makers to select investments that support NSF’s core mission, and to
                                                                   identify duplicative or misaligned initiatives.
                                                                • Reduce IT diversity and complexity within NSF by promoting standards
                                                                   and the sharing and reuse of common technologies.
                                                                • Improve interoperability through the establishment of enterprise-wide
                                                                   standards that promote platform and vendor independence, enabling
                                                                   greater interoperability across disparate applications, both internal and
                                                                   external.
A method and metrics have been established
to measure enterprise architecture strategic
                                                    ○           NSF has yet to establish a method and metrics to measure enterprise
                                                                architecture strategic mission value. Specifically, the agency has
mission value (outcomes and benefits).                          established metrics for measuring the percent of IT investments that
                                                                comply with the agency’s transition strategy, the percent of IT projects that
                                                                comply with the agency’s Enterprise Architecture Modernization Roadmap,
                                                                the percent of IT services associated with an appropriate segment
                                                                architecture, and the percent of approved software and technical
                                                                architectures fulfilling opportunities to reuse shared services and IT
                                                                infrastructure (i.e., ensuring that solution architectures in development
                                                                reuse common infrastructure components or develop components for
                                                                future reuse where possible). However, these metrics measure output
                                                                (i.e., direct products and services) of the enterprise architecture program,
                                                                rather than outcomes (e.g., cost avoidance, improved mission
                                                                performance from reengineered business processes and modernizing
                                                                systems, or benefits to Congress and the American taxpayer).
                                                                NSF officials stated that the agency is exploring opportunities to mature its
                                                                process for measuring enterprise architecture outcomes, and plans to
                                                                revise its Enterprise Architecture Program Management Plan to align with
                                                                new OMB enterprise architecture guidance by the end of fiscal year 2012.
Enterprise architecture outcomes and
benefits are periodically measured and
                                                    ○           NSF has yet to measure and report enterprise architecture outcomes and
                                                                benefits.
reported to the agency’s enterprise
architecture executive committee.
                                           Source: GAO analysis of agency-provided data.




                                           Page 57                                                GAO-12-791 Organizational Transformation
                                           Appendix II: Detailed Assessments of
                                           Individual Departments and Agencies against
                                           Relevant Elements of Our Enterprise
                                           Architecture Management Maturity Framework




Nuclear Regulatory                         Table 26 shows the Nuclear Regulatory Commission’s (NRC) satisfaction
Commission                                 of relevant framework elements in version 2.0 of GAO’s EAMMF.

Table 26: Nuclear Regulatory Commission Satisfaction of EAMMF Elements

Element                                        Satisfied? Summary
The enterprise architecture’s intended
purpose or strategic goals are defined.
                                                    ●           NRC has defined the purpose and goals for its architecture. Specifically,
                                                                the purpose is to support IT goals that were established in its IT strategic
                                                                plan for fiscal years 2012 through 2016. These include:
                                                                • NRC staff and stakeholders can quickly and easily access the
                                                                   information they need.
                                                                • IT business solutions are easy to use, cost effective, and strengthen
                                                                   agency performance, which according to agency officials, is focused on
                                                                   avoiding duplication and cost savings.
                                                                • IT infrastructure is available, cost effective, and responsive to agency
                                                                   needs.
A method and metrics have been established
to measure enterprise architecture strategic
                                                    ◐           NRC has established a metric for measuring enterprise architecture
                                                                outcomes, but has yet to establish a method. Specifically, for the goal that
mission value (outcomes and benefits).                          the agency’s IT infrastructure is available, cost effective, and responsive to
                                                                agency business needs, NRC plans to measure progress toward having
                                                                common access controls by measuring the reduction in passwords and/or
                                                                sign-ons. However, a methodology with detailed steps for measuring
                                                                enterprise architecture outcomes has not yet been established.
Enterprise architecture outcomes and
benefits are periodically measured and
                                                    ○           NRC has not periodically measured and reported enterprise architecture
                                                                outcomes.
reported to the agency’s enterprise
architecture executive committee.
                                           Source: GAO analysis of agency-provided data.




                                           Page 58                                                 GAO-12-791 Organizational Transformation
                                          Appendix II: Detailed Assessments of
                                          Individual Departments and Agencies against
                                          Relevant Elements of Our Enterprise
                                          Architecture Management Maturity Framework




Office of Personnel                       Table 27 shows the Office of Personnel Management’s (OPM)
Management                                satisfaction of relevant framework elements in version 2.0 of GAO’s
                                          EAMMF.

Table 27: Office of Personnel Management Satisfaction of EAMMF Elements

Element                                      Satisfied? Summary
The enterprise architecture’s intended
purpose or strategic goals are defined.
                                                   ●           According to OPM’s IT Strategic Plan for 2010-2013, the agency’s
                                                               enterprise architecture defines IT management principals, goals, and
                                                               objectives and establishes a roadmap to achieve the enterprise
                                                               architecture vision of centralizing and managing OPM’s IT infrastructure
                                                               for the benefits and efficiencies that can be realized through technology.
                                                               The IT strategic plan also includes the objective to utilize the enterprise
                                                               architecture as a management and governance tool to strengthen decision
                                                               making and standard setting, coordinating with OPM business lines to
                                                               ensure technology decisions and implementations for new systems align
                                                               with the agency’s as well as the federal government’s enterprise
                                                               architecture.
 A method and metrics have been
established to measure enterprise
                                                   ◐           OPM has established cost savings as a metric to measure enterprise
                                                               architecture results and outcomes, and developed an Enterprise
architecture strategic mission value                           Architecture Return on Investment Framework. According to the
(outcomes and benefits).                                       framework, return on investment is calculated over a period of time and
                                                               relates the value contributed in dollars to the cost in dollars of the
                                                               enterprise achitecture program. The framework identifies steps the
                                                               enterprise architecture office plans to follow to determine the architecture’s
                                                               role in cost savings or improving mission, including defining enterprise
                                                               architecture’s role (strategic partner, collaborator, change agent etc.) in
                                                               improving business and IT and the percentage to attribute to enterprise
                                                               architecture for each role; collaboratively identifying the role enterprise
                                                               architecture is supposed to play before each major effort; evaluating the
                                                               actual role of enterprise architecture after each major effort; and soliciting
                                                               feedback on how well the role was performed. However, the framework
                                                               does not include steps to determine the cost savings or mission
                                                               improvement to which enterprise architecture contributes or to calculate in
                                                               dollars architecture’s return on investment. According to OPM officials,
                                                               one of the agency’s challenges in developing a method is quantifying
                                                               value based on the contributions of enterprise architecture to
                                                               business/mission improvement.
Enterprise architecture outcomes and
benefits are periodically measured and
                                                   ○           According to OPM officials, the agency has measured enterprise
                                                               architecture cost savings and reported them to the Chief Information
reported to the agency’s enterprise                            Officer. However, officials have not provided documentation to support
architecture executive committee.                              that the cost savings have been reliably measured.
                                          Source: GAO analysis of agency-provided data.




                                          Page 59                                                 GAO-12-791 Organizational Transformation
                                          Appendix II: Detailed Assessments of
                                          Individual Departments and Agencies against
                                          Relevant Elements of Our Enterprise
                                          Architecture Management Maturity Framework




Small Business                            Table 28 shows the Small Business Administration’s (SBA) satisfaction of
Administration                            relevant framework elements in version 2.0 of GAO’s EAMMF.

Table 28: Small Business Administration Satisfaction of EAMMF Elements

Element                                      Satisfied? Summary
The enterprise architecture’s intended
purpose or strategic goals are defined.
                                                   ●           According to SBA’s 2009 Capital Planning and Investment Control Policy
                                                               Guide, the agency’s enterprise architecture process is a management
                                                               practice that is to support strategic planning; capital planning and
                                                               investment control; system development; and IT asset management
                                                               activities to optimize the agency’s resources and achieve its performance
                                                               goals. Specifically, according to the guide, decision makers are to
                                                               leverage the agency’s architecture to help ensure that investments
                                                               •    support the business needs,
                                                               •    address specific and measurable performance gaps,
                                                               •    align with the agency’s mission and goals,
                                                               •    comply with the agency’s standards, and
                                                               •    reduce or eliminate spending on unneeded, redundant, and/or
                                                                    duplicative IT assets.
 A method and metrics have been
established to measure enterprise
                                                   ○           SBA has not established a method or metrics to measure enterprise
                                                               architecture outcomes and benefits. Nonetheless, according to agency
architecture strategic mission value                           officials, SBA measures the extent to which proposed IT investments align
(outcomes and benefits).                                       with the enterprise architecture during its capital planning and investment
                                                               control process. However, such a metric measures output (i.e., direct
                                                               products and services) of the program rather than outcomes (i.e., results
                                                               of enterprise architecture products and services such as benefits to
                                                               Congress and the American taxpayer) of the program. Agency officials
                                                               also stated that the architecture program maintained Financial Assistance
                                                               and Disaster Assistance segment architectures, which were used to
                                                               facilitate planning and decision making, and that the agency achieved
                                                               high- level performance goals in fiscal year 2011 associated with the
                                                               segments. However, the agency did not provide documentation showing
                                                               that a method and metrics had been established for measuring
                                                               architecture outcomes, or that the high-level performance outcomes were
                                                               linked to enterprise architecture.
Enterprise architecture outcomes and
benefits are periodically measured and
                                                   ○           SBA is not periodically measuring enterprise architecture outcomes and
                                                               benefits.
reported to the agency’s enterprise
architecture executive committee.
                                          Source: GAO analysis of agency-provided data.




                                          Page 60                                               GAO-12-791 Organizational Transformation
                                           Appendix II: Detailed Assessments of
                                           Individual Departments and Agencies against
                                           Relevant Elements of Our Enterprise
                                           Architecture Management Maturity Framework




Social Security                            Table 29 shows the Social Security Administration’s (SSA) satisfaction of
Administration                             relevant framework elements in version 2.0 of GAO’s EAMMF.

Table 29: Social Security Administration Satisfaction of EAMMF Elements

Element                             Satisfied?    Summary
The enterprise architecture’s
intended purpose or strategic
                                       ●          SSA has defined its enterprise architecture goals in its 2010 Enterprise Architecture
                                                  Program Plan. Specifically, the architecture is to provide visibility for IT initiatives
goals are defined.                                and support alignment with SSA’s strategic business plans; support design and
                                                  configuration management decisions and alignment of IT initiatives with SSA’s
                                                  infrastructure; and support decisions regarding operations, maintenance, and the
                                                  development of IT resources and services.
                                                  In addition, as we previously reported, SSA’s enterprise architecture for years 2011
                                                  through 2016 described a vision that includes eliminating existing stove-piped
                                                  application software, and reusing services to develop service-oriented architecture
                                                  applications to replace aging online and back-office desktop applications.a
                                                  According to the SSA’s Enterprise Architecture Transition Strategy, these efforts are
                                                  expected to help reduce costs and increase productivity.
A method and metrics have been
established to measure
                                       ○          SSA has yet to establish a method and metrics to measure enterprise architecture
                                                  outcomes. SSA has established a metric to measure the number of IT projects
enterprise architecture strategic                 compliant with agency architecture standards. In addition, SSA officials stated that
mission value (outcomes and                       they measure the percent of IT investments aligned to the agency’s strategic
benefits).                                        portfolios. However, these metrics measure outputs (i.e., direct products and
                                                  services) of the program, rather than outcomes (e.g., benefits to Congress and the
                                                  American taxpayer). SSA officials stated that they are considering developing
                                                  additional metrics to measure enterprise architecture value, for example, a metric to
                                                  measure the extent to which architecture helps identify opportunities to reuse
                                                  services and related software modules. However, they noted that the lack of
                                                  guidelines and best practices contribute to the difficulty in measuring outcomes. To
                                                  address this challenge, SSA officials said that they will participate in the Enterprise
                                                  Architecture Value Measurement workgroup of the federal CIO Council Strategy and
                                                  Planning Committee’s Architecture Subcommittee. They added that as new metrics
                                                  are identified they will be documented and communicated throughout the agency.
Enterprise architecture outcomes
and benefits are periodically
                                       ○          SSA has not periodically measured and reported enterprise architecture outcomes
                                                  and benefits.
measured and reported to the
agency’s enterprise architecture
executive committee.
                                           Source: GAO analysis of agency-provided data.
                                           a
                                           GAO, Social Security Administration: Improved Planning and Performance Measures are Needed to
                                           Help Ensure Successful Technology Modernization, GAO-12-495 (Washington, D.C.: Apr. 26, 2012).




                                           Page 61                                             GAO-12-791 Organizational Transformation
                                           Appendix II: Detailed Assessments of
                                           Individual Departments and Agencies against
                                           Relevant Elements of Our Enterprise
                                           Architecture Management Maturity Framework




U.S. Agency for                            Table 30 shows the U.S. Agency for International Development’s (USAID)
International Development                  satisfaction of relevant framework elements in version 2.0 of GAO’s
                                           EAMMF.

Table 30: U.S. Agency for International Development Satisfaction of EAMMF Elements

Element                             Satisfied? Summary
The enterprise architecture’s
intended purpose or strategic
                                       ●      USAID has developed architecture goals. Specifically, the agency’s December 2011
                                              Enterprise Architecture Program Charter identifies the following goals:
goals are defined.                            • Support improvement of mission-critical business processes through business
                                                process analysis, and identification and application of enterprise architecture
                                                standards.
                                              • Guide analytical efforts to locate, validate, and promote the strategic use of agency
                                                information.
                                              • Facilitate analysis of the agency’s IT environment, including IT hardware, software,
                                                and enterprise applications, to promote the effective and efficient deployment of IT
                                                services.
                                              • Provide governance for USAID technology efforts by designing and supporting the
                                                implementation of enterprise architecture models and standards.
A method and metrics have been
established to measure
                                       ●      USAID has established two metrics for measuring enterprise architecture outcomes:
                                              (1) cost savings and avoidance due to process efficiency, technology standardization,
enterprise architecture strategic             retirement, and consolidation; and (2) client satisfaction based on client survey
mission value (outcomes and                   responses on architecture’s value to business (such as facilitating decision making on
benefit).                                     technology and processes using enterprise architecture tools). The agency has also
                                              established guidance for measuring cost savings and avoidance and an approach to
                                              measure client satisfaction by developing and implementing surveys.
Enterprise architecture outcomes
and benefits are periodically
                                       ●      The agency has periodically measured enterprise architecture outcomes and,
                                              according to agency officials, these outcomes are reported to the Deputy CIO and CIO.
measured and reported to the                  In addition, the agency has established a website for CIO staff, including the Deputy
agency’s enterprise architecture              CIO and CIO, to review monthly architecture outcomes.
executive committee.                          According to its February 2012 Enterprise Architecture Performance Results report, the
                                              agency achieved $12.3 million in savings and $9.5 million in cost avoidance by
                                              transitioning disparate human resource systems to a human resource shared services
                                              center. The Federal Enterprise Architecture and the agency’s enterprise architecture
                                              were used to select a shared services center.
                                              In addition, the agency reported estimated savings of $15.7 million (not including $4
                                              million in migration costs) over the next 5 years, beginning in fiscal year 2013, by
                                              moving its e-mail service to a cloud-based service. According to agency officials, the
                                              cloud-based solution was recommended by the architecture team because it can
                                              replace multiple installations of the current e-mail solution. According to the agency’s
                                              return on investment analysis, this will reduce hardware and software maintenance,
                                              and labor and other expenses. However, as of September 2012, the new service had
                                              yet to be approved. Officials explained that delay in approval was causing a reduction
                                              in the return on investment and that the cost savings were being updated, accordingly.
                                           Source: GAO analysis of agency-provided data.




                                           Page 62                                          GAO-12-791 Organizational Transformation
Appendix III: Comments from the
              Appendix III: Comments from the Department
              of Labor



Department of Labor




              Page 63                                      GAO-12-791 Organizational Transformation
Appendix IV: Comments from the
             Appendix IV: Comments from the Department
             of the Treasury



Department of the Treasury




             Page 64                                     GAO-12-791 Organizational Transformation
Appendix V: Comments from the Department
             Appendix V: Comments from the Department
             of Agriculture



of Agriculture




             Page 65                                    GAO-12-791 Organizational Transformation
Appendix V: Comments from the Department
of Agriculture




Page 66                                    GAO-12-791 Organizational Transformation
Appendix VI: Comments from the
             Appendix VI: Comments from the Department
             of Commerce



Department of Commerce




             Page 67                                     GAO-12-791 Organizational Transformation
Appendix VII: Comments from the
             Appendix VII: Comments from the Department
             of Defense



Department of Defense




             Page 68                                      GAO-12-791 Organizational Transformation
Appendix VII: Comments from the Department
of Defense




Page 69                                      GAO-12-791 Organizational Transformation
Appendix VIII: Comments from the
             Appendix VIII: Comments from the Department
             of Education



Department of Education




             Page 70                                       GAO-12-791 Organizational Transformation
Appendix VIII: Comments from the Department
of Education




Page 71                                       GAO-12-791 Organizational Transformation
Appendix IX: Comments from the
             Appendix IX: Comments from the Department
             of Homeland Security



Department of Homeland Security




             Page 72                                     GAO-12-791 Organizational Transformation
Appendix IX: Comments from the Department
of Homeland Security




Page 73                                     GAO-12-791 Organizational Transformation
Appendix X: Comments from the Department
             Appendix X: Comments from the Department
             of the Interior



of the Interior




             Page 74                                    GAO-12-791 Organizational Transformation
Appendix XI: Comments from the
             Appendix XI: Comments from the Department
             of State



Department of State




             Page 75                                     GAO-12-791 Organizational Transformation
Appendix XI: Comments from the Department
of State




Page 76                                     GAO-12-791 Organizational Transformation
Appendix XI: Comments from the Department
of State




Page 77                                     GAO-12-791 Organizational Transformation
Appendix XI: Comments from the Department
of State




Page 78                                     GAO-12-791 Organizational Transformation
Appendix XII: Comments from the
             Appendix XII: Comments from the Department
             of Veterans Affairs



Department of Veterans Affairs




             Page 79                                      GAO-12-791 Organizational Transformation
Appendix XII: Comments from the Department
of Veterans Affairs




Page 80                                      GAO-12-791 Organizational Transformation
Appendix XII: Comments from the Department
of Veterans Affairs




Page 81                                      GAO-12-791 Organizational Transformation
Appendix XIII: Comments from the National
              Appendix XIII: Comments from the National
              Aeronautics and Space Administration



Aeronautics and Space Administration




              Page 82                                     GAO-12-791 Organizational Transformation
Appendix XIII: Comments from the National
Aeronautics and Space Administration




Page 83                                     GAO-12-791 Organizational Transformation
Appendix XIV: Comments from the Social
             Appendix XIV: Comments from the Social
             Security Administration



Security Administration




             Page 84                                  GAO-12-791 Organizational Transformation
Appendix XIV: Comments from the Social
Security Administration




Page 85                                  GAO-12-791 Organizational Transformation
Appendix XV: Comments from the
             Appendix XV: Comments from the
             Environmental Protection Agency



Environmental Protection Agency




             Page 86                           GAO-12-791 Organizational Transformation
Appendix XVI: Comments from the
             Appendix XVI: Comments from the Department
             of Health and Human Services



Department of Health and Human Services




             Page 87                                      GAO-12-791 Organizational Transformation
Appendix XVI: Comments from the Department
of Health and Human Services




Page 88                                      GAO-12-791 Organizational Transformation
Appendix XVII: Comments from the
             Appendix XVII: Comments from the
             Department of Energy



Department of Energy




             Page 89                            GAO-12-791 Organizational Transformation
Appendix XVII: Comments from the
Department of Energy




Page 90                            GAO-12-791 Organizational Transformation
Appendix XVII: Comments from the
Department of Energy




Page 91                            GAO-12-791 Organizational Transformation
Appendix XVII: Comments from the
Department of Energy




Page 92                            GAO-12-791 Organizational Transformation
Appendix XVIII: Comments from the
             Appendix XVIII: Comments from the
             Department of Housing and Urban
             Development


Department of Housing and Urban
Development




             Page 93                             GAO-12-791 Organizational Transformation
Appendix XVIII: Comments from the
Department of Housing and Urban
Development




Page 94                             GAO-12-791 Organizational Transformation
Appendix XVIII: Comments from the
Department of Housing and Urban
Development




Page 95                             GAO-12-791 Organizational Transformation
Appendix XIX: GAO Contact and Staff
                  Appendix XIX: GAO Contact and Staff
                  Acknowledgments



Acknowledgments

                  Valerie C. Melvin at (202) 512-6304 or melvinv@gao.gov
GAO Contact
                  In addition to the contact named above, Neelaxi Lakhmani (Assistant
Staff             Director), Mark Bird (Assistant Director), Virginia Chanley, Kelly Dodson,
Acknowledgments   Cheryl Dottermusch, Michael Holland, James Houtz, Catherine Hurley,
                  Stuart Kaufman, Lee McCracken, Tyler Mountjoy, Donald Sebers,
                  Jennifer Stavros-Turner, and Merry Woo made key contributions to this
                  report.




(310964)
                  Page 96                                 GAO-12-791 Organizational Transformation
GAO’s Mission         The Government Accountability Office, the audit, evaluation, and
                      investigative arm of Congress, exists to support Congress in meeting its
                      constitutional responsibilities and to help improve the performance and
                      accountability of the federal government for the American people. GAO
                      examines the use of public funds; evaluates federal programs and
                      policies; and provides analyses, recommendations, and other assistance
                      to help Congress make informed oversight, policy, and funding decisions.
                      GAO’s commitment to good government is reflected in its core values of
                      accountability, integrity, and reliability.

                      The fastest and easiest way to obtain copies of GAO documents at no
Obtaining Copies of   cost is through GAO’s website (http://www.gao.gov). Each weekday
GAO Reports and       afternoon, GAO posts on its website newly released reports, testimony,
                      and correspondence. To have GAO e-mail you a list of newly posted
Testimony             products, go to http://www.gao.gov and select “E-mail Updates.”

Order by Phone        The price of each GAO publication reflects GAO’s actual cost of
                      production and distribution and depends on the number of pages in the
                      publication and whether the publication is printed in color or black and
                      white. Pricing and ordering information is posted on GAO’s website,
                      http://www.gao.gov/ordering.htm.
                      Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
                      TDD (202) 512-2537.
                      Orders may be paid for using American Express, Discover Card,
                      MasterCard, Visa, check, or money order. Call for additional information.
                      Connect with GAO on Facebook, Flickr, Twitter, and YouTube.
Connect with GAO      Subscribe to our RSS Feeds or E-mail Updates. Listen to our Podcasts.
                      Visit GAO on the web at www.gao.gov.
                      Contact:
To Report Fraud,
Waste, and Abuse in   Website: http://www.gao.gov/fraudnet/fraudnet.htm
                      E-mail: fraudnet@gao.gov
Federal Programs      Automated answering system: (800) 424-5454 or (202) 512-7470

                      Katherine Siggerud, Managing Director, siggerudk@gao.gov, (202) 512-
Congressional         4400, U.S. Government Accountability Office, 441 G Street NW, Room
Relations             7125, Washington, DC 20548

                      Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800
Public Affairs        U.S. Government Accountability Office, 441 G Street NW, Room 7149
                      Washington, DC 20548




                        Please Print on Recycled Paper.