oversight

Information Technology: Executive office for U.S. Attorneys Needs to Institutionalize Key IT Management Disciplines

Published by the Government Accountability Office on 2003-07-25.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

             United States General Accounting Office

GAO          Report to Congressional Requesters




July 2003
             INFORMATION
             TECHNOLOGY
             Executive Office for
             U.S. Attorneys Needs
             to Institutionalize Key
             IT Management
             Disciplines




GAO-03-751
             a
                                                July 2003


                                                INFORMATION TECHNOLOGY

                                                Executive Office for U.S. Attorneys Needs
Highlights of GAO-03-751, a report to           to Institutionalize Key IT Management
congressional requesters
                                                Disciplines



The Executive Office for United                 To varying degrees, EOUSA has partially defined and implemented certain IT
States Attorneys (EOUSA) of the                 management disciplines that are critical to successfully achieving the Justice
Department of Justice is                        Department’s strategic goal of improving the integrity, security, and
responsible for managing                        efficiency of its IT systems. However, it has yet to institutionalize any of
information technology (IT)                     these disciplines, meaning that it has not defined existing policies and
resources for the United States
Attorneys’ Offices. GAO was asked
                                                procedures in accordance with relevant guidance, and it has yet to fully
to determine the extent to which                implement what it has defined. In particular, while EOUSA has developed an
EOUSA has institutionalized key IT              enterprise architecture—a blueprint for guiding operational and
management capabilities that are                technological change—the architecture was not developed in accordance
critical to achieving Justice’s                 with certain best practices. In addition, while the office has implemented
strategic goal of improving the                 certain process controls for selecting, controlling, and evaluating its IT
integrity, security, and efficiency of          investments, it has not yet implemented others that are necessary in order to
its IT systems.                                 develop an effective foundation for investment management. Further, it has
                                                not implemented important management practices that are associated with
                                                an effective security program. In contrast, it has defined—and is
To strengthen EOUSA’s IT                        implementing on a major system that we reviewed—most, but not all, of the
management capacity and to                      management practices associated with effective systems acquisition.
increase its chances of effectively
leveraging IT to improve its                    Institutionalization of these IT management disciplines has not been an
mission performance, GAO                        agency priority and is not being guided by plans of action or sufficient
recommends that the Attorney                    resources. Until each discipline is given the priority it deserves, EOUSA will
General direct the Director of                  not have the IT management capabilities it needs to effectively achieve the
EOUSA to (1) designate                          department’s strategic goal of improving the integrity, security, and
institutionalization of each of the             efficiency of its IT systems.
IT management disciplines as
priorities and (2) develop and
implement action plans in each of               EOUSA’s Institutionalization of Four Key IT Management Disciplines
the four IT disciplines to address               Management         Fully                Comment
                                                 discipline         Institutionalized?
the weaknesses that are identified
                                                 Enterprise         No                   Has an approved enterprise architecture but, for
in this report. EOUSA agreed with                architecture                            example, has yet to develop a policy for maintaining it.
the majority of GAO’s findings and               management
recommendations, and stated that                 IT investment      No                   Has several basic elements of proper IT investment
it will address most of the                      management                              management but has not yet, for example, used its
recommendations. It also stated                                                          defined investment selection process.
                                                 Information        No                   Is not fully satisfying any of the tenets of effective
that it has made notable progress in             security                                security, such as monitoring the effectiveness of
institutionalizing the IT                        management                              security controls and promoting security awareness.
management disciplines,                          System acquisition No                   Is successfully employing most of the practices
particularly information security,               management                              associated with effective software acquisition
and that each is currently an office                                                     management on one key project but does not have, for
                                                                                         example, a policy for software acquisition planning.
priority.
                                                Source: GAO.




www.gao.gov/cgi-bin/getrpt?GAO-03-751.

To view the full product, including the scope
and methodology, click on the link above.
For more information, contact Randolph C.
Hite at 202-512-3439 or hiter@gao.gov.
Contents



Letter                                                                                             1
                             Results in Brief                                                      2
                             Background                                                            3
                             EOUSA Has Yet to Institutionalize Key IT Management
                               Disciplines                                                         6
                             Conclusions                                                          23
                             Recommendations                                                      23
                             Agency Comments and Our Evaluation                                   25


Appendixes
              Appendix I:    Objective, Scope, and Methodology                                    29
             Appendix II:    Assessment of ECMS Acquisition Practices against Level 2 of
                             SEI’s Software Acquisition Capability Maturity Model                 32
             Appendix III:   Comments from the Department of Justice                              44
                             GAO Comments                                                         50
             Appendix IV:    GAO Contact and Staff Acknowledgments                                54
                             GAO Contact                                                          54
                             Acknowledgments                                                      54


Tables                       Table 1: Summary of Version 1.0 of GAO’s EA Management Maturity
                                       Framework Stages                                            9
                             Table 2: Assessment of EOUSA’s EA Efforts against GAO’s EA
                                       Maturity Framework                                         10
                             Table 3: The Five Stages of GAO’s ITIM Maturity Framework            12
                             Table 4: Assessment of EOUSA’s ITIM Efforts against Stage 2 of
                                       GAO’s ITIM Maturity Framework                              13
                             Table 5: SA-CMM Levels and Descriptions                              21
                             Table 6: Assessment of ECMS Acquisition against SEI’s SA-CMM
                                       Level 2 Key Process Area                                   22
                             Table 7: Software Acquisition Planning                               32
                             Table 8: Solicitation                                                34
                             Table 9: Requirements Development and Management                     36
                             Table 10: Project Management                                         38
                             Table 11: Contract Tracking and Oversight                            40
                             Table 12: Evaluation                                                 42


Figures                      Figure 1: Simplified Diagram of EOUSA’s Network Connections           4
                             Figure 2: Estimated IT Expenditures for Fiscal Year 2003              6



                             Page i                                    GAO-03-751 EOUSA IT Management
Contents




Abbreviations

CIO          chief information officer
EA           enterprise architecture
ECMS         Enterprise Case Management System
EOUSA        Executive Office for United States Attorneys
GWAC         Governmentwide Acquisition Contracts
IRB          Investment Review Board
IT           information technology
ITIM         IT investment management
JCN          Justice Consolidated Network
LIONS        Legal Information Office Network System
SA-CMM       Software Acquisition Capability Maturity Model
SEI          Software Engineering Institute
USAO         United States Attorneys’ Office
VANITS       Value Added Niche Information Technology Service
VPN          virtual private network
WAN          wide-area network

 This is a work of the U.S. government and is not subject to copyright protection in the
 United States. It may be reproduced and distributed in its entirety without further
 permission from GAO. However, because this work may contain copyrighted images or
 other material, permission from the copyright holder may be necessary if you wish to
 reproduce this material separately.




Page ii                                                GAO-03-751 EOUSA IT Management
A
United States General Accounting Office
Washington, D.C. 20548



                                    July 25, 2003                                                                            Leter




                                    The Honorable F. James Sensenbrenner, Jr.
                                    Chairman
                                    The Honorable John Conyers, Jr.
                                    Ranking Minority Member
                                    Committee on the Judiciary
                                    House of Representatives

                                    The Honorable Christopher Cannon
                                    Chairman
                                    The Honorable Melvin L. Watt
                                    Ranking Minority Member
                                    Subcommittee on Commercial and Administrative Law
                                    House of Representatives

                                    This report is one of a series in response to your request that we evaluate
                                    the management activities of the Executive Office for United States
                                    Attorneys (EOUSA) and the U.S. Attorneys’ Offices (USAO). As part of this
                                    request, you asked us to determine the extent to which EOUSA—the
                                    organization responsible for managing information technology (IT)
                                    resources for the USAOs—has institutionalized key IT management
                                    capabilities critical to achieving the Department of Justice’s strategic goal
                                    of improving the integrity, security, and efficiency of its IT systems. As
                                    agreed, to meet this objective, we focused on whether EOUSA had
                                    institutionalized four important IT management disciplines: enterprise
                                    architecture management, investment management, system acquisition
                                    management, and security management. Research shows that these
                                    disciplines are institutionally employed by leading public and private sector
                                    organizations. They are also provided for in legislation and federal
                                    guidance.1

                                    Details on our objective, scope, and methodology are in appendix I.




                                    1
                                     See, for example, Clinger-Cohen Act of 1996, Public Law 104-106; Office of Management
                                    and Budget, Management of Federal Information Resources, Circular No. A-130 (February
                                    1996); Office of Management and Budget, Funding Information Systems Investments,
                                    Memorandum M-97-02; and National Institute of Standards and Technology, Generally
                                    Accepted Principles and Practices for Securing Information Technology Systems, NIST SP
                                    800-14 (September 1996).




                                    Page 1                                              GAO-03-751 EOUSA IT Management
Results in Brief   To varying degrees, EOUSA has partially defined and implemented each of
                   four key IT management disciplines that are critical to successfully
                   achieving the Department of Justice’s strategic goal of improving the
                   integrity, security, and efficiency of its IT systems. However, it has yet to
                   institutionalize any of these disciplines—meaning that it has not defined its
                   policies and procedures in accordance with relevant guidance—and it has
                   yet to fully implement what it has defined. In particular, while EOUSA has
                   an enterprise architecture—a blueprint for guiding operational and
                   technological change—it has not developed the architecture in accordance
                   with certain best practices. In addition, while the office has implemented
                   certain process controls for selecting, controlling, and evaluating its IT
                   investments, it has not yet implemented others that are necessary in order
                   to develop an effective foundation for investment management. Further, it
                   has not implemented important management practices that are associated
                   with an effective security program, such as implementing and monitoring
                   security policies and controls and promoting security awareness. To its
                   credit, the office has defined—and is implementing on a major system—
                   most, though not all, of the management practices associated with effective
                   systems acquisition.

                   The institutionalization of these IT management disciplines has not been a
                   sufficiently high priority for EOUSA, as evidenced by the absence of plans
                   for fully implementing best practices for each discipline and, in some cases,
                   an absence of requisite resources. Until each discipline is given the priority
                   it deserves, EOUSA will not have the IT management capabilities that are
                   critical to effectively achieving the Justice Department’s strategic goal of
                   improving the integrity, security, and efficiency of its IT systems. We are
                   making recommendations to the Attorney General to strengthen
                   management of each of these disciplines.

                   In its written comments on a draft of this report, EOUSA agreed with our
                   specific findings for three of the four IT management disciplines, and
                   stated that it would address the weaknesses that we identified in each.
                   However, it stated that its security program is strong, citing a number of
                   security initiatives, including ones recently planned or started that are
                   consistent with our recommendations for addressing security weaknesses.
                   We do not question the initiatives that EOUSA cited. However, our analysis
                   of its security program, including these initiatives, identified serious
                   security weaknesses, and as a result, we do not agree that EOUSA’s security
                   program is strong. Further, it stated that institutionalization of each of the
                   IT disciplines is currently an office priority and that the state of its IT



                   Page 2                                         GAO-03-751 EOUSA IT Management
             management capabilities is not an impairment to achieving departmental
             strategic goals. However, it did not dispute either of our two reasons for
             concluding otherwise; namely, that it did not have a plan for fully
             implementing best practices for each discipline, and it had not allocated
             adequate resources to support such a plan. EOUSA’s comments are
             summarized and evaluated in the Agency Comments and Our Evaluation
             section of this report.



Background   U.S. Attorneys prosecute criminal cases brought forward by the federal
             government, prosecute and defend civil cases in which the United States is
             a party, and collect debts owed to the federal government that are
             administratively uncollectible. EOUSA was established in 1953 as a
             component of the Department of Justice to, among other things, provide
             general executive assistance and administrative and operational support to
             the 93 USAOs located throughout the 50 states, the District of Columbia,
             Guam, the Marianas Islands, Puerto Rico, and the U. S. Virgin Islands2 and
             to coordinate with other Department of Justice organizational units and
             other federal agencies on behalf of the U.S. Attorneys. One of EOUSA’s key
             responsibilities is managing the USAOs’ IT resources, including preparing
             their annual IT budget submissions and supporting their acquisition and
             maintenance of IT assets.

             IT plays an important role in helping the USAOs meet their mission
             objectives and, according to EOUSA planning documents, the USAOs’
             reliance on IT is to increase in response to expected growth in the number
             and complexity of their cases. Currently, EOUSA manages an IT
             environment consisting of central and distributed computing and
             communication resources in Washington, D.C., and 93 USAOs, respectively.
             Connectivity among these offices, Justice headquarters, and Justice’s Data
             Center in Rockville, MD,3 is through a virtual private network (VPN) 4
             connection on the Justice Consolidated Network (JCN), with such security




             2
             U.S. Attorneys’ Offices and their branches comprise over 240 sites.
             3
              Access to the Internet and to research services is provided through Justice’s Rockville, MD,
             Data Center.
             4
              A virtual private network uses a public or shared telecommunication infrastructure to
             provide remote users with secure access to an organization's network.




             Page 3                                                   GAO-03-751 EOUSA IT Management
safeguards as firewalls5 between USAO local area networks and JCN. The
VPN/firewall combination, which provides the foundation for secure
communications between EOUSA and the sites mentioned above, is
currently being replaced. Figure 1 generally depicts EOUSA’s network
topology. The USAOs’ support is also provided by such application systems
as the Legal Information Office Network System, which is a case
management system that compiles, maintains, and tracks information
about defendants, crimes, criminal charges, court events, and witnesses,
and the Victim Notification System, which notifies crime victims of the
status of their cases and assists with checking compliance with regulations
and policies concerning victim notification.



Figure 1: Simplified Diagram of EOUSA’s Network Connections




5
 Network firewalls are devices or systems that control the flow of traffic between networks
with different security requirements. Organizations employ firewalls to prevent
unauthorized access to their respective systems and resources.




Page 4                                                 GAO-03-751 EOUSA IT Management
Recognizing the importance of IT to achieving the USAOs’ mission, EOUSA
appointed a Chief Information Officer (CIO) in May 2001 and assigned the
CIO accountability and responsibility for managing central and distributed
IT resources and services, including

• managing the IT budget for the office and all of the USAOs;

• developing and acquiring new systems, including case management
  systems, and providing support for existing systems;

• managing network, telephone, and video communications; and

• securing IT assets (data, applications, and supporting networks).

In fiscal year 2003, EOUSA reports that it plans to spend approximately
$125 million on about 20 initiatives. Roughly $110 million of this amount is
to be spent on IT infrastructure and office automation projects (e.g.,
telecommunications programs). The remainder is to be spent on acquiring
mission support systems (e.g., Enterprise Case Management System
(ECMS),6 Victim Notification System) and maintaining existing ones.
Figure 2 shows the breakdown of estimated expenditures for fiscal year
2003.




6
 ECMS is intended to replace the Legal Information Office Network System (LIONS) and a
system for reporting data stored in LIONS and to be the enterprise solution for managing
and tracking case workload within the USAOs.




Page 5                                                GAO-03-751 EOUSA IT Management
                          Figure 2: Estimated IT Expenditures for Fiscal Year 2003




EOUSA Has Yet to          Research into the IT management practices that are employed by leading
                          public- and private-sector organizations has identified key institutional IT
Institutionalize Key IT   management disciplines that are interrelated and critical to ensuring,
Management                among other things, the integrity, security, and efficiency of IT systems.
                          These disciplines are also addressed in legislation and federal guidance.7
Disciplines



                          7
                           See, for example, Clinger-Cohen Act of 1996, Public Law 104-106; Office of Management
                          and Budget, Management of Federal Information Resources, Circular No. A-130 (February
                          1996); Office of Management and Budget, Funding Information Systems Investments,
                          Memorandum M-97-02; and National Institute of Standards and Technology, Generally
                          Accepted Principles and Practices for Securing Information Technology Systems, NIST SP
                          800-14 (September 1996).




                          Page 6                                              GAO-03-751 EOUSA IT Management
They include

1. enterprise architecture management, which involves defining,
   maintaining, and implementing an institutional blueprint that defines
   both the business and the supporting technology of the organization’s
   current and target operating environments and a roadmap to achieve
   the target environment;

2. IT investment management, which involves selecting, controlling, and
   evaluating a portfolio of investments within the context of an
   enterprise architecture;

3. IT security management, which involves protecting the integrity,
   confidentiality, and availability of an organization’s IT assets (e.g., data,
   application systems, and networks) and reducing the risks of
   tampering, unauthorized intrusions and disclosures, and disruption of
   operations.

4. system acquisition management, which involves managing selected
   investments (system projects) in a manner that increases the
   probability of promised system capabilities being delivered on time and
   within budget.

As we have previously reported,8 to successfully institutionalize these
disciplines, organizations should develop integrated plans to guide their
efforts that (1) specify measurable goals, objectives, and milestones;
(2) specify needed resources; and (3) assign clear responsibility and
accountability for accomplishing well-defined tasks. In addition, these
plans should be approved by senior management. In implementing these
plans, it is important that organizations allocate adequate resources and
measure and report progress against planned commitments and that
appropriate corrective actions be taken to address deviations.

EOUSA has defined and implemented each of the four IT management
disciplines mentioned above to some degree. However, none has been
institutionalized, meaning that they are not fully defined in accordance


8
 U.S. General Accounting Office, Information Technology: INS Needs to Better Manage the
Development of Its Enterprise Architecture, GAO/AIMD-00-212 (Washington, D.C.: August
2000); Information Technology: DLA Needs to Strengthen Its Investment Management
Capability, GAO-02-314 (Washington, D.C.: March 2002).




Page 7                                               GAO-03-751 EOUSA IT Management
                            with best practices and what has been defined has not been fully
                            implemented. While these disciplines have been given attention since the
                            recent appointment of the CIO, they have not been treated as priorities, in
                            that action plans needed for successful institutionalization have not been
                            developed or resourced. As a result, EOUSA is currently limited in its
                            ability to meet Justice’s strategic goal of improving its IT systems, and the
                            USAOs will be challenged in their ability to effectively and efficiently meet
                            their mission goals and priorities.



EOUSA Is Not Performing     An enterprise architecture (EA) is an investment blueprint that defines,
Important Practices         both in logical terms (including business functions and applications, work
                            locations, information needs and users, and the interrelationships among
Associated with Effective
                            these variables) and in technical terms (including hardware, software, data
Enterprise Architecture     communications, and security) how an organization operates today (“as
Management                  is”), how it intends to operate tomorrow (“to be”), and a roadmap for
                            transitioning from today to tomorrow. The development, maintenance, and
                            implementation of architectures are recognized hallmarks of successful
                            public and private organizations. According to a guide published by the
                            federal CIO Council,9 effective architecture management consists of a
                            number of core elements.

                            In February 2002, we published version 1.0 of our EA management maturity
                            framework, which arranges the core elements of the CIO Council’s guide
                            into five hierarchical stages.10 The framework provides an explicit
                            benchmark for gauging the effectiveness of architecture management and
                            provides a roadmap for making improvements. Table 1 summarizes the
                            framework’s five stages of maturity.




                            9
                             Chief Information Officers Council, A Practical Guide to Federal Enterprise Architecture,
                            Version 1.0 (February 2001).
                            10
                             U.S. General Accounting Office, Information Technology: Enterprise Architecture Use
                            across the Federal Government Can Be Improved, GAO-02-6 (Washington, D.C.: February
                            2002). We issued version 1.1 of the framework in April 2003—A Framework for Assessing
                            and Improving Enterprise Architecture Management, GAO-03-584G—but did not use it for
                            our evaluation because we had already completed our audit work evaluating EOUSA against
                            the initial framework.




                            Page 8                                                 GAO-03-751 EOUSA IT Management
Table 1: Summary of Version 1.0 of GAO’s EA Management Maturity Framework
Stages

Stage                           Description
Stage 5: Leveraging the EA to   This stage entails, among other things, incorporating the
manage change                   EA into corporate decision making to (1) avoid
(includes all elements in       unwarranted overlap across investments, (2) enable
stage 4)                        maximum systems interoperability, and (3) ensure
                                selection and funding of IT investments with manageable
                                risks and returns.
Stage 4: Completing the EA      This stage is characterized by having EA products that
products                        have been approved by the EA steering committee
(includes all elements in       (established in stage 2) or an investment review board,
stage 3)                        and by the CIO.
Stage 3: Developing the EA      This stage focuses on developing EA products according
products                        to the selected framework, methodology, tool, and
(includes all elements in       established management plans.
stage 2)
Stage 2: Building the EA        This stage focuses on assigning EA management roles
management foundation           and responsibilities, establishing plans for developing EA
                                products, and committing the resources necessary for
                                developing these products.
Stage 1: Creating EA            This stage is characterized either by no plans to develop
awareness                       and use an EA or by plans that do not demonstrate an
                                awareness of the value of having and using an EA.
Source: GAO.


EOUSA has satisfied many of the framework’s core elements. Specifically,
it has satisfied about 80 percent of the elements associated with building
the EA management foundation—stage 2 of our EA management maturity
framework—and half of the 12 core elements associated with higher
maturity stages. At stage 2, it has established a chief architect and has
selected a framework (the Federal Enterprise Architecture Framework)
and, according to officials, selected a tool (the Enterprise Architecture
Management System) to serve as a repository for its EA artifacts. At the
higher stages of our framework, the CIO, for example, approved a version
of an EA in May 2002 that describes the “as is” and “to be” environments for
its core business functions.

However, the office has yet to satisfy several of the core elements that are
critical to effective EA management. For example, a committee or group
representing the enterprise has not yet been established to guide and
oversee the development of future versions of the architecture. Instead, the
current version of its architecture has been primarily guided and directed
by the CIO’s office. Until a committee or group representing the enterprise


Page 9                                                 GAO-03-751 EOUSA IT Management
                                                is established, there is increased risk that the architecture will not
                                                represent a corporate decision-making tool and will not be viewed and
                                                endorsed officewide as such a tool.

                                                Another example is the absence of a written or approved policy for
                                                maintaining the EA. Without a documented, approved policy for EA
                                                maintenance that, for example, assigns responsibility and accountability
                                                for configuration management and version control, EOUSA risks allowing
                                                its architecture to become outdated and irrelevant, thus limiting its
                                                effectiveness in selecting and guiding IT investments.

                                                EOUSA does not have a written plan of action for strengthening EA
                                                management and evolving the current version of its EA, because, according
                                                to the CIO, developing such a plan is not a priority. Table 2 shows EOUSA’s
                                                performance in addressing the core elements of our maturity framework.



Table 2: Assessment of EOUSA’s EA Efforts against GAO’s EA Maturity Framework

Stage                    Core element                                Satisfied?      Comments
Stage 5: Leveraging      Written/approved policy exists for EA       No              According to agency officials, there is no
the EA for managing      maintenance.                                                written/ approved policy for EA
change                                                                               maintenance.
(includes all elements   EA steering committee, investment           No              The EA has not been reviewed by any
from stage 4)            review board, or agency head has                            steering committee or investment review
                         approved EA.                                                board or by the agency head.
                         Metrics exist for measuring EA benefits.    No              According to agency officials, metrics for
                                                                                     measuring EA benefits have not been
                                                                                     developed.
Stage 4: Completing Written/approved policy exists for IT            No              While there are criteria for ranking IT
architecture products investment compliance with EA.                                 investments that call for determining
(includes all elements                                                               compliance with the EA, no
from stage 3)                                                                        written/approved policy addresses this.
                         EA products describe the enterprise’s       Yes             EA products describe EOUSA’s core
                         business—and the data, applications,                        business functions and the data,
                         and technology that support it.                             applications, and technology that support
                                                                                     them.
                         EA products describe the “as is”            Yes             EA products describe the “as is”
                         environment, the “to be” environment,                       environment, the “to be” environment, and a
                         and a sequencing plan.                                      sequencing plan.
                         Agency CIO has approved EA.                 Yes             The CIO approved the EA in May 2002.




                                                Page 10                                        GAO-03-751 EOUSA IT Management
(Continued From Previous Page)
Stage                    Core element                                Satisfied?   Comments
Stage 3: Developing      Written/approved policy exists for EA       No           According to agency officials, there is no
architecture products    development.                                             approved policy for EA development.
(includes all elements   EA products are under configuration         No           Although agency officials reported that EA
from stage 2)            management.                                              products are under configuration
                                                                                  management, they could not provide
                                                                                  documentation to support this statement.
                         EA products describe or will describe the   Yes          EA products describe core business
                         enterprise’s business—and the data,                      functions and the data, applications, and
                         applications, and technology that support                technology that support them.
                         it.
                         EA products describe or will describe the   Yes          EA products describe the “as is”
                         “as is” environment, the “to be”                         environment, the “to be” environment, and a
                         environment, and a sequencing plan.                      sequencing plan.
                         EA scope is enterprise-focused.             Yes          EA products describe the “as is” and “to be”
                                                                                  environments for the enterprise’s core
                                                                                  business functions.
Stage 2: Building the    Committee or group representing the         No           There is no committee or group
EA management            enterprise is responsible for directing,                 representing the enterprise that is
foundation               overseeing, or approving EA.                             responsible for directing, overseeing, or
                                                                                  approving the EA. The CIO is currently
                                                                                  responsible for direction, oversight, and
                                                                                  approval of the architecture.
                         Program office responsible for EA           Yes          Roles and responsibilities for developing the
                         development exists.                                      EA were assigned to a group of individuals.
                         Chief architect exists.                     Yes          The CIO has been designated as the chief
                                                                                  architect.
                         EA is being developed using a framework     Yes          The EA was developed using the Federal
                         and an automated tool.                                   Enterprise Architecture Framework. In
                                                                                  addition, agency officials reported that they
                                                                                  are using the Enterprise Architecture
                                                                                  Management System as their EA tool.
                         EA plans call for describing the            Yes          EA products describe core business
                         enterprise in terms of business, data,                   functions and the data, applications, and
                         applications, or technology.                             technology that support them.
                         EA plans call for describing “as is”        Yes          EA products describe the “as is”
                         environment, “to be” environment, or                     environment, the “to be” environment, and a
                         sequencing plan.                                         sequencing plan.
Stage 1: EA              Agency is aware of EA.                      Yes          In August 2002, the CIO issued a memo to
awareness                                                                         agency staff notifying them of the need to
                                                                                  use the EA to inform investment
                                                                                  management decisions.
Source: GAO.




                                                   Page 11                                  GAO-03-751 EOUSA IT Management
EOUSA Has Not Established   Effective IT investment management provides for evaluating each
Key Capabilities            proposed and ongoing investment, based on EA alignment and measurable
                            risks and returns and for selecting and controlling these investments as a
Needed to Effectively       portfolio of competing investment options. We have developed a
Manage IT Investments       framework that defines and measures an organization’s maturity in IT
                            investment management (ITIM) and provides a basis for improving
                            investment management.11 This framework, which is based on the IT
                            investment management practices of leading private- and public-sector
                            organizations, is structured to permit progression through five maturity
                            stages (shown in table 3). Each maturity stage consists of critical processes
                            and key practices that should be implemented for an organization to
                            become more effective in managing its IT investments.



                            Table 3: The Five Stages of GAO’s ITIM Maturity Framework

                            Stage                 Description
                            Stage 5:              Investment benchmarking and IT-enabled change management
                            Leveraging IT for     techniques are deployed to strategically shape business outcomes.
                            strategic outcomes
                            Stage 4:              Process evaluation techniques focus on improving the performance
                            Improving the         and management of the organization’s IT investment portfolio.
                            investment process
                            Stage 3:            Comprehensive techniques are in place for selection and control of
                            Developing a        the IT investment portfolio that incorporate benefit and risk criteria
                            complete investment linked to mission goals and strategies.
                            portfolio
                            Stage 2:              Repeatable investment control techniques are in place and the key
                            Building the          foundation capabilities have been implemented.
                            investment
                            foundation
                            Stage 1:            IT management processes are ad hoc, project-centric, and have
                            Creating investment widely variable outcomes.
                            awareness
                            Source: GAO.


                            According to the framework, the first key step toward an effective
                            investment management process is to build the investment foundation. An


                            11
                             U.S. General Accounting Office, Information Technology Investment Management: A
                            Framework for Assessing and Improving Process Maturity (Exposure Draft), GAO/AIMD-
                            10.1.23 (Washington, D.C.: May 2000).




                            Page 12                                                 GAO-03-751 EOUSA IT Management
                                               organization with this foundation (stage 2 maturity) has attained
                                               repeatable, successful investment control processes and basic selection
                                               processes at the project level. Successful management at this level allows
                                               an organization to measure the progress of existing IT projects and to
                                               identify variances in cost, schedule, and performance expectations by
                                               following established, disciplined processes. The organization should also
                                               be able to take corrective action, if appropriate, and should possess basic
                                               capabilities for selecting new project proposals. To accomplish this level of
                                               basic control, an organization should establish an investment board,
                                               identify the business needs and opportunities to be addressed by each
                                               project, and use this knowledge in the selection of new proposals.

                                               The office has satisfied two of the critical processes for stage 2, but it has
                                               not satisfied the other three. Specifically, it has established an investment
                                               governing board, known as the Investment Review Board (IRB) and
                                               developed a guide to direct its operations. It is also defining project needs
                                               in alignment with the agency’s mission goals. However, the office has not,
                                               for example, defined procedures for project oversight. In addition, while an
                                               IT project and systems inventory exists as part of its “as is” architecture, a
                                               policy specifying how it will be used for investment management purposes
                                               has not been defined. Until EOUSA satisfies all critical processes for stage
                                               2, it will not have the foundation it needs to build its investment
                                               management capability and it will not have an effective investment
                                               process. Table 4 summarizes our assessment of stage 2 capabilities.



Table 4: Assessment of EOUSA’s ITIM Efforts against Stage 2 of GAO’s ITIM Maturity Framework

Critical process       Description                                   Satisfied?      Comments
IT investment board    Define and establish (1) the governing        Yes             A governing board (the IRB) has been
operation              board(s) responsible for selecting,                           defined and established, and guidance
                       controlling, and evaluating IT investments                    directing the board’s operations exists.
                       and (2) a guide directing the board(s)
                       operations.
IT project oversight   Regularly oversee each IT project’s           No              According to agency officials, projects are
                       progress toward cost and schedule                             managed using earned value management a
                       milestones, using established criteria,                       to regularly determine project progress and
                       and require corrective actions when                           control project costs and schedules.
                       milestones are not achieved.                                  However, there are no procedures defining
                                                                                     how the investment board is to regularly
                                                                                     oversee each project’s progress and require
                                                                                     corrective actions when milestones are not
                                                                                     achieved.




                                               Page 13                                         GAO-03-751 EOUSA IT Management
(Continued From Previous Page)
Critical process        Description                                        Satisfied?           Comments
IT project and system Create and maintain an IT project and                No                   An IT project and systems inventory exists
identification        systems inventory to assist in managerial                                 as part of the EA. However, there is no
                      decision making.                                                          policy defining how this inventory is to be
                                                                                                used in managerial decision making.
Business needs          Ensure that each IT project supports the           Yes                  Business needs and associated users have
identification for IT   organization’s business needs and meets                                 been identified, and users participate in the
projects                users’ needs.                                                           management of the project. We verified this
                                                                                                for the Enterprise Case Management
                                                                                                System project, which officials told us is
                                                                                                representative of how they intend to acquire
                                                                                                systems.
Proposal selection      Ensure that an established, structured             No                   A selection process using risk and return
                        process is used to select new IT                                        criteria is defined. However, officials stated
                        proposals.                                                              that this selection process has not yet been
                                                                                                used.
Source: GAO.
                                               a
                                                Earned value is a management technique that relates resource planning to schedules and to technical
                                               cost and schedule requirements. There are two major objectives of an earned value system: to
                                               encourage contractors to use effective internal cost and schedule management control systems, and
                                               to permit the customer to rely on timely data produced by those systems for determining product-
                                               oriented contract status.


                                               EOUSA has not demonstrated that maturing its IT investment management
                                               process is a priority by developing a plan for doing so and devoting
                                               resources to execute the plan. Until the office develops and implements a
                                               plan for establishing mature IT investment management processes
                                               (including all critical processes for building the investment management
                                               foundation), EOUSA will not have the full suite of capabilities it needs to
                                               ensure that project selection and control processes are repeatable or that it
                                               has the best mix of investments to meet agency priorities.




                                               Page 14                                                      GAO-03-751 EOUSA IT Management
EOUSA Has Not                     Effective information security management is critical to EOUSA’s ability to
Implemented Effective             ensure the reliability, availability, and confidentiality of its information
                                  assets, and thus it is fundamental to its ability to perform its mission. Our
Security Practices                research into public- and private-sector organizations with strong
                                  information security programs shows that leading organizations’ programs
                                  include (1) establishing a central security focal point with appropriate
                                  resources, (2) continuously assessing business risks, (3) implementing and
                                  maintaining policies and controls, (4) promoting awareness, and
                                  (5) monitoring and evaluating the effectiveness of policies and controls.12

                                  Currently, EOUSA is not fully satisfying any of these tenets of effective
                                  security. In addition, it has not demonstrated that institutionalizing
                                  effective security practices is a priority by developing a plan to guide its
                                  efforts to address security weaknesses and committing resources to
                                  perform essential security functions. Until such a plan is developed and
                                  effectively implemented, data, systems, and networks are at risk of
                                  inadvertent or deliberate misuse, fraud, improper disclosure, or
                                  destruction—possibly without detection. For example, the reliability and
                                  integrity of case information may be compromised, or sensitive crime
                                  victim information may be improperly disclosed.

Central Security Focal Point Is   According to our framework, central management of key security functions
Established but Has Not Been      is the foundation of an effective information security program, because it
Appropriately Resourced           allows knowledge and expertise to be incorporated and applied on an
                                  enterprisewide basis. Having a central security focal point supported by
                                  appropriate resources is especially important for managing the increased
                                  risks associated with a highly connected computing environment, such as
                                  JCN, where security weaknesses in one segment of an organization’s
                                  network can compromise the security of another segment’s IT assets. In
                                  addition, centralizing the security management function provides a focal
                                  point for coordinating the activities associated with the other four elements
                                  of a strong information security program.




                                  12
                                   U.S. General Accounting Office, Information Security Management: Learning from
                                  Leading Organizations, GAO/AIMD-98-68 (Washington, D.C.: May 1998).




                                  Page 15                                            GAO-03-751 EOUSA IT Management
                             In June 2001, EOUSA appointed a security officer with responsibility for
                             centrally managing all aspects of IT security. However, EOUSA has not
                             assigned sufficient staff to adequately carry out these responsibilities. For
                             example, no staff has been assigned to monitor firewall logs13 or support
                             the development of a centrally managed IT security training program—
                             activities that fall under the security officer’s purview. Each of these
                             activities is discussed further in the following sections.

                             Officials said that they recognize the need for additional staff resources to
                             perform these activities. They also stated that they were in the process of
                             hiring two people to support security functions, but they agreed that this
                             would still not allow for performance of key security responsibilities.
                             Without an appropriately resourced security program, security breaches
                             may not be detected or addressed in a timely manner, awareness of
                             security requirements across the organization may be inconsistent, and
                             vulnerabilities in the current IT environment may not be appropriately
                             addressed.

Risks Have Not Always Been   According to our framework, identifying and assessing business risks is an
Assessed                     essential step in determining what IT security controls are needed and
                             what resources should be invested in these controls. Federal guidance
                             advocates performing risk assessments at least once every 3 years—or
                             when a significant change in a system or the systems environment (e.g.,
                             new threats) has occurred. These assessments should address the risks
                             that are introduced through connections to other networks and the impact
                             on an organization’s mission should network security be compromised. In
                             line with this guidance, EOUSA’s certification and accreditation14 process
                             requires that a risk assessment be completed for each system before any
                             office can use it.




                             13
                               Logs keep track of accesses to and attempts to access the networks that the firewalls are
                             intended to secure.
                             14
                              Certification is the technical and nontechnical evaluation that is conducted to verify that
                             IT systems comply with security requirements. Accreditation is the formal declaration that
                             the appropriate safeguards have been properly implemented and that the residual risk is
                             acceptable.




                             Page 16                                                 GAO-03-751 EOUSA IT Management
                                 According to EOUSA, a major system that recently underwent EOUSA’s
                                 certification and accreditation process is the replacement for the existing
                                 firewall/VPN system. This system is intended to be the foundation for
                                 secure communications between EOUSA, Justice, and the geographically
                                 dispersed USAOs. Accordingly, we analyzed this system and found that
                                 while the firewall/VPN replacement system has been certified and
                                 accredited, the existing firewall/VPN system—which was deployed in 1996
                                 and, as of May 9, 2003, was operating at 75 of the 240 sites15—had not had a
                                 risk assessment performed and had not been certified and accredited.
                                 Officials told us that they have not performed such an assessment on this
                                 network because (1) it is not cost-effective to use limited resources to
                                 perform an assessment on a network that is to be fully replaced by June 30,
                                 2003, and (2) the risks inherent in the network are minimal, given that it
                                 resides on Justice’s JCN, for which they said they assume Justice had
                                 performed risk assessments.

                                 We agree that it does not make sense at this point to perform a risk
                                 assessment on the existing firewall/VPN system given that the replacement
                                 system is expected to be fully deployed by the end of June 2003. However,
                                 this does not change the fact that EOUSA has operated the network for
                                 about 7 years without understanding its exposure to risk. This is
                                 particularly important, because EOUSA officials could not provide us with
                                 evidence to support the assumption that Justice had performed a risk
                                 assessment for JCN. Moreover, previous studies have shown that Justice
                                 has had long-standing weaknesses in several aspects of its IT security
                                 program.16 According to EOUSA, its recently established certification and
                                 accreditation program will not allow this to happen again.

Key Security Controls Have Not   According to our framework, risk-based, cost-effective security policies
Been Implemented                 and related technology controls (such as firewalls configured to explicit
                                 rules and intrusion detection devices17 located to monitor key network
                                 assets) and procedural controls (such as contingency plans) are needed to


                                 15
                                  EOUSA is responsible for IT operations at 93 geographically dispersed USAOs and their
                                 branches, which comprise about 240 sites.
                                 16
                                  U.S. Department of Justice, FY 2000 Performance Report-FY 2002 Performance Plan,
                                 April 2001; U.S. General Accounting Office, Major Management Challenges and Program
                                 Risks: Department of Justice, GAO-03-105 (January 2003).
                                 17
                                  Intrusion detection devices are software or hardware systems that monitor network traffic
                                 and help identify cyberthreats.




                                 Page 17                                                GAO-03-751 EOUSA IT Management
                            protect a system from compromise, subversion, and tampering. Federal
                            and Justice guidance also advocate establishing these policies and
                            controls.

                            While EOUSA is guided by many Justice security policies, it has not yet
                            implemented key security controls that are needed to satisfy them. For
                            example, CIO officials told us that the existing firewall/VPN system, which,
                            as of May 9, 2003, was operating at 75 sites, is not based on explicit firewall
                            rules. Moreover, according to these officials, no intrusion detection devices
                            monitor the wide-area network (WAN)18 routers, firewalls, and VPN
                            devices. Rather, the intrusion detection devices that are currently
                            implemented are located only within the local area network environment
                            (i.e., within a USAO). Also, the contingency plan developed for the
                            replacement firewall/VPN system was not prepared according to federal
                            guidelines. For example, the contingency plan does not specify procedures
                            for notifying recovery personnel or assessing damage to systems. CIO
                            officials told us that they had not implemented these security controls
                            because, as previously noted, they believe the risks inherent in the network
                            are minimal given that it resides on Justice’s JCN, for which they said they
                            assumed Justice had performed risk assessments. However, as previously
                            stated, EOUSA provided no evidence to support this assumption, and
                            Justice has had longstanding security weaknesses.

                            Until EOUSA implements security controls, it may be unaware of
                            vulnerabilities, increasing the risk that intruders may take control of
                            network devices or that data passing through its firewalls can be read or
                            manipulated. In addition, EOUSA may not be able to respond to security
                            breaches adequately and in a timely manner. This is particularly
                            threatening given the sensitivity of the information used by the USAOs in
                            performing their work.

EOUSA Does Not Adequately   According to our framework, promoting user awareness through education
Promote User Awareness      and training is essential to successfully implementing information security
                            policies, achieving user understanding of security policies, and ensuring
                            that security controls are instituted properly. This is because computer
                            users—and others with access to information resources—are not able to
                            comply with policies of which they are unaware or which they do not fully
                            understand. Our framework suggests that a central group be tasked with


                            18
                              A wide-area network is a network that provides data communications to a large number of
                            independent users and spans a relatively large geographical area.




                            Page 18                                               GAO-03-751 EOUSA IT Management
                              educating users about current information security risks and helping to
                              ensure consistent understanding and administration of policies.

                              As previously mentioned, the security officer is responsible for promoting
                              awareness of computer security. However, the security officer does not
                              carry out this responsibility because provision of the resources to do so has
                              not been viewed as an agency priority. According to the security officer,
                              each district is thus responsible for managing its own IT training program,
                              and the security officer does not know to what extent these programs
                              address awareness of computer security. Without a centralized approach to
                              security education and training, the security officer cannot adequately
                              ensure that users are consistently aware of or fully understand the
                              organizational policies and procedures with which they are expected to
                              comply, thus risking the integrity, reliability, and confidentiality of data and
                              systems. According to EOUSA officials, they plan to hire staff to develop
                              and implement a centralized program by August 2003.

EOUSA Is Not Monitoring the   Our framework recognizes the need to continuously monitor controls,
Effectiveness of Security     through tests and evaluations, to ensure that the controls have been
Controls                      appropriately implemented and are operating as intended. Further,
                              Justice’s policy requires annual testing of security controls and requires
                              EOUSA to (1) verify that the policies and procedures in component
                              organizations are consistent with this policy and (2) enforce compliance
                              with component and Justice security policies, including identifying
                              sanctions and penalties for noncompliance. In addition, our framework and
                              related best practices—as well as Justice’s own policy—advocate keeping
                              summary records of security incidents, to allow measurement of the
                              frequency of various types of violations and the damage suffered from
                              these incidents. This type of oversight is critical because it enables
                              management to identify problems and their causes—and to make the
                              necessary corrections.

                              CIO officials told us that testing has never been conducted to determine
                              whether EOUSA’s policies and procedures are consistent with Justice’s and
                              whether security controls are generally effective. According to these
                              officials, testing has not been a priority because they assumed that Justice
                              was performing tests of the WAN environment. However, Justice officials
                              told us that, although they had evaluated the contractor’s management of
                              the WAN’s circuits, they had not performed any tests to determine the
                              effectiveness of technical and other controls associated with the WAN. The
                              lack of testing heightens the risk that individuals both within and outside
                              Justice could compromise EOUSA’s external and internal security controls



                              Page 19                                          GAO-03-751 EOUSA IT Management
                            to gain extensive unauthorized access to its networks and to networks to
                            which it is connected.

                            EOUSA officials also told us that, contrary to Justice’s policy, they do not
                            maintain summary records of security incidents. Specifically, the
                            production firewall/VPN software and routers at over 240 locations do not
                            have audit logs that are activated, and the replacement routers, firewalls,
                            and VPN devices are being implemented with no audit logs activated.
                            According to these officials, they have not activated the audit logs because
                            resources have not been allocated to provide for this security control. This
                            lack of auditing heightens the risk of undetected intruders using EOUSA’s
                            systems to modify, bypass, or negate its firewalls and routers. Additionally,
                            without these audit logs the office would be unable to reconstruct security-
                            related incidents.



EOUSA Is Employing          Rigorous and disciplined system acquisition processes and practices can
Important Acquisition       reduce the risk of fielding systems that do not perform as intended, are
                            delivered late, or cost more than planned. The Software Engineering
Management Practices on a
                            Institute (SEI), recognized for its expertise in acquiring software-intensive
Key System                  systems, has published models and guides for determining an
                            organization’s acquisition process maturity. One of those models, referred
                            to as the Software Acquisition Capability Maturity Model (SA-CMM),19
                            addresses an organization’s acquisition management ability.20 The SA-CMM
                            model defines organizational maturity according to five levels (see table 5).




                            19
                             Carnegie Mellon University’s Software Engineering Institute, Software Acquisition
                            Capability Maturity Model, Version 1.02, CMU/SEI-99-TR-002 (April 1999).
                            20
                                 EOUSA officials told us that they primarily acquire their systems.




                            Page 20                                                     GAO-03-751 EOUSA IT Management
Table 5: SA-CMM Levels and Descriptions

Level          Description
Level 5:       Continuous process improvement is empowered by quantitative feedback
Optimizing     from the process and from piloting innovative ideas and technologies.
Level 4:     Detailed measures of the acquisition processes, products, and services are
Quantitative quantitatively and qualitatively understood and controlled.
Level 3:       The acquisition organization’s software acquisition process is documented
Defined        and standardized. All projects use an approved, tailored version of the
               organization’s standard process for acquiring their products and services.
Level 2:       Basic management processes for acquisition projects are established to
Repeatable     plan all aspects of the acquisition, manage requirements, track project
               team and contractor team performance, manage the project’s cost and
               schedule baselines, evaluate the products and services, and successfully
               transition to its support organization. The necessary process discipline is in
               place to repeat earlier successes on projects in similar domains.
Level 1:       The acquisition process is characterized as ad hoc, and occasionally even
Initial        chaotic. Few processes are defined and success depends on individual
               effort.
Source: SEI.


According to SEI, level 2 (the repeatable level) demonstrates that basic
management processes, known as key process areas, have been
established to track performance, cost, and schedule, and that the
organization has the means to repeat earlier successes on similar projects.
An organization that has these processes in place is in a much better
position to successfully acquire software-intensive systems than an
organization that does not.

As a Justice component, EOUSA must comply with all departmental
policies and procedures, including Justice’s system development life-cycle
management guidance. Since EOUSA officials told us that the Enterprise
Case Management System (ECMS), which is intended to be the enterprise
solution for managing and tracking case workload within the USAOs, is the
first acquisition effort to follow Justice guidance from its inception, we
compared this project, and the Justice guidance used to manage it, against
SEI’s SA-CMM, and we found that the project was being managed in
accordance with the majority of the applicable level 2 practices. Table 6
represents a summary of our findings for this acquisition (see app. I for an
expanded analysis).




Page 21                                                   GAO-03-751 EOUSA IT Management
Table 6: Assessment of ECMS Acquisition against SEI’s SA-CMM Level 2 Key Process Area

                                                                                                                                  Key
                                                                                                                     Key    practices
SA-CMM level 2 key process                                                                           Total key practices          not
area                              Description                                                        practices performed   performed
Software acquisition planning     Ensure that reasonable planning for the acquisition is
                                  conducted and that all elements of the project are included.             15         13           2
Solicitation                      Ensure that award is made to the contractor most capable of
                                  satisfying the specified requirements.                                   18         16           2
Requirements development and Establish a common and unambiguous definition of acquisition
management                   requirements to be used by the acquisition team, the system’s
                             users, and the contractor.                                                    14         14           0
Project management                Manage the activities of the project office and supporting
                                  contractor(s) to ensure a timely, efficient, and effective
                                  acquisition.                                                             16         16           0
Contract tracking and oversight   Ensure that the activities under contract are being performed in
                                  accordance with contractual requirements and that products
                                  and services will satisfy contract requirements.                         17         16           1
Evaluation                        Determine that the acquired products and services satisfy
                                  contract requirements before accepting and supporting them.              15          6           9
Source: GAO.


                                                More specifically, the office has performed all of the key practices in the
                                                requirements development and management and project management
                                                key process areas. These include (1) establishing a written policy for
                                                developing and managing system-related contractual requirements; (2)
                                                having bi-directional traceability between the contractual requirements and
                                                the contractor’s work products and services; (3) measuring and reporting
                                                to management on the status of requirements development and
                                                management activities; (4) designating responsibility for project
                                                management; (5) keeping plans current during the life of the project as re-
                                                planning occurs, issues are resolved, requirements are changed, and new
                                                risks are discovered; and (6) tracking the risks associated with cost,
                                                schedule, resources, and the technical aspects of the project.

                                                EOUSA has also performed the majority of the key practices in the
                                                remaining four process areas. However, it does not have written policies
                                                for either the contract tracking and oversight or the software acquisition
                                                planning key process areas. Policies in general are key to establishing
                                                well-defined and enduring processes and procedures. In these two areas,
                                                policies would ensure that the office’s approach to tracking and overseeing
                                                contractors and planning the acquisition is defined in a repeatable and



                                                Page 22                                               GAO-03-751 EOUSA IT Management
                  measurable fashion. In addition, during the solicitation process, the office
                  did not document its plans for solicitation activities, which would provide
                  those involved with objectives for the solicitation process and a defined
                  way to manage and control solicitation activities and decisions. In
                  evaluation, the office has yet to satisfy 9 of the 15 required practices.
                  Officials told us that they intend to satisfy them but that they do not have a
                  plan for addressing those practices or for implementing all of the required
                  practices on future system acquisitions. According to these officials,
                  developing such a plan is currently not a priority.

                  By developing and implementing a plan for satisfying all of these key
                  process areas on ECMS and future acquisitions, EOUSA can increase its
                  chances of successfully acquiring needed system capabilities on time and
                  within budget.



Conclusions       EOUSA has taken important steps to define and implement four key IT
                  management disciplines. Nevertheless, key aspects of each discipline have
                  yet to be institutionalized, leaving the office challenged in its ability to
                  achieve the department’s strategic goal of improving the integrity, security,
                  and efficiency of its IT systems. Critical to the office’s success going
                  forward will be treating institutionalization of each of these management
                  disciplines as priority matters by developing integrated plans of action for
                  addressing the weaknesses that we identified in each and effectively
                  implementing these plans—including assignment of appropriate resources
                  and measurement and reporting of progress. Without taking these steps,
                  EOUSA is unlikely to fully establish the IT management capabilities it
                  needs.



Recommendations   To strengthen the office’s IT management capacity and increase its chances
                  of improving the integrity, security, and efficiency of its IT systems, we
                  recommend that the Attorney General direct the EOUSA Director to treat
                  institutionalization of EA management, IT investment management, IT
                  security management, and system acquisition management as priorities
                  by developing and implementing action plans to address the weaknesses in
                  each discipline that are identified in this report. These plans should, at a
                  minimum, provide for accomplishing the following:




                  Page 23                                        GAO-03-751 EOUSA IT Management
For EA management,

• establish a committee or group representing the enterprise that is
  responsible for directing, overseeing, or approving the EA;

• ensure that EA products are under configuration management;

• define, approve, and implement a policy for IT investment compliance
  with the EA;

• specify metrics for measuring EA benefits; and

• define, approve, and implement a policy for maintaining the EA.

For IT investment management,

• regularly oversee each IT project’s progress toward cost and schedule
  milestones, using established criteria, and require corrective actions
  when milestones have not been achieved;

• define and implement a policy for using the IT project and systems
  inventory for managerial decision making; and

• ensure that an established, structured process is used to select new IT
  proposals.

For IT security management,

• allocate the appropriate resources to enable the responsibilities of the
  security officer to be fully performed;

• ensure that risk assessments are performed on all existing and future
  systems;

• implement intrusion detection devices to monitor activity at the routers,
  firewalls, and VPN devices, and implement other network security
  controls as noted in the report;

• develop and implement a centralized approach to security education
  and training; and




Page 24                                       GAO-03-751 EOUSA IT Management
                      • perform regular tests to determine compliance with policies and
                        procedures and the effectiveness of security controls.

                      For system acquisition management,

                      • develop and implement a policy for contract tracking and oversight;

                      • develop and implement a policy for system acquisition planning; and

                      • address the remaining key practices associated with evaluation as
                        ECMS progresses in the life cycle; and

                      • ensure that the Software Engineering Institute acquisition practices
                        identified in this report are used in future system acquisitions.

                      In developing these plans, the Director should ensure that each plan (1) is
                      integrated with the other three plans; (2) defines clear and measurable
                      goals, objectives, and milestones; (3) specifies resource needs; and
                      (4) assigns clear responsibility and accountability for implementing the
                      plan. In implementing each plan, the Director should ensure that the
                      needed resources are provided and that progress is measured and reported
                      periodically to the Attorney General.



Agency Comments and   In written comments on a draft of this report signed by the EOUSA Director
                      (reprinted in app. III), the office agreed with our findings relative to
Our Evaluation        enterprise architecture management, IT investment management, and
                      system acquisition management. EOUSA also agreed with our
                      recommendations in these three areas and stated that it intends to
                      implement the recommendations. However, EOUSA stated that it disagreed
                      with our findings and our recommendations regarding information security
                      management, although at the same time it cited certain actions that it
                      intends to take, such as implementing a centralized security training
                      program and monitoring security audit logs, that are consistent with our
                      security findings and associated recommendations. Further, the office
                      disagreed that the state of its efforts to institutionalize best management
                      practices in the four areas is due to it not treating each area as an office
                      priority. It also disagreed with our conclusion that the state of its efforts to
                      institutionalize best practices currently limits its ability to meet Justice’s
                      strategic goal of improving its IT systems, and that the USAOs will be
                      challenged in their ability to effectively and efficiently meet mission goals




                      Page 25                                          GAO-03-751 EOUSA IT Management
and priorities. Each of these three areas of disagreement is addressed
below.

First, with respect to information security management, EOUSA stated that
it has one of the strongest security programs in Justice, and perhaps the
federal government. To support this statement, the office cited 10 security
initiatives it has implemented, such as certification and accreditation of
more than eight systems, real-time encryption of all data in laptops and
handheld devices, and conduct of vulnerability assessments and
penetration testing. It also noted, among other things, that it had added 10
field security positions and 2 headquarters positions, and that its data are
monitored 24 hours a day, seven days a week, and have never been
compromised. We do not question these statements concerning the office’s
information security program and associated activities because (1) the
purpose and scope of our review was not to compare EOUSA to other
Justice component organizations or other federal agencies, and thus
EOUSA’s relative standing is not relevant to the findings in our report and
(2) the message of our report is not that EOUSA has not taken steps to
improve its information security posture, but rather that the office’s
information security management efforts, including ongoing and complete
improvement steps, are weak in a number of areas relative to information
security management best practices. Accordingly, we make
recommendations aimed at addressing identified weaknesses, including a
recommendation to implement network intrusion detection devices and
other security controls. While EOUSA’s comments cited plans that are
consistent with many of our security-related recommendations, it
disagreed with the recommendation relative to its wide area network on
the grounds that this network is managed, secured, and monitored by
Justice and Sprint. We understand that the WAN is not managed by EOUSA,
and accordingly our recommendation was aimed at actively monitoring the
network routers, firewalls, and VPN devices, which are managed by
EOUSA. To avoid any confusion about this recommendation, we have
clarified its wording to better reflect our intentions. Similarly, in light of the
recent progress that EOUSA has made replacing its VPN system, we have
adjusted our finding and recommendation concerning the office’s exposure
to risk from its old VPN system.

Second, with respect to our statements that EOUSA has not treated
institutionalization of each of the four IT management disciplines—
enterprise architecture management, IT investment management, system
acquisition management, and information security management—as
agency priorities, the office stated that these statements were unfair and



Page 26                                           GAO-03-751 EOUSA IT Management
that it did not agree with them. To support its position, EOUSA made the
following two points: (1) it has made tremendous progress, as evidenced by
our report recognizing those best practices that it is satisfying, and (2) it
has received the highest level of support from Justice, as evidenced by the
establishment of the EOUSA CIO position in 2001, the progress that has
been made in the last 2 years compared to other Justice component
organizations, and EOUSA’s being viewed by Justice senior management as
a leader in IT management. We do not challenge EOUSA’s two points
because they are not relevant to our position regarding treating
institutionalization of each of the four IT management disciplines as agency
priorities. Our position is based on two facts that EOUSA did not dispute:
(1) plans for addressing the weaknesses cited in our report do not exist and
(2) limitations in resources to address these weaknesses were cited by
EOUSA officials as the reason why the weaknesses exist. In our view, if
each of these areas were an agency priority, then plans would be in place to
address the weaknesses, and resources to execute the plans would be
committed.

Third, with respect to our conclusion that EOUSA is currently limited in its
ability to meet Justice’s strategic goal of improving its IT systems, and that
the USAOs are thereby challenged in their ability to effectively and
efficiently meet their mission goals and objectives, the office disagreed but
did not offer any comments to counter our conclusion beyond those cited
above. Given that any organization’s ability to effectively leverage
technology is determined in large part by its institutionalized capabilities in
these four IT disciplines, we have not modified our conclusion.

EOUSA provided additional comments that have been incorporated in the
report as appropriate. EOUSA’s written comments are reproduced in
appendix III, along with our detailed evaluation of each comment.


As arranged with your office, unless you publicly announce its contents
earlier, we plan no further distribution of this report until 30 days after the
date of this letter. At that time, we will send copies of this report to
interested congressional committees. We will also send copies to the
Director of the Office of Management and Budget, the Attorney General of
the United States, the EOUSA Director, and the EOUSA CIO. We will also




Page 27                                         GAO-03-751 EOUSA IT Management
send copies to others upon request. In addition, copies will be available at
no charge on our Web site at www.gao.gov.

Should you or your offices have questions on matters discussed in this
report, please contact me at (202) 512-3439. I can also be reached by E-mail
at hiter@gao.gov. An additional GAO contact and staff acknowledgments
are listed in appendix IV.




Randolph C. Hite
Director, Information Technology
Architecture and Systems Issues




Page 28                                        GAO-03-751 EOUSA IT Management
Appendix I

Objective, Scope, and Methodology                                                                      AA
                                                                                                        ppp
                                                                                                          ep
                                                                                                           ned
                                                                                                             n
                                                                                                             x
                                                                                                             id
                                                                                                              e
                                                                                                              x
                                                                                                              Iis




              Our objective was to determine the extent to which the Executive Office
              for United States Attorneys (EOUSA) has institutionalized key information
              technology (IT) management capabilities to achieve the Department of
              Justice’s strategic goal of improving the integrity, security, and efficiency of
              its IT systems. To meet this objective, we focused on whether EOUSA had
              institutionalized four key IT management disciplines: enterprise
              architecture management, IT investment management, information
              security management, and system acquisition management.

              • To evaluate EOUSA’s enterprise architecture (EA) management, we first
                solicited responses to an EA management questionnaire, reviewed EA
                plans and products, and interviewed officials to verify their responses.
                Next, we compared the information that we had collected with GAO’s
                February 2002 EA management maturity framework1 to determine the
                extent to which EOUSA was employing effective EA management
                practices. This framework is based on the Practical Guide to Federal
                Enterprise Architecture, published by the Chief Information Officers’
                (CIO) Council.2 We did not use the revised framework issued in April
                20033 because, by then, we had already completed our work.

              • To evaluate EOUSA’s IT investment management (ITIM), we used GAO’s
                ITIM framework4 and assessed the extent to which EOUSA had satisfied
                the critical processes associated with stage 2 of the five-stage
                framework—building the investment foundation. We focused on stage 2
                processes because officials told us that they had only recently begun
                defining and implementing the specific practices that are associated
                with this stage. To conduct our assessment, we reviewed relevant
                EOUSA and Justice policies, procedures, guidance, and
                documentation—including the office’s investment management guide
                and associated memorandums, project proposals, and budget

              1
               U.S. General Accounting Office, Information Technology: Enterprise Architecture Use
              across the Federal Government Can Be Improved, GAO-02-6 (Washington, D.C.: February
              2002).
              2
               Chief Information Officers Council, A Practical Guide to Federal Enterprise Architecture,
              Version 1.0 (February 2001).
              3
               U.S. General Accounting Office, A Framework for Assessing and Improving Enterprise
              Architecture Management, Version 1.1, GAO-03-584G (Washington, D.C.: April 2003).
              4
               U.S. General Accounting Office, Information Technology Investment Management: A
              Framework for Assessing and Improving Process Maturity (Exposure Draft), GAO/AIMD-
              10.1.23 (Washington, D.C.: May 2000).




              Page 29                                                GAO-03-751 EOUSA IT Management
Appendix I
Objective, Scope, and Methodology




   documents. We also interviewed the CIO and the senior official who is
   responsible for implementing IT investment management. We then
   compared this information with our maturity framework to determine
   the extent to which the office was employing effective IT investment
   management practices.

• To evaluate EOUSA’s information security management, we used our
  executive guide for information security management,5 as well as
  Justice policy and guidance and relevant EOUSA U.S. Attorney
  Procedures.6 We reviewed internal Justice and other reports identifying
  security weaknesses at Justice and EOUSA and information on how
  these weaknesses will be addressed. We also reviewed the certification
  and accreditation package and the deployment schedule for the virtual
  private network7 that the office is currently deploying, because EOUSA
  and the USAOs rely on this network to carry out its mission. We
  interviewed Justice officials and EOUSA officials within the Office of
  the CIO about the office’s security management.

• To evaluate EOUSA’s system acquisition management, we used the
  Software Engineering Institute’s Software Acquisition Capability
  Maturity Model,8 focusing on six of the seven key process areas that are
  defined for level 2 of the model’s five-level maturity scale.9 We focused
  on level 2 processes because they represent the minimum level of
  maturity needed to effectively manage system acquisition projects. We
  used the office’s acquisition of the Enterprise Case Management System
  as a case study because officials stated that it is representative of how

5
 U.S. General Accounting Office, Information Security Management: Learning from
Leading Organizations, GAO/AIMD-98-68 (Washington, D.C.: May 1998).
6
 See, for example, U.S. Department of Justice, Information Technology Security (DOJ
2640.2D (July 2001) and EOUSA, Access to Sensitive But Unclassified IT Resources, USAP
3-16.010.30.001(M) (March 2002).
7
 A virtual private network uses a public or shared telecommunication infrastructure to
provide remote users with secure access to an organization's network.
8
 Carnegie Mellon University’s Software Engineering Institute, Software Acquisition
Capability Maturity Model, Version 1.02, CMU/SEI-99-TR-002 (April 1999).
9
 The six key process areas that we evaluated are software acquisition planning,
solicitation, requirements development and management, project management, contract
tracking and oversight, and evaluation. We did not include the seventh key process area—
transition to support—in our evaluation because the system that we assessed had not yet
progressed to the point that this process area was relevant.




Page 30                                                GAO-03-751 EOUSA IT Management
Appendix I
Objective, Scope, and Methodology




   they intend to acquire systems. In addition, this system will be critical in
   providing fundamental support to the U.S. Attorneys as they work to
   achieve mission goals. We reviewed key project documentation, such as
   the concept of operations, project plan, and requirements traceability
   matrix, and we interviewed system acquisition officials. We also
   reviewed the Justice guidance used to manage the project. We then
   compared this information to the Software Acquisition Capability
   Maturity Model to determine the extent to which the office was
   employing effective system acquisition management practices.

We performed our work at EOUSA headquarters in Washington, D.C., from
November 2002 to May 2003, in accordance with generally accepted
government auditing standards.




Page 31                                         GAO-03-751 EOUSA IT Management
Appendix II

Assessment of ECMS Acquisition Practices
against Level 2 of SEI’s Software Acquisition
Capability Maturity Model                                                                                                                Appendx
                                                                                                                                               Ii




Table 7: Software Acquisition Planning

Common
feature         CMM key practice                                Satisfied?   Comments
Commitment 1    The acquisition organization has a written      No           EOUSA does not have a written policy for planning the
                policy for planning the software acquisition.                software acquisition.

Commitment 2    Responsibility for software acquisition         Yes          Responsibility for software acquisition activities was
                activities is designated.                                    designated to the ECMS project manager.
Ability 1       A group that is responsible for planning the    Yes          A group responsible for planning exists and includes the
                software acquisition exists.                                 project manager, administrative contract officer’s
                                                                             technical representative, and assistant directors of Case
                                                                             Management staff.
Ability 2       The acquisition organization provides           Yes          The acquisition organization provided experienced
                experienced software acquisition                             software acquisition management personnel to support
                management personnel to support project                      project software acquisition planning.
                software acquisition planning.
Ability 3       Adequate resources are provided for             Yes          According to EOUSA officials, adequate resources were
                software acquisition planning activities.                    provided for software acquisition planning activities.
Activity 1      Software acquisition planning personnel are Yes              Software acquisition planning personnel were involved
                involved in system acquisition planning.                     in system acquisition planning.

Activity 2      The project’s software acquisition planning     Yes          The project’s software acquisition planning was
                is accomplished in conjunction with system                   accomplished in conjunction with system acquisition
                acquisition planning.                                        planning.
Activity 3      The software acquisition strategy for the       No           There is no software acquisition strategy document.
                project is developed and documented.
Activity 4      Software acquisition planning addresses the Yes              Software acquisition planning addresses most critical
                elements of the software acquisition                         elements of the software acquisition process.
                process.
Activity 5      The project’s software acquisition planning Yes              Software acquisition planning information is included in
                is documented, and the planning                              the project management plan, which has been updated
                documentation is maintained over the life of                 once.
                the project.
Activity 6      Life-cycle support of the software is           Yes          Certain life-cycle support provisions (user training,
                included in software acquisition planning                    system growth) are documented in the project
                documentation.                                               management plan.
Activity 7      Life-cycle cost and schedule estimates for      Yes          Life-cycle cost and schedule estimates for the initial
                the software products and services being                     release of ECMS were prepared by the project team
                acquired are prepared and independently                      and independently reviewed by the administrative
                reviewed.                                                    contract officer’s technical representative.
Measurement 1   Measurements (e.g., planned vs. completed Yes                Measurements (e.g., estimated vs. actual cost and
                works) are made and used to determine the                    schedule) were made by the project team and used to
                status of the software acquisition planning                  determine the status of the software acquisition
                activities and resultant products.                           planning activities and resultant products.




                                              Page 32                                               GAO-03-751 EOUSA IT Management
                                                                 Appendix II
                                                                 Assessment of ECMS Acquisition Practices
                                                                 against Level 2 of SEI’s Software Acquisition
                                                                 Capability Maturity Model




(Continued From Previous Page)
Common
feature                  CMM key practice                                         Satisfied?       Comments
Verification 1           Software acquisition planning activities are             Yes              The project team reviews software acquisition planning
                         reviewed by acquisition organization                                      activities on a periodic basis.
                         management on a periodic basis.
Verification 2           Software acquisition planning activities are             Yes              The project manager reviews software acquisition
                         reviewed by the project manager on both a                                 planning activities on both a periodic and event-driven
                         periodic and event-driven basis.                                          basis.
Source: Key practice data from SEI; analysis and comments from GAO.




                                                                 Page 33                                                 GAO-03-751 EOUSA IT Management
                                               Appendix II
                                               Assessment of ECMS Acquisition Practices
                                               against Level 2 of SEI’s Software Acquisition
                                               Capability Maturity Model




Table 8: Solicitation

Common
feature          CMM key practice                                Satisfied?       Comments
Commitment 1     The acquisition organization has a written      Yes              The acquisition organization used the Department of
                 policy for the conduct of the solicitation.                      Transportation’s Value Added Niche Information
                                                                                  Technology Services (VANITS) vehicle, which provides
                                                                                  federal, state, and local government clients with access
                                                                                  to specialized technology services and support.
Commitment 2     Responsibility for the software portion of the Yes               Responsibility for the software portion of the solicitation
                 solicitation was designated.                                     was designated to a technical point of contact and an
                                                                                  administrative contract officer’s technical
                                                                                  representative.
Commitment 3     A selection official was designated to be       Yes              The technical point of contact and the administrative
                 responsible for the selection process and                        contract officer’s technical representative were
                 the decision.                                                    responsible for the selection process and the decision.
Ability 1        A group that is responsible for coordinating    Yes              A group consisting of assistant directors of the
                 and conducting the solicitation activities                       information technology staff exists. With the technical
                 exists.                                                          point of contact as the chair, this group conducted an
                                                                                  evaluation of vendors’ proposals.
Ability 2        Adequate resources were provided for the        Yes              Adequate resources were provided for solicitation
                 solicitation activities.                                         activities. EOUSA budgeted and paid a fee for using
                                                                                  services provided under the VANITS vehicle.
Ability 3        Individuals performing the solicitation         Yes              According to EOUSA officials, individuals performing
                 activities have experience or receive                            the solicitation activities have formal training or
                 training.                                                        experience.
Ability 4        The groups supporting the solicitation (e.g., No                 The supporting groups received an orientation, but this
                 end user, system engineering, and                                did not cover solicitation procedures.
                 application domain experts) receive
                 orientation on the solicitation’s objectives
                 and procedures.
Activity 1       The project’s solicitation activities were      No               The project team did not document its plans for
                 performed in accordance with its plans.                          solicitation activities.
Activity 2       Solicitation activities are conducted in a      Yes              The project team followed standard procedures
                 manner compliant with relevant laws,                             required by the Governmentwide Acquisition Contracts
                 policies, and guidance.                                          (GWAC).
Activity 3       The software and evaluation requirements        Yes              The project team incorporated the software and
                 are incorporated into the solicitation                           evaluation requirements into the solicitation package
                 package and resulting contract.                                  and resulting contract.
Activity 4       The project’s proposal evaluation activities    Yes              According to EOUSA officials, the project’s proposal
                 were performed in accordance with its                            evaluation activities were performed in accordance with
                 plans.                                                           its plans.
Activity 5       Cost and schedule estimates for the             Yes              Cost and schedule estimates for the software activity
                 software activity were prepared.                                 were prepared by the project team.




                                               Page 34                                                   GAO-03-751 EOUSA IT Management
                                                                 Appendix II
                                                                 Assessment of ECMS Acquisition Practices
                                                                 against Level 2 of SEI’s Software Acquisition
                                                                 Capability Maturity Model




(Continued From Previous Page)
Common
feature                   CMM key practice                                         Satisfied?       Comments
Activity 6                The software cost and schedule estimates                 Yes              Software cost and schedule estimates were
                          were independently reviewed for                                           independently reviewed for comprehensiveness and
                          comprehensiveness and realism.                                            realism by the administrative contracting officer’s
                                                                                                    representative.
Activity 7                The selection official uses proposal                     Yes              The selection official used proposal evaluation results
                          evaluation results to support his or her                                  to support his decision.
                          decision to select an offerer.
Activity 8                The project team and the offerer(s) review               Yes              The team reviewed four proposals and asked
                          the project’s software requirements and                                   contractors to provide presentations to ensure mutual
                          plans during negotiations to ensure mutual                                understanding.
                          understanding.
Measurement 1             Measurements were made and used to                       Yes              The VANITS program office kept the project team
                          determine the status of the solicitation                                  informed of the status of all activities. Measurements
                          activities and resultant products.                                        used to determine the status included the length of
                                                                                                    time taken for each activity.
Verification 1            The activities for solicitation were reviewed Yes                         The activities for solicitation were reviewed bi-weekly
                          by acquisition organization management on                                 by the designated selection official or acquisition
                          a periodic basis.                                                         organization management.
Verification 2            The activities for solicitation were reviewed            Yes              The activities for solicitation were reviewed by the
                          by the project manager or designated                                      project manager on both a periodic and an event-
                          selection official on both a periodic and an                              driven basis.
                          event-driven basis.
Source: Key practice data from SEI; analysis and comments from GAO.




                                                                 Page 35                                                  GAO-03-751 EOUSA IT Management
                                             Appendix II
                                             Assessment of ECMS Acquisition Practices
                                             against Level 2 of SEI’s Software Acquisition
                                             Capability Maturity Model




Table 9: Requirements Development and Management

Common
feature         CMM key practice                               Satisfied?       Comments
Commitment 1    The acquisition organization has a written     Yes              The project management plan includes guidelines for
                policy for developing and managing                              defining and controlling technical and nontechnical
                software-related requirements.                                  (software-related) requirements.
Commitment 2    Responsibility for requirements development Yes                 The ECMS project team is responsible for
                and management is designated.                                   requirements development and management.
Ability 1       A group that is responsible for performing     Yes              A Joint Application Development group is responsible
                requirements development and                                    for performing requirements development. The
                management activities exists.                                   contractor is responsible for performing requirements
                                                                                management.
Ability 2       Adequate resources are provided for            Yes              According to EOUSA officials, adequate resources
                requirements development and                                    were provided for requirements development and
                management activities.                                          management activities.
Ability 3       Individuals performing requirements            Yes              According to EOUSA officials, individuals performing
                development and management activities                           requirements development and management activities
                have experience or receive training.                            have experience or received training.
Activity 1      The project team performs its activities in    Yes              EOUSA officials reported that the project team
                accordance with its documented                                  performs its activities in accordance with its
                requirements development and                                    documented requirements development and
                management plans.                                               management plans.
Activity 2      The project team develops, baselines, and      Yes              According to EOUSA officials, the project team
                maintains software-related contractual                          develops, baselines, and maintains software-related
                requirements.                                                   contractual requirements.
Activity 3      The project team appraises requests for        Yes              The project team reviews requests for changes to
                changes to system requirements for their                        system requirements for their impact on ECMS.
                impact on the software being acquired.
Activity 4      The project team appraises all changes to      Yes              The project team reviews all changes to the software-
                the software-related contractual                                related contractual requirements for their impact on
                requirements for their impact on                                performance, architecture, supportability, system
                performance, architecture, supportability,                      resource utilization, and contract schedule and cost.
                system resource utilization, and contract
                schedule and cost.
Activity 5      Bi-directional traceability between the     Yes                 Bi-directional traceability between the contractual
                contractual requirements and the contractor                     requirements and the contractor’s team software work
                team’s software work products and services                      products and services is maintained by the project
                is maintained throughout the effort.                            team.

Activity 6      The end user and other affected groups are     Yes              EOUSA officials reported that the end user and other
                involved in the development of all software-                    affected groups are involved in the development of all
                related contractual requirements and any                        software-related contractual requirements and any
                subsequent change activity.                                     subsequent change activity.

Measurement 1   Measurements are made and used to              Yes              Measurements are made and used by the project team
                determine the status of the requirements                        to determine the status of the requirements
                development and management activities                           development and management activities and resultant
                and resultant products.                                         products.



                                             Page 36                                                 GAO-03-751 EOUSA IT Management
                                                                 Appendix II
                                                                 Assessment of ECMS Acquisition Practices
                                                                 against Level 2 of SEI’s Software Acquisition
                                                                 Capability Maturity Model




(Continued From Previous Page)
Common
feature                   CMM key practice                                         Satisfied?       Comments
Verification 1            Requirements development and                             Yes              Requirements development and management
                          management activities are reviewed by                                     activities are reviewed by the project team (and the
                          acquisition organization management (and                                  contractor) on a periodic basis.
                          the contractor) on a periodic basis.
Verification 2            Requirements development and                             Yes              Requirements development and management
                          management activities are reviewed by the                                 activities are reviewed by the project manager on both
                          project manager on both a periodic and                                    a periodic and event-driven basis.
                          event-driven basis.
Source: Key practice data from SEI; analysis and comments from GAO.




                                                                 Page 37                                                 GAO-03-751 EOUSA IT Management
                                             Appendix II
                                             Assessment of ECMS Acquisition Practices
                                             against Level 2 of SEI’s Software Acquisition
                                             Capability Maturity Model




Table 10: Project Management

Common
feature        CMM key practice                                 Satisfied?       Comments
Commitment 1   The acquisition organization has a written       Yes              A policy memo was issued requiring all information
               policy for executing the software project.                        technology projects to follow a streamlined life-cycle
                                                                                 methodology.
Commitment 2   Responsibility for project management is         Yes              Responsibility for project management is designated
               designated.                                                       to the ECMS project manager.
Ability 1      A team that is responsible for performing the Yes                 A team that is responsible for performing the project’s
               project’s software acquisition management                         software acquisition management exists. It includes a
               exists.                                                           project manager and case management staff.
Ability 2      Adequate resources for the project team are Yes                   According to EOUSA officials, adequate resources for
               provided for the duration of the software                         the project team are provided for the duration of the
               acquisition project.                                              software acquisition project.
Ability 3      When project trade-offs are necessary, the       Yes              When project trade-offs are necessary, the project
               project manager is permitted to alter the                         manager is permitted to alter the performance, cost, or
               performance, cost, or schedule software                           schedule software acquisition baseline after
               acquisition baseline.                                             appropriate review.
Ability 4      The project team has experience or               Yes              EOUSA officials reported that the project team
               receives training in project software                             members have experience or received training in
               acquisition management activities.                                project software acquisition management activities.
Activity 1     The project team performs its activities in      Yes              EOUSA officials reported that the project team
               accordance with its documented software                           performs its activities in accordance with its project
               acquisition management plans.                                     management plan.
Activity 2     The roles, responsibilities, and authority for   Yes              The roles, responsibilities, and authority for the project
               the project functions are documented,                             functions are documented in the ECMS project plan,
               maintained, and communicated to affected                          maintained, and communicated to affected groups.
               groups.
Activity 3     The project team’s commitments, and              Yes              The project team’s commitments, and changes to
               changes to commitments, are                                       commitments, are communicated to affected groups
               communicated to affected groups.                                  via an on-line discussion forum.
Activity 4     The project team tracks the risks associated Yes                  Project-wide risks are documented in the risk
               with cost, schedule, resources, and the                           management plan. Ancillary risks that affect project
               technical aspects of the project.                                 execution, and plans for mitigating those risks, are
                                                                                 documented in the weekly reports.
Activity 5     The project team tracks project issues,      Yes                  According to EOUSA officials, the project team tracks
               status, execution, funding, and expenditures                      project issues, status, execution, funding, and
               against project plans and takes action.                           expenditures against project plans and takes action.
Activity 6     The project team implements a corrective         Yes              The project team identifies, records, and tracks issues
               action system for the identification,                             using Rational’s ClearQuest product. These data are
               recording, tracking, and correction of                            then used to correct problems discovered during the
               problems discovered during the software                           software acquisition. The team is moving toward using
               acquisition.                                                      Merant’s PVCS Dimensions software.




                                             Page 38                                                  GAO-03-751 EOUSA IT Management
                                                                 Appendix II
                                                                 Assessment of ECMS Acquisition Practices
                                                                 against Level 2 of SEI’s Software Acquisition
                                                                 Capability Maturity Model




(Continued From Previous Page)
Common
feature                   CMM key practice                                         Satisfied?        Comments
Activity 7                The project team keeps its plans current      Yes                          The project team updates its plans during the life of
                          during the life of the project as re-planning                              the project as re-planning occurs, issues are resolved,
                          occurs, issues are resolved, requirements                                  requirements are changed, and new risks are
                          are changed, and new risks are discovered.                                 discovered.

Measurement 1             Measurements are made and used to          Yes                             Measurements are made and used by the ECMS
                          determine the status of project management                                 project team to determine the status of project
                          activities and the resultant products.                                     management activities and the resultant products.
Verification 1            Project management activities are reviewed Yes                             Project management activities are reviewed by
                          by acquisition organization management on                                  acquisition organization management on a bi-weekly
                          a periodic basis.                                                          basis.
Verification 2            Project management activities are reviewed Yes                             Project management activities are reviewed by the
                          by the project manager on both a periodic                                  project manager on both a periodic and an
                          and an event-driven basis.                                                 event-driven basis.
Source: Key practice data from SEI; analysis and comments from GAO.




                                                                 Page 39                                                 GAO-03-751 EOUSA IT Management
                                             Appendix II
                                             Assessment of ECMS Acquisition Practices
                                             against Level 2 of SEI’s Software Acquisition
                                             Capability Maturity Model




Table 11: Contract Tracking and Oversight

Common
feature         CMM key practice                                Satisfied?       Comments
Commitment 1    The acquisition organization has a written     No                The acquisition organization does not have a written
                policy for the contract tracking and oversight                   policy for the contract tracking and oversight of the
                of the contracted software effort.                               contracted software effort.

Commitment 2    Responsibility for contract tracking and        Yes              Responsibility is designated to the project manager
                oversight is designated.                                         and the administrative contracting officer’s
                                                                                 representative.
Commitment 3    The project team includes contracting           Yes              These specialists include the operations staff, the
                specialists in the execution of the contract.                    contract officer’s technical representative, and
                                                                                 contracting and procurement staff.
Ability 1       A group that is responsible for managing        Yes              The project management team is responsible for
                contract tracking and oversight activities                       managing contract tracking and oversight activities.
                exists.
Ability 2       Adequate resources are provided for             Yes              According to EOUSA officials, adequate resources are
                contract tracking and oversight activities.                      provided for contract tracking and oversight activities.
Ability 3       Individuals performing contract tracking and Yes                 According to EOUSA officials, individuals performing
                oversight activities have experience or                          contract tracking and oversight activities have
                receive training.                                                experience or receive training.
Activity 1      The project team performs its activities in     Yes              The project team performs its activities in accordance
                accordance with its documented contract                          with its documented contract tracking and oversight
                tracking and oversight plans.                                    plans. Several reporting mechanisms are used to
                                                                                 monitor and control the contractor’s performance,
                                                                                 including sticking to the project schedule and reporting
                                                                                 any problems that are encountered.
Activity 2      The project team reviews required               Yes              The project team reviews required contractor software
                contractor software planning documents                           planning documents, which provide a basis for
                which, when satisfactory, are used to                            overseeing the contractor team’s software engineering
                oversee the contractor team’s software                           efforts.
                engineering effort.
Activity 3      The project team conducts periodic reviews Yes                   There are weekly meetings and monthly written status
                and interchanges with the contractor team.                       reports between the project team and the contractor.

Activity 4      The actual cost and schedule of the             Yes              Actual cost and schedule of the contractor’s software
                contractor’s software engineering effort are                     engineering effort are compared to the planned costs
                compared to planned schedules and                                and schedule, and issues are identified. Issues so far
                budgets, and issues are identified.                              are primarily related to the contractor’s staff getting
                                                                                 the security clearances needed to do the work.
Activity 5      The size, critical computer resources, and   Yes                 The contractor provides information about the size,
                technical activities associated with the                         critical computer resources, and technical activities to
                contractor team’s work products are tracked,                     the ECMS project team for tracking purposes and
                and issues are identified.                                       issue identification.




                                             Page 40                                                  GAO-03-751 EOUSA IT Management
                                                                 Appendix II
                                                                 Assessment of ECMS Acquisition Practices
                                                                 against Level 2 of SEI’s Software Acquisition
                                                                 Capability Maturity Model




(Continued From Previous Page)
Common
feature                   CMM key practice                                         Satisfied?        Comments
Activity 6                The project team reviews and tracks the                  Yes               The project team reviews and tracks the development
                          development of the software engineering                                    of the software engineering environment.
                          environment required to provide life-cycle
                          support for the acquired software, and
                          issues are identified.
Activity 7                Any issues found by the project team during Yes                            According to EOUSA officials, any issues found by the
                          contract tracking and oversight are recorded                               project team during contract tracking and oversight
                          in the appropriate corrective action system,                               are recorded in the appropriate corrective action
                          action is taken, and the issue is tracked to                               system, action is taken, and the issue is tracked to
                          closure.                                                                   closure.
Activity 8                The project team ensures that changes to        Yes                        The project team ensures that changes to the
                          the software-related contractual                                           software-related contractual requirements are
                          requirements are coordinated with all                                      coordinated with all affected groups and individuals,
                          affected groups and individuals, such as the                               including the administrative contracting officer’s
                          contracting official, contractor, and end user.                            representative and end users.

Measurement 1             Measurements are made and used to             Yes                          Measurements are made and used by the
                          determine the status of the contract tracking                              administrative contracting officer’s representative to
                          and oversight activities and resultant                                     determine the status of the contract tracking and
                          products.                                                                  oversight activities and resultant products.
Verification 1            Contract tracking and oversight activities are Yes                         The administrative contracting officer’s representative
                          reviewed by acquisition organization                                       reviews contract tracking and oversight activities on a
                          management on a periodic basis.                                            periodic basis.
Verification 2            Contract tracking and oversight activities are Yes                         The project manager, on both a periodic and an event-
                          reviewed by the project manager on both a                                  driven basis, reviews contract tracking and oversight
                          periodic and event-driven basis.                                           activities.
Source: Key practice data from SEI; analysis and comments from GAO.




                                                                 Page 41                                                  GAO-03-751 EOUSA IT Management
                                              Appendix II
                                              Assessment of ECMS Acquisition Practices
                                              against Level 2 of SEI’s Software Acquisition
                                              Capability Maturity Model




Table 12: Evaluation

Common
feature         CMM key practice                                Satisfied?       Comments
Commitment 1    The acquisition organization has a written      Yes              The acquisition organization has defined guidelines for
                policy for managing the evaluation of the                        testing and certifying ECMS.
                acquired software products and services.
Commitment 2    Responsibility for evaluation activities is     Yes               Responsibility for evaluation activities is designated to
                designated.                                                       the ECMS project team.
Ability 1       A group that is responsible for planning,       Yes               The ECMS project team is responsible for planning,
                managing, and performing evaluation                               managing, and performing evaluation activities for the
                activities for the project exists.                                project.
Ability 2       Adequate resources are provided for             Yes               According to EOUSA officials, the evaluation activities
                evaluation activities.                                            have been budgeted.
Ability 3       Individuals performing evaluation activities    Yes               Individuals performing evaluation activities have
                have experience or receive training.                              experience or receive training. The contractors were
                                                                                  required to submit resumes along with their proposals.
Ability 4       Members of the project team and groups          No                Because of ECMS’s stage in the life cycle (design
                supporting the software acquisition receive                       phase), this key practice has not yet been addressed.
                orientation on the objectives of the
                evaluation approach.
Activity 1      The project team performs its activities in     No                Because of ECMS’s stage in the life cycle (design
                accordance with its documented evaluation                         phase), this key practice has not yet been addressed
                plans.
Activity 2      The project’s evaluation requirements are       No                Because of ECMS’s stage in the life cycle (design
                developed in conjunction with the                                 phase), this key practice has not yet been addressed.
                development of the system or software
                technical requirements.
Activity 3      The project’s evaluation activities are         No                Because of ECMS’s stage in the life cycle (design
                planned to minimize duplication and take                          phase), this key practice has not yet been addressed.
                advantage of all evaluation results, where
                appropriate.
Activity 4      The project team appraises the contractor       Yes               The project team assesses the contractor team’s
                team’s performance over the full period of                        performance continuously.
                the contract for compliance with
                requirements.
Activity 5      Planned evaluations are performed on the        No                Because of ECMS’s stage in the life cycle (design
                evolving software products and services                           phase), this key practice has not yet been addressed.
                prior to acceptance and operational use.
Activity 6      Results of the evaluations are analyzed and No                    Because of ECMS’s stage in the life cycle (design
                compared with the contract’s requirements                         phase), this key practice has not yet been addressed.
                to establish an objective basis to support the
                decision to accept the products and services
                or to take further action.
Measurement 1   Measurements are made and used to               No                Because of ECMS’s stage in the life cycle (design
                determine the status of the evaluation                            phase), this key practice has not yet been addressed.
                activities and resultant products.




                                              Page 42                                                  GAO-03-751 EOUSA IT Management
                                                                 Appendix II
                                                                 Assessment of ECMS Acquisition Practices
                                                                 against Level 2 of SEI’s Software Acquisition
                                                                 Capability Maturity Model




(Continued From Previous Page)
Common
feature                   CMM key practice                                         Satisfied?       Comments
Verification 1            Evaluation activities are reviewed by                    No                Because of ECMS’s stage in the life cycle (design
                          acquisition organization management on a                                   phase), this key practice has not yet been addressed.
                          periodic basis.
Verification 2            Evaluation activities are reviewed by the                No                Because of ECMS’s stage in the life cycle (design
                          project manager on both a periodic and an                                  phase), this key practice has not yet been addressed.
                          event-driven basis.
Source: Key practice data from SEI; analysis and comments from GAO.




                                                                 Page 43                                                 GAO-03-751 EOUSA IT Management
Appendix III

Comments from the Department of Justice                         Appendx
                                                                      iI




Note: GAO comments
supplementing those in
the report text appear
at the end of this
appendix.




See comment 1.



See comment 2.




See comment 3.




                         Page 44   GAO-03-751 EOUSA IT Management
                 Appendix III
                 Comments from the Department of Justice




See comment 4.




See comment 5.


See comment 6.



See comment 7.


See comment 8.




                 Page 45                                   GAO-03-751 EOUSA IT Management
                  Appendix III
                  Comments from the Department of Justice




See comment 9.




See comment 10.




See comment 11.




                  Page 46                                   GAO-03-751 EOUSA IT Management
                  Appendix III
                  Comments from the Department of Justice




See comment 12.
See comment 13.
See comment 14.


See comment 15.




See comment 16.




See comment 17.




                  Page 47                                   GAO-03-751 EOUSA IT Management
                  Appendix III
                  Comments from the Department of Justice




See comment 18.


See comment 19.




See comment 20.


See comment 21.




See comment 22.


See comment 23.




See comment 24.




See comment 25.




                  Page 48                                   GAO-03-751 EOUSA IT Management
                  Appendix III
                  Comments from the Department of Justice




See comment 26.




See comment 27.




                  Page 49                                   GAO-03-751 EOUSA IT Management
               Appendix III
               Comments from the Department of Justice




               The following are GAO’s comments on the Department of Justice’s letter
               dated June 16, 2003.



GAO Comments   1. We disagree. Our position that institutionalization has not been a
                  priority is based on two facts that EOUSA did not dispute: (1) plans for
                  addressing the weaknesses cited in our report do not exist and (2)
                  limitations in resources to address these weaknesses were cited by
                  EOUSA officials as the reason why the weaknesses exist. If each of
                  these areas were an agency priority, then plans would be in place to
                  address the weaknesses, and resources to execute the plans would be
                  committed.

               2. We do not question EOUSA’s statement that it has made “tremendous
                  progress.” Our work focused on determining the extent to which
                  EOUSA currently satisfies key practices in the four IT management
                  disciplines. It did not include developing a baseline from which to
                  measure progress. To EOUSA’s credit, our review showed that the
                  office has satisfied many key practices in each discipline, and we have
                  noted this in our report.

               3. We agree and include both of these facts in our report.

               4. We disagree. EOUSA’s comments did not include any information to
                  refute our conclusion. Given that it did not have a plan for fully
                  implementing best practices for each discipline, and had not allocated
                  adequate resources to support such a plan, we have not modified our
                  conclusion.

               5. We do not question these statements about the position of EOUSA and
                  the USAOs relative to other Justice components. Such a comparison
                  was not part of the scope of our work.

               6. We disagree. EOUSA has not gained this maturity level. Rather,
                  according to EOUSA, the contractor that maintains its LIONS
                  application is certified as a level 3 software developer. In contrast, our
                  work focused on EOUSA’s capabilities as a software acquirer, and thus
                  addresses a different organization, discipline, and maturity model.

               7. See comment 1.




               Page 50                                        GAO-03-751 EOUSA IT Management
Appendix III
Comments from the Department of Justice




8. We do not question this statement because the position of EOUSA and
   the USAOs relative to other Justice components or other law
   enforcement entities was not part of the scope of our work.

9. As noted in our report, EOUSA satisfied about 80 percent of the
   elements of just stage 2 of the EA management framework. It has
   satisfied about 60 percent of the elements (12 out of 19) of the entire
   framework.

10. We have modified the report to reflect this comment.

11. We agree. However, according to GAO’s IT Investment Management
    Framework, to satisfy the proposal selection critical process, EOUSA
    would need to demonstrate the use of the criteria it has defined.
    Because it has not yet done so, it is not satisfying the critical process
    and thus has met two out of five elements of stage 2 of the framework.

12. We disagree. Our assessment is based on EOUSA’s satisfaction of key
    practices laid out in our executive guide for information security
    management.1 This assessment showed that EOUSA has not fully
    satisfied any of these key practices. For example, EOUSA does not
    (1) have a central security focal point with appropriate resources,
    (2) adequately promote user awareness, and (3) regularly monitor the
    effectiveness of security controls. Until EOUSA addresses these and
    other security weaknesses we identify in our report, it will not have
    implemented effective security practices.

13. See comment 8.

14. We do not question this statement because determining whether the
    data of the United States Attorneys have never been compromised and
    are monitored 24 hours a day, 7 days a week was not within the scope
    of our work and EOUSA did not provide any evidence supporting its
    assertions.

15. See comment 1. Additionally, our finding is that the
    institutionalization of information security management has not been
    an agency priority.


1
  U.S. General Accounting Office, Information Security Management: Learning from
Leading Organizations, GAO/AIMD-98-68 (Washington, D.C.: May 1998).




Page 51                                             GAO-03-751 EOUSA IT Management
Appendix III
Comments from the Department of Justice




16. We do not question these security initiatives. Additionally, we
    emphasize that our message is not that EOUSA has not taken steps to
    improve its information security posture, but rather that the office’s
    information security management efforts, including ongoing and
    completed improvement steps, are weak in a number of areas relative
    to information security management best practices.

17. We agree, but would add that our recommendation could be addressed
    by actively monitoring activity at the routers, firewalls, and wide area
    network devices, which we understand are remotely managed by
    EOUSA. To avoid any potential confusion on this point, we have
    clarified our recommendation. Implementing an intrusion detection
    system to monitor activity at the routers, firewalls, and other network
    devices would enable EOUSA to detect hostile attempts to manage
    those devices.

18. We do not question EOUSA’s statement that it has been working to
    resolve vulnerabilities identified during a security audit conducted by
    the Justice Inspector General. The scope of the Inspector General’s
    audit, however, was narrower than ours in that it focused on EOUSA’s
    local area network environment.

19. We agree that given EOUSA’s recent progress in deploying the
    replacement network its exposure to risk is currently limited. We have
    modified the security risk assessment section of the report and the
    associated recommendation to reflect this change in circumstances.

20. We agree and thus do not conclude that EOUSA’s risk assessment
    program is inadequate. Rather, based on the fact that a risk assessment
    was not performed on the network that EOUSA has operated since
    1996 and, until recently, relied exclusively on, we conclude that EOUSA
    has not always performed risk assessments. Additionally, to recognize
    the recent change in circumstances we have modified our
    recommendation concerning risk assessments.

21. We do not question these statements. We support the use of automated
    tools to review audit logs, particularly because these logs were not
    being reviewed, and EOUSA attributed this to a lack of resources. We
    also support EOUSA’s plan to conduct regular tests to determine
    compliance with policies and procedures. Both of these planned
    actions are consistent with our recommendations.




Page 52                                       GAO-03-751 EOUSA IT Management
Appendix III
Comments from the Department of Justice




22. We do not question this statement. However, as noted in our report, the
    office did not have a plan to address the issues that are discussed in our
    report.

23. We do not question these statements because our review did not
    address contingency plans for all certified and accredited systems. As
    stated in the report, while a contingency plan was developed for the
    replacement network, it was not prepared in accordance with federal
    guidelines. For example, the plan did not specify procedures for
    notifying recovery personnel. To clarify our position, we have added
    examples to the report of this plan’s noncompliance with federal
    guidelines.

24. We support EOUSA’s stated commitment to establish a centralized
    security training program. Establishing such a program is consistent
    with our recommendations.

25. We have modified the report to reflect that the Enterprise Case
    Management System is the first acquisition to follow the Justice life-
    cycle methodology from its inception.

26. See comment 6.

27. We have modified the report to reflect that EOUSA’s acquisitions are
    processed through the department and must comply with all
    departmental policies and procedures.




Page 53                                        GAO-03-751 EOUSA IT Management
Appendix IV

GAO Contact and Staff Acknowledgments                                                      Appendx
                                                                                                 iIV




GAO Contact       Lester P. Diamond, (202) 512-7957



Acknowledgments   In addition to the individual named above, Nabajyoti Barkakati, Jamey
                  Collins, Joanne Fiorino, Anh Q. Le, Sabine R. Paul, and William F.
                  Wadsworth made key contributions to this report.




(310257)          Page 54                                     GAO-03-751 EOUSA IT Management
GAO’s Mission            The General Accounting Office, the audit, evaluation and investigative arm of
                         Congress, exists to support Congress in meeting its constitutional responsibilities
                         and to help improve the performance and accountability of the federal government
                         for the American people. GAO examines the use of public funds; evaluates federal
                         programs and policies; and provides analyses, recommendations, and other
                         assistance to help Congress make informed oversight, policy, and funding
                         decisions. GAO’s commitment to good government is reflected in its core values of
                         accountability, integrity, and reliability.


Obtaining Copies of      The fastest and easiest way to obtain copies of GAO documents at no cost is
                         through the Internet. GAO’s Web site (www.gao.gov) contains abstracts and full-
GAO Reports and          text files of current reports and testimony and an expanding archive of older
                         products. The Web site features a search engine to help you locate documents
Testimony                using key words and phrases. You can print these documents in their entirety,
                         including charts and other graphics.
                         Each day, GAO issues a list of newly released reports, testimony, and
                         correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site
                         daily. The list contains links to the full-text document files. To have GAO e-mail this
                         list to you every afternoon, go to www.gao.gov and select “Subscribe to
                         e-mail alerts” under the “Order GAO Products” heading.


Order by Mail or Phone   The first copy of each printed report is free. Additional copies are $2 each. A check
                         or money order should be made out to the Superintendent of Documents. GAO
                         also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a single
                         address are discounted 25 percent. Orders should be sent to:
                         U.S. General Accounting Office
                         441 G Street NW, Room LM
                         Washington, D.C. 20548
                         To order by Phone:     Voice: (202) 512-6000
                                                TDD: (202) 512-2537
                                                Fax: (202) 512-6061


To Report Fraud,         Contact:
                         Web site: www.gao.gov/fraudnet/fraudnet.htm
Waste, and Abuse in      E-mail: fraudnet@gao.gov
Federal Programs         Automated answering system: (800) 424-5454 or (202) 512-7470



Public Affairs           Jeff Nelligan, Managing Director, NelliganJ@gao.gov (202) 512-4800
                         U.S. General Accounting Office, 441 G Street NW, Room 7149
                         Washington, D.C. 20548
United States                  Presorted Standard
General Accounting Office      Postage & Fees Paid
Washington, D.C. 20548-0001           GAO
                                 Permit No. GI00
Official Business
Penalty for Private Use $300
Address Service Requested