oversight

Assessing Risks and Returns: A Guide for Evaluating Federal Agencies' IT Investment Decision-making

Published by the Government Accountability Office on 1997-02-03.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                                 United States General Accounting Office
_________________________________________________________________________________
                                 Accounting and Information Management
GAO                              Division



_________________________________________________________________________________
February 1997                    Assessing Risks and Returns:
Version 1
                                 A Guide for Evaluating
                                 Federal Agencies' IT
                                 Investment Decision-making




_________________________________________________________________________________
GAO/AIMD-10.1.13
                                               PREFACE

Investments in information technology (IT) can have a dramatic impact on an organization's
performance. Well-managed IT investments that are carefully selected and focused on
meeting mission needs can propel an organization forward, dramatically improving
performance while reducing costs. Likewise, poor investments, those that are inadequately
justified or whose costs, risks, and benefits are poorly managed, can hinder and even restrict
an organization's performance.

Despite making a huge investment in IT, many government operations are still hampered by
inaccurate data and inadequate systems. Too often, federal IT projects have cost too much,
produced too little, and failed to significantly improve mission performance. Yet there is also
general agreement that the government's ability to improve its service and performance will
depend heavily upon how well IT can be integrated into fundamental business/mission needs.
Outdated computer systems must be replaced; inefficient, paper-oriented processes must be
automated; accurate financial data must be developed and maintained; and an ever-increasing
amount of information must be stored and managed.

Several recent management reforms, including revisions to the Paperwork Reduction Act
(PRA), the Clinger-Cohen Act,1 the Government Performance and Results Act (GPRA), and
the Chief Financial Officers (CFO) Act, have introduced requirements emphasizing the need
for federal agencies to significantly improve their management processes, including how they
select and manage IT resources. For instance, a key goal of the Clinger-Cohen Act is that
agencies should have processes and information in place to help ensure that IT projects are
being implemented at acceptable costs, within reasonable and expected time frames, and are
contributing to tangible, observable improvements in mission performance. Moreover, these
agency processes should be institutionalized throughout the organization, and should be used
for all IT-related decisions. The ultimate goal of these various legislative reforms is for
agencies to make better decisions that will measurably increase the performance of the
organization.

PURPOSE AND USE OF THIS GUIDE

This evaluation guide was developed to provide a structure for evaluating and assessing how
well a federal agency is selecting and managing its IT resources and to identify specific areas
where improvements can be made. To accomplish this, the guide focuses on assessing an
organization from three levels:




1
 The fiscal year 1997 Omnibus Consolidated Appropriations Act, Pub. L. 104-208, renamed both Division D
(the Federal Acquisition Reform Act) and E (the Information Technology Management Reform Act) of the
1996 DOD Authorization Act, Pub. L. 104-106, as the "Clinger-Cohen Act of 1996."

                                                     1
•   the processes that the organization is using to select, manage, and evaluate its
    investments in information technology,
•   the data (cost, benefit, and risk) that are being used to make IT decisions, and
•   the IT decisions that are being made using the defined processes and data.

It may be unusual, especially early on, to find an agency that meets all of the requirements
outlined in this guide. Managing IT projects and systems as investments is a relatively new
focus within the government. But while an entire IT investment management structure may
not be in place, evidence of certain aspects of the investment management process may be
found, although these pieces may be incomplete or conducted on an infrequent basis. To
take this gradation into account, we are proposing the use of a reporting method that
highlights both positives and negatives in an agency's investment decision-making, and that
also targets specific critical areas that should be improved. An example of this reporting
structure, based on a previously completed evaluation of one organization's IT investment
decision-making, is provided in appendix 1.

In addition, while the overall evaluation approach of this guide is focused on evaluating
enterprise or agencywide processes and decisions, information and questions in the guide can
also be tailored for reviews of individual IT projects or systems development activities. And
the guide can be used for evaluations of past activities (after-the-fact audits) or assessments
of current policies. Users should determine what information is needed based on the type of
evaluation that is being conducted, and then adapt the criteria and questions to meet their
specific needs.

HOW THIS GUIDE WAS DEVELOPED

This guide was developed based on information we have gained while analyzing the
management practices of several leading private and public sector organizations.2 We have
since updated this information to ensure that it remains up-to-date, and we have examined
the IT investment decision-making processes of additional private, state, and local
organizations. In addition, to determine how well these practices translated to the federal
government, we evaluated the IT investment management practices of a small group of
federal agencies and compared these practices to those of leading organizations.3




2
 Executive Guide: Improving Mission Performance Through Strategic Information Management and
Technology (GAO/AIMD-94-115, May 1994).

3
 Information Technology Investment: Agencies Can Improve Performance, Reduce Costs, and Minimize
Risks (GAO/AIMD-96-64, Sept. 30, 1996).

                                                   2
We also assisted the Office of Management and Budget (OMB) in its development of a guide
for federal agencies to use to evaluate IT investments.4 The information provided in this
guide is consistent with the IT investment approach outlined in OMB's document and
provides additional discussion of particular topics. Finally, many of the concepts outlined in
this guide have been incorporated into OMB's Capital Programming Guide, which provides
guidance on the planning, budgeting, acquisition, and management of different kinds of
capital assets, including information technology.5

The decision-making approach for information technology outlined in this guide may be
incorporated into federal agencies' capital planning processes in many different ways. For
instance, the approach may (a) be incorporated into an overall capital planning process, such
as that outlined by OMB in the Capital Programming Guide, (b) provide input on IT-related
projects to a more comprehensive capital planning process, or (c) be used for technology
decision-making in the absence of an overall capital programming and planning process.
Obviously, federal agencies will be at different stages of maturity and implementation of their
overall capital programming processes. Over time, we would expect greater convergence
between these capital processes and the IT investment decision-making process discussed in
this document.

CAVEATS TO THE USER

(1) This guide incorporates provisions from major federal legislation and executive branch
guidance addressing IT investment decision-making. This guide conforms to requirements
contained in several pieces of legislation, including PRA, the Clinger-Cohen Act, GPRA, the
CFO Act, and the Federal Acquisition Streamlining Act (FASA), as well as requirements found
in a number of key OMB circulars, bulletins, and policy documents. However, the IT
investment approach described in this guide is a complex process and agencies have been
given a great deal of latitude in how they go about selecting and managing their IT resources.
But while specific steps may differ from agency to agency, key fundamental components of
an investment decision-making process should be in place at every agency.

(2) This guide is intended to be a flexible evaluation approach that is applicable
governmentwide. When evaluating an organization's6 IT decision-making, it is important to
take into account the organization's operating environment, as well as its goals and missions.


4
 Evaluating Information Technology Investments, A Practical Guide, Executive Office of the President,
Office of Management and Budget, November 1995.

5
Capital Programming Guide (Exposure Draft, Dec. 10, 1996), Executive Office of the President, Office of
Management and Budget.

6
 Throughout this guide we use the term "organization" to refer to the entity being used as the unit of
analysis (i.e., department, agency, bureau, office, etc.). To ensure that all related processes are included, it
will be important that the scope of the evaluation be clearly identified.

                                                          3
An organization with all IT functions and services centrally organized may implement
decision processes differently from an organization with a highly decentralized IT structure.
Likewise, an organization with a primary research and development mission may go about
evaluating IT differently from an agency focused exclusively on program delivery.

Because of the wide variance among organizations and the complexity of the investment
management process, this guide introduces a flexible, multitiered evaluation approach that
allows for organizational diversity and flexibility. However, while accounting for variation
and differences, there are still common elements that should be present in any organization's
IT investment management process. This guide focuses on these common elements.

STRUCTURE OF THIS GUIDE

This guide is laid out in four sections. Section 1 gives an overview of the IT investment
management process. Section 2 provides an in-depth explanation of the evaluation
framework that this guide proposes for evaluating an agency's IT investment decision-making,
as well as assumptions about what the organization has already accomplished. Section 3
provides the specific criteria and evaluation questions that can be used to help guide your
review. Finally, Section 4 lists all relevant legislation and executive branch policy documents
that are associated with the IT investment management process.

CONTACTS

This guide was developed by the Information Resources Management Policies and Issues
Group under the direction of Christopher W. Hoenig, Director, Information Management and
Technology Issues. If you have any questions about this guide or the IT investment
management approach outlined here, please contact Dave McClure, Senior Assistant Director,
at (202) 512-6257 (mcclured.aimd@gao.gov), Shane Hartzler, Business Process Analyst, at
(202) 512-6296 (hartzlers.aimd@gao.gov), or John Rehberger, Information Systems Analyst, at
(202) 512-3687 (rehbergerj.aimd@gao.gov).

An electronic version of this guide is available from GAO's World-Wide Web server at the
following Internet address: http://www.gao.gov/policy/itguide/index.htm. The automated
version is also available under the "Tools" section of AIMD's Pathways intranet at
http://gaoweb/issues/pathways.aimd/aimd.htm.




Gene L. Dodaro                                              Brian P. Crowley
Assistant Comptroller General                               Assistant Comptroller General
Accounting and Information Management                        for Policy
 Division

                                              4
                              TABLE OF CONTENTS


Preface                                                1

  Purpose and Use of This Guide                        1
  How This Guide Was Developed                         2
  Caveats to the User                                  3
  Structure of This Guide                              4
  Contacts                                             4

Table of Contents                                     5

Section 1: Overview of the IT Investment              6
Management Process

  Critical Success Factors                            10
  Legislative Requirements                            11

Section 2: Framework for Assessing Agencies' IT       13
Investment Decision-making

  Assumptions About Related Management Decisions      18
    and Processes
  Linkages to IT Performance Management               20
  Potential Information Sources                       20

Section 3: Criteria and Questions                     22

Section 4: Related Legislation and Executive Branch   82
Guidance

  Legislation                                         82
  Executive Branch Guidance                           89

Appendix I: Proposed Structure for Presenting         94
Results

Appendix II: Example of One Organization's Criteria   98
for Comparing and Ranking Projects

Glossary                                              104




                                           5
                                          SECTION 1

                              OVERVIEW OF THE IT INVESTMENT
                                  MANAGEMENT PROCESS

An IT investment management process is an integrated approach to managing IT investments
that provides for the continuous identification, selection, control, life-cycle management, and
evaluation of IT investments. This structured process provides a systematic method for
agencies to minimize risks while maximizing the return of IT investments.

To be most successful, an IT investment management process should have elements of three
essential phases--select, control, and evaluate. However, each phase should not be viewed as
a separate step. Rather, each is conducted as part of a continual, interdependent
management effort. Information from one phase is used to support activities in the other
two phases. The following figure illustrates the three phases of an IT investment
management process and the relationships between the various phases.

Figure 1: Fundamental Phases in the IT Investment Management Process




                                         Control
                                      What are you doing to
                                      ensure that the projects
                                      will deliver the
                                      benefits projected?


            Select
         How do you know
         you have selected
         the best projects?



                                          Evaluate
                                         Based on your
                                         evaluation, did the
                                         systems deliver
                                         what you expected?
                                                                 Process


                                                                 Information




                                                  6
An analysis of the existing portfolio of IT investments, commonly done as part of a systems
or applications disposition process, helps to ensure that senior managers are informed of
current costs, benefits, and risks associated with the exisiting portfolio. This disposition
analysis and planning process also helps ensure that an accurate systems and applications
inventory is maintained and it forms the basis for a systems retirement and replacement
strategy. Such a strategy can provide a solid foundation for keeping, stopping, transforming,
or replacing information systems and technology. In addition, the disposition analysis can
assist in technical infrastructure planning and helps to ensure the integrity of the
organization's information technology architecture.

The IT investment management process begins with the Selection phase. In this phase, the
organization determines priorities and makes decisions about which projects will be funded
during the year.7 A starting point for the Selection phase is the screening process, in which
projects being submitted for funding are compared against a uniform set of screening criteria
and thresholds in order to determine whether the projects meet minimal requirements and to
identify at what organizational level the projects should be reviewed. The costs, benefits,
and risks of all IT projects--proposed, under development, operational, etc.--are then assessed
(usually by an independent group) and the projects are compared against each other and
ranked or prioritized. As part of this process, weighting factors may be attached to the
ranking criteria. These ranking criteria should, at a minimum, include cost, risk, and benefit
factors, as well as an assessment of how well the project meets mission needs. Finally, a
senior management decision-making body makes decisions about which projects to select for
funding based on mission needs and organizational priorities. The systems and projects that
are selected for funding make up the portfolio of IT investments.8

The Selection phase helps ensure that the organization (1) selects those IT projects that will
best support mission needs and (2) identifies and analyzes a project's risks and proposed
benefits before a significant amount of project funds are spent. A critical aspect of this
phase is management understanding and participation and decision-making that is driven by
accurate, up-to-date data and an emphasis on using IT to enhance mission performance.

Once selected, all of the projects in the portfolio are consistently controlled and managed.
Progress reviews, in which the progress of projects are compared against projected cost,
schedule, and expected mission benefits, are conducted at key milestones in each project's
life cycle. The type and frequency of these reviews are usually determined based on the
analyses of risk, complexity, and cost that went into selecting the project. If a project is late,


7
 The Selection phase can be conducted quarterly, annually, or to coincide with whatever time frame is used
by the organization itself or, as is the case in the public sector, what is mandated by law.

8
 The senior management decision-making body may or may not have "final" approval of the funding request.
In most cases, the portfolio of IT investments selected by the decision-making body will be forwarded to
the agency head for final approval and inclusion in the agency budget request. In addition, OMB reviews
and congressional budget decisions ultimately serve as final approval steps.

                                                      7
over cost, or not meeting performance expectations, senior executives decide whether it
should be continued, modified, or canceled, and actions are quickly taken to mitigate the
effects of changes in risks and costs.

The Control phase helps ensure that as a project is developed and investment costs rise, that
the project continues to meet mission needs, and if it does not or if problems have arisen,
mitigating steps are quickly taken to address the deficiencies. Decisions made at the Control
phase may include cancelling the project, modifying it to better meet mission requirements,
accelerating development of the project, or continuing its development as planned.

Finally, once projects have been fully implemented, actual versus expected results are
evaluated to (1) assess the project's impact on mission performance, (2) identify any changes
or modifications to the project that may be needed, and (3) revise the investment
management processes based on lessons that were learned.

This IT investment decision-making approach is a fluid and dynamic process. Figure 2
illustrates how this process can work when IT spending for all projects--new proposals and
ongoing projects--is decided each year as part of an annual budget process. Both proposed
and ongoing projects enter in to an IT investment disposition analysis, which examines the
existing inventory of systems and applications to review existing costs, benefits, and risks
associated with all IT investments. Selection decisions are made based on an analysis of
where needs are greatest and in line with the organization's systems retirement and
replacement plans and implementation strategy. Projects that are terminated or delayed as
part of selection decisions are evaluated immediately to allow the organization to assess the
impact on future proposals and to quickly benefit from lessons that are learned.

Figure 2 also illustrates how projects selected as part of an IT portfolio of investments enter
in to an investment control process for the remainder of the year. Investment control
meetings are conducted on a regular basis throughout the year. These meetings may
coincide with episodic project events that automatically trigger a management review (i.e.,
deviations in cost, schedule or performance outside of accepted thresholds) or with critical
life-cycle milestones. Post-implementation reviews (PIRs) may also be conducted for
projects that are completed or canceled during the year. The results of these control
meetings and post-implementation reviews provide input into the following year's Selection
process.




                                               8
Figure 2: Example of an Annual IT Investment Decision-making Process
                                                    Feedback



                                                               Evaluation of projects
                                                                stopped or delayed


                            IT
                       investment
                                         Analysis and               SELECTION
         New IT        disposition
                                        ranking of all IT
         proposals     analysis &                                   of IT portfolio
                                           projects
                        planning


                                                                      CONTROL
                                                               of on-going projects via
                                      On-going                    reviews based on
                                                                critical milestones or
                                      IT projects
                                                               conditions that "trigger"
                                                                      immediate
                                                               management attention




                                                                    EVALUATION             Feedback
                                                                      of projects
                           Feedback                                  implemented
                                                                      or canceled




The selection, control, and evaluation decision-making processes described above can be
applied to almost any organization, even one that is highly decentralized. In the federal
government, for instance, major cabinet departments often have several agencies under their
purview. Separate IT investment decision-making processes can exist at both the
departmental and agency levels, provided that the organization can identify which IT projects
and resources are shared (and should be reviewed at the departmental level) and which are
unique to each individual agency. See figure 3 for an illustration of how decision-making can
be applied at various organizational levels.




                                                     9
Figure 3: Levels of IT Investment Decision-making

    S     Select                             Corporate decisions
                                                                           IT Project Portfolios

                                                                          Change in Decision-making
    C     Control
                                                         C                Authority

                                                                          Projects that are:
    E     Evaluate
                                                                           -- high-dollar amount or
                                       S                                      high-risk;
                                                                           -- cross-departmental; or
                                                                           -- common infrastructure
                                                           E                  projects.




         Business unit 1 decisions     Business unit 2 decisions    Business unit 3 decisions



                          C                              C                              C
           S                             S                            S
                            E                             E                              E




Criteria for determining projects that should bump up to the highest review level may include
(1) high-dollar, high-risk projects (the risk and dollar thresholds for identifying these projects
should be predetermined by the organization), (2) cross-functional projects (two or more
organizational units will benefit from the project), or (3) common infrastructure support (e.g.,
hardware and telecommunications). Projects that meet these particular threshold criteria are
strong candidates for discussion, review, and decisions at a departmentwide level.

Determining where in the organization a project should be reviewed is something agencies
have been given freedom and flexibility to decide. The key to making good decisions is
having clearly defined roles, responsibilities, and criteria for determining the types of projects
that will be reviewed at the different organizational levels.

CRITICAL SUCCESS FACTORS

To be successful, an agency's IT investment management processes should generally include
the following elements:

•       Key organizational decisionmakers are committed to the process and are involved
        throughout each project's life cycle.

                                                        10
    · Projects are assessed jointly by program, financial, and IT managers.
•   The investment management process is repeatable, efficient, and conducted uniformly and
    completely across the organization.
    · The process includes provisions for continually selecting, managing, and evaluating
       projects in the investment portfolio.
•   Decisions are made consistently throughout the organization.
    · Decisions at any level of the organization are made using uniform decision criteria.
    · Decisions are driven by accurate and up-to-date cost, risk, and benefit information.
    · Decisions are made from an overall mission focus (there is an explicit link with the
       goals and objectives established in the organization's strategic plan or annual
       performance plans and with the organization's information technology architecture).
•   Accountability and learning from previous projects is reinforced.
•   The emphasis is on optimizing the portfolio mix in order to manage risk and maximize
    the rate of return.
•   The process incorporate all IT investments, but recognizes and allows for differences
    between various project types (mission critical, administrative, infrastructure) and phases
    (new, under development, operational, etc.).

LEGISLATIVE REQUIREMENTS

In recent years, several pieces of legislation have been passed that have required federal
agencies to examine and change their current operations and management practices in order
to improve performance and achieve greater mission outcomes. The following is a brief
description of the five major management reforms.

The Paperwork Reduction Act of 1995 (PRA)
(Public Law 104-13)

PRA requires agencies to use information resources to improve the efficiency and
effectiveness of their operations and fulfillment of their missions. It emphasizes achieving
program benefits and meeting agency goals through the effective use of IT. As such, it is
the "umbrella" IT legislation for the federal government with other statutes elaborating
on the goals contained within PRA.

The Clinger-Cohen Act of 1996 (formally the Information Technology Management
Reform Act, Division E of Public Law 104-106)

The Clinger-Cohen Act requires federal agencies to focus on the results they are achieving
through IT investments. Specifically, the act introduces much more rigor and structure into
how agencies approach the selection and management of IT projects. Among other things,
the head of each agency is required to implement a process for maximizing the value and
assessing and managing the risks of the agency's IT acquisitions.




                                              11
Government Performance and Results Act of 1993 (GPRA)
(Public Law 103-62)

GPRA requires agencies to set goals, measure performance, and report on their
accomplishments. A key tenant of GPRA is that agencies will develop strategic plans, as
well as annual performance plans that are linked to the strategic plans, that establish the
organization's goals and objectives as well as strategies for achieving these goals. With these
plans in place, an agency can begin to assess whether its activities, core processes, and
resources are aligned to support its mission and achieve desired outcomes. GPRA also
requires agencies to establish performance measures and benchmarks in order to begin
identifying gaps between actual and desired performance levels and mission outcomes.

The Federal Acquisition Streamlining Act of 1994 (FASA)
(Public Law 103-355)

Title V of FASA requires agencies to define cost, schedule, and performance goals
for federal acquisition programs (including IT projects) and to monitor these
programs to ensure that they remain within prescribed tolerances. If a program falls out of
tolerance (failure to meet 90 percent of cost, schedule, and performance goals), FASA gives
the agency head the authority to review, and if necessary terminate, the program.

Chief Financial Officers Act of 1990 (CFO)
(Public Law 101-576)

The CFO Act focuses on the need to significantly improve the financial management and
reporting practices of the federal government. Having accurate financial data is critical to
understanding the costs and assessing the returns on IT investments.




                                              12
                                         SECTION 2

                      FRAMEWORK FOR ASSESSING AGENCIES'
                        IT INVESTMENT DECISION-MAKING

As agencies begin taking actions to improve their IT investment decision-making to conform
with the various legislative requirements, their focus should be on three particular areas that
will have direct bearing on whether real, positive change is produced. Specifically, agency
heads need to (1) institutionalize management processes, (2) regularly validate cost, benefit,
and risk data used to support IT investment decisions, and (3) focus on measuring and
evaluating results.

Informed management decisions can only occur if accurate, reliable, and up-to-date
information is included in the decision-making process. Project cost data must be tracked
and easily accessible. Benefits must be defined and quantitatively and qualitatively measured
in outcome-oriented terms. And risks must be quantified and mitigated to better ensure
project success. Early signs of success will be examples where the contributions of IT to
gains in productivity, reductions in cost and cycle-time, and increases in service delivery
quality and satisfaction are quantitatively documented and independently reported.

Agencies must also have hard numbers and facts on what was spent on IT and what the
agency achieved with the investment. These evaluations, wherever possible, must focus on
the measurable contribution that IT is making towards improving mission performance.
Early signs of success will be examples of measurable impact or where IT projects with
questionable results were cancelled or delayed as a result of IT investment control processes.

In evaluating an agency's investment management practices and decisions and assessing how
well they are responding to these critical areas, three sets of questions should be answered:

•   Does the agency have decision-making and management processes in place to select IT
    projects and systems, control and monitor these projects throughout their life cycle, and
    evaluate results and revise the processes based on lessons learned?
•   Are IT decisions being driven by cost, risk, and benefit information, and is this
    information accurate and up-to-date? Are project justifications updated as funding is
    spent and interim benefits assessed?
•   And most important, is the organization making decisions that maximize benefits while
    minimizing risks? Is the organization ensuring that the projects being funded are meeting
    the most critical needs of the organization? Are the selected projects being monitored
    and controlled, and are actions being taken to address problems as they are identified or
    as requirements change?

We designed this assessment guide to help evaluators determine the answers to these
questions, and in doing so, begin to offer tangible advice and targeted recommendations on
how federal agencies can better select, control, and evaluate their IT projects. This guide is

                                              13
structured so that evaluators will assess the Selection, Control, and Evaluation phases from
three review levels--process, data, and decisions. See figure 4 for an illustration of the IT
investment evaluation model. A short description and examples of each review level are also
provided below.

Figure 4: Dimensions of the IT Investment Evaluation Approach

                                             Repeatability
                                              Efficiency
                                             Completeness

          Pr                Select                  Control                Evaluate
            oc
              es
                s



           Da
             ta



          De
            cis
               io
                 ns




Process

This is an assessment of the investment management processes9 that the organization is
following to select IT investments, control and monitor progress of these investments, and
evaluate final results. The central question to be answered is: "Does the organization have
defined, documented processes for selecting, controlling, and evaluating its IT
investments?" The goal in assessing an agency's processes is to identify to what extent the
organization has a structure in place for managing and evaluating IT investments.

•   There should be documented evidence (in guidance or policy) that an IT investment
    management process is in place (consisting of selection, control, and evaluation pieces),
    that it is repeatable and implemented consistently throughout the organization, and that
    decision-making roles, responsibilities, and authority have been clearly defined.


9
Process is assumed to include requisite policies, practices, and procedures that are used to manage and
make decisions about IT investments.

                                                      14
An important point to remember when making an assessment of existing processes is that
the evaluation should be focused solely on the organization's policies, practices, and
procedures, not on actual decisions. Having institutionalized management processes, honed
to work in the culture of the organization, is critical to producing consistently good results.
The investment processes should accurately reflect the way the organization actually
functions and makes decisions.

Data

An IT investment process cannot operate without accurate, reliable, and up-to-date data on
project costs, benefits, and risks. It is the basis for informed decision-making. In addition,
documentation of management decisions are essential to begin to assemble a track record of
results. Evaluating the data involved in the IT investment management process requires
evaluating two different types of data:

ex ante - the information that is being used as inputs to the IT investment process (e.g., the
cost/benefit/risk analyses that are used to justify the selection and continued funding of
projects, the performance measures that are used to monitor a project's progress, etc.).

ex post - information that is produced based on decisions that are made (e.g., project review
schedules and risk mitigation plans should be developed once a decision is made to fund a
project).

•   All projects (proposed, under development, operational, etc.) should have complete and
    accurate project information--cost and benefit data, risk assessments, links to
    business/program goals and objectives, and performance measures, as well as up-to-date
    project-specific data, including current costs, implementation plans, staffing plans, and
    performance levels. In addition, the organization should have qualitative and quantitative
    project requirements and decision criteria in place to help screen IT projects, assess and
    rank projects, and control and evaluate the projects as they move through the various
    phases of their life cycle.

•   All management actions and decisions that are made should be documented and
    maintained. Moreover, some decisions require that additional information be produced.
    For instance, after a project is selected, project-specific review schedules and risk
    mitigation plans should be developed.

Decisions

One of the most important goals of this guide is enabling evaluators to assess the
effectiveness of the organization's IT investment process and the extent to which it is
contributing to the improved mission performance of the organization. After evaluating the
processes that the organization uses to select, control, and evaluate IT investments and the
data that are used to make decisions, evaluators will be in a much better position to reach

                                               15
conclusions about the specific decisions that the organization is making. The central focus
of analysis is on whether management decisions and actions are being taken using the
investment control processes and requisite project data.

•   The IT investment portfolio should represent a mixture of those projects that best meet
    the mission needs of the organization. Projects in the portfolio should be consistently
    monitored and decisions should be made at key milestones to ensure that the project is
    continuing to have its expected business or programmatic impact with a focus on
    minimizing risk and maximizing return. Completed projects are evaluated to compare
    actual performance levels to estimated levels and to feed lessons learned back in to the
    Selection and Control phases.

Three Critical Factors

In addition to these three phases, there are three critical attributes--repeatability,
efficiency, and completeness--that cut across each phase and that should be assessed at
each review level.

Repeatability focuses on the extent to which the processes, data, or decisions being
reviewed are conducted consistently over time and across different organizational units
(recognizing that processes should naturally evolve as lessons are learned and improvements
are made).

•   Projects are selected uniformly across more than one budget cycle, project reviews are
    conducted for all projects at established intervals, an evaluation methodology is in place
    and is used to assess all fully implemented projects.
•   The organization has identified all necessary information (cost/benefit/risk analyses,
    proposed schedule, user and business requirements, etc.) for making decisions, and this
    information is maintained, updated, and used to drive all project decisions. Specific,
    quantifiable decision criteria have been established and are used at all decision levels.
•   Projects are selected and managed based on established criteria or documented
    justifications.

Two essential aspects of repeatability are whether (1) roles, responsibilities, and authority
have been defined and documented and (2) uniform decision criteria are in place.

Efficiency focuses on how well management processes, the generation of project data, and
decision-making is working. The focus is on the overall quality (the accuracy, reliability, and
timeliness) of the investment approach. In addition, the same data generated to support IT
investment selection, control, and evaluation should be used to manage IT projects through
their life-cycle.




                                              16
•   All projects are subjected to a similar investment management process (consisting of
    Select, Control, and Evaluate phases) and this process is documented so that everyone
    knows the steps that are conducted and the analyses that are required.
•   Cost, risk, and benefit data, both qualitative and quantitative, are accurate and are
    updated as information is gained. Project information is readily available and an
    organizational track record is maintained. Project results and lessons learned are tracked
    and aggregated in order to further refine and improve decision-making.
•   Decisions are being made at the right level. Senior managers' limited time is being
    utilized to the best extent possible. Actions are quickly taken to address deficiencies.

Completeness focuses on the extent to which all phases of the process (Select, Control, and
Evaluate) are being followed and whether use of the IT investment process is
institutionalized across the organization. Traditionally, federal agencies have focused their
management efforts on selecting IT projects, but once selected, the projects are never
revisited and assessed to determine whether requirements or risks have changed and to
ensure that the project is continuing to meet mission needs.

As the evaluation is conducted and questions are asked, a concerted effort should be made to
keep these three critical factors in mind. Evaluators should continually ask themselves
whether the processes, data, or decisions that they are assessing are repeatable, efficient, and
complete.

Using this initial evaluation approach, we developed key elements for each of the nine areas
(i.e., Select--Process, Control--Process, Evaluate--Process, etc.). These key elements represent
the most critical factors that each organization should have in place. Figure 5 shows the IT
investment evaluation approach with the additional key elements.




                                              17
    Figure 5: The IT Investment Evaluation Approach With Key Elements



                                                    Repeatability
                                                      Efficiency
                                                    Completeness


Pr                     Select                                 Control                             Evaluate
  oc
    es        Selection processes include           Control processes include          Evaluation processes include
      s        screening projects                    consistently monitoring projects   conducting post-implementation
               analyzing and ranking all projects    involving the right people         reviews (PIRs) using a standard
               based on benefit, cost, and risk      documenting all major actions and methodology
               criteria                              decisions                          feeding lessons learned back in to the
               selecting a portfolio of projects     feeding lessons learned back in to Selection and Control phases
               establishing project review           the Selection phase
 Da            schedules
   ta         Selection data include               Control data include                 Evaluation data include
               evidence that each project has met measures of interim results            measurements of actual vs. projected
               project submission requirements      updated analyses of each project's performance
               analyses of each project's costs,    costs, benefits, schedule, and risks documented "track record" (project
               benefits, and risks                                                       and process)
               data on the existing portfolio
               scoring and prioritization outcomes
De
  cis          project review schedules
     io        Selection decisions include           Control decisions include           Evaluation decisions include
       ns
                determining whether projects met      deciding whether to cancel, modify, assessing the project's impact on
                process-stipulated requirements       continue, or accelerate a project   mission performance and determining
                deciding upon the mixture of          aggregating data and reviewing      future prospects for the project
                projects in the overall IT investment collective actions taken to date    revising the Selection and Control
                portfolio                                                                 phases based on lessons learned




    ASSUMPTIONS ABOUT RELATED MANAGEMENT
    DECISIONS AND PROCESSES

    In developing this guide, we made several assumptions about what the organization being
    reviewed has already accomplished. These assumptions correspond closely with OMB's
    policy guidance that requires agencies to determine, before committing resources to any new
    capital asset, whether (1) the asset will support functions that are mission critical, (2) any




                                                             18
other government or private entity can provide the service better, and (3) agency business
processes have first been reengineered to provide optimal performance at the lowest cost.10

First, because an essential factor in selecting and managing IT projects is ensuring that the
projects support organizational goals and objectives, it is critical that the organization have a
clear statement of its mission. Mission goals and objectives, as well as strategies for
accomplishing them, should be explained in the agency's strategic and annual performance
plans.11 In addition, the agency should have an established information technology
architecture12 that systems and projects are expected to follow.

Second, before making a significant investment in an IT project, managers should determine
the extent to which existing resources allow their programs to meet the goals and objectives
spelled out in the agency's strategic and annual performance plans. To help make this
determination, the agency should have an accurate inventory of its existing systems and
applications, including an identification of their related costs and organizational benefits.

Finally, a key hazard in acquiring information technology is that the new system will
automate outmoded, inefficient business processes. The Clinger-Cohen Act requires agency
heads to revise mission-related and administrative processes (as appropriate) before making
a significant investment in IT systems to support them. Thus, an assessment of current
processes (process mapping, baselining, benchmarking) should be completed before any
decision is made about acquiring technology. (For guidance on assessing how well an
organization is addressing key tasks and risks associated with business process re-
engineering, see GAO's forthcoming evaluation guide entitled Business Process Reengineering
Assessment Guide).




10
 See Policy Memorandum M-97-02, "Funding Information Systems Investments," October 25, 1996, Office of
Management and Budget; and "Capital Programming Guide," (Exposure Draft, December 10, 1996), Office of
Management and Budget.

11
  OMB memorandum 96-22, "Implementation of the Government Performance and Results Act of 1995"
(April 11, 1996) required agencies to submit to OMB by the fall of 1996 draft mission statements, general
long-term goals and objectives, and annual performance goals. In addition, GPRA requires agency heads to
submit completed strategic plans to OMB and the Congress by September 30, 1997, and to establish and
begin submitting annual performance plans by fiscal year 1999.

12
  Section 5125(d) of the Clinger-Cohen Act defines an information technology architecture as an integrated
framework for evolving or maintaining existing IT and acquiring new IT to achieve the agency's strategic
and information resources management goals. A complete IT architecture should consist of both logical
and technical components. The logical architecture provides the high-level description of the agency's
mission, functional requirements, information requirements, system components, and information flows
among the components. The technical architecture defines the specific IT standards and rules that will be
used to implement the logical architecture.

                                                      19
LINKAGES TO IT PERFORMANCE MANAGEMENT

A key requirement of GPRA is that federal agencies are to begin establishing performance
measures that identify the primary business outcomes on which the agency is focused on
achieving. In addition, the Clinger-Cohen Act specifically requires agencies to justify IT
investments in terms of improvements in agency operations and performance. Moreover,
agency managers are expected to use performance measures as the basis for continuing,
modifying, or terminating IT projects. Because the investments that an organization makes in
IT should be aligned with the business or mission results that it is trying to achieve, the IT
performance measures that are used to monitor progress should reflect this focus on results.
IT performance measures should be inherently tied to a project's expected benefits. A key
question for evaluators to ask is: "Are the expenditures on IT products and services
reasonable given the expected/actual improvements in mission performance?"

POTENTIAL INFORMATION SOURCES

When reviewing an agency's IT investment decision-making and management processes,
information will come from a wide variety of sources. These sources will include the
following:

Agency-generated policy documents

The organization should have documented policies or procedures that outline how various
aspects of the IT investment management process are to be conducted. These documents
may include the following:

•   Charters or operating procedures for decision-making bodies
•   Organizational charts
•   Project submission requirements
•   Methodologies (for conducting post-implementation reviews, cost/benefit analyses, risk
    analyses, cost estimation, systems development)
•   Operating policies and procedures (including concept of operations)
•   Cost and risk models
•   Criteria for justifying and making decisions about projects (both a listing of these criteria
    and definitions),
•   Return on investment (ROI) guidance/calculations (thresholds),
•   Risk assessment methods, tools, and guidance
•   Business case justification guidance
•   Strategic business/mission plans
•   Strategic IT/IRM plans
•   Information or systems architecture policy




                                               20
Reporting requirements

Federal agencies are required by various pieces of legislation and executive branch guidance
to submit information on agency plans and actions. Information required includes the
following:

•   Budget submissions (OMB Circular A-11: Section 43 exhibits, 300 A and C exhibits; OMB
    Policy Memo M-97-02 --IT Funding Requirements)
•   GPRA strategic plans
•   GPRA annual performance plans
•   GPRA annual program performance reports
•   Clinger-Cohen IT performance reports
•   PRA organizational, strategic, and IRM mission plans,
•   PRA IRM strategic and operational plans

Project-specific data

Finally, the development and management of IT projects requires project-specific information
to be generated and maintained. This information may include:

•   project plans and progress reports, including requirements documents,
    design/development schedules, a software development plan, a test plan, interface
    documents, a conversion plan, etc.
•   system design alternatives documentation
•   work and information flow diagrams
•   requirements analysis documentation
•   performance results/assessments (post-implementation reviews)
•   business case information

In addition, agency information may include documentation of management decisions and
actions, such as memos and minutes of high-level decision-making bodies. Agency budget
justification documents submitted to OMB and to congressional appropriations committees
may also contain valuable information.




                                             21
                                        SECTION 3

                              CRITERIA AND QUESTIONS


This section explains the common elements associated with each phase of an IT investment
decision-making approach. Each element is cross-indexed to applicable criteria contained in
federal laws and executive branch policies and guidance (abbreviations used for these laws
and policies, as well as brief summaries for each citation, are included in Section 4).

The evaluation questions provided in this section are intended to help guide evaluators in
their assessment by providing a minimum set of questions for each area; however, they are
not meant to be an exhaustive list. Depending on the unit of analysis for IT investment
decision-making--cabinet-level department, independent agency, a bureau within a department
or agency, or a single program office--evaluators may find that additional questions are still
needed.

The questions that are provided cover explicit criteria contained in legislation and executive
branch policy guidance and are linked to critical elements associated with each investment
decision-making phase. In many instances, though, existing legislative and policy criteria are
not specific in explaining how agencies are to implement requisite processes. This provides
agencies flexibility in how they implement their IT investment management processes.
However, regardless of the approach, common elements associated with this type of
decision-making framework should be in place. The questions focus on these common
elements.




                                              22
                                                                                                   Select   Control   Evaluate


                                                                                       Process



                                                                                         Data



                                                                                       Decisions




                                     Select--Process
  Key process elements to be reviewed:

  1.1   Screening projects
  1.2   Analyzing and ranking all projects based on benefit, cost, and risk criteria
  1.3   Selecting a portfolio of projects
  1.4   Establishing project review schedules




Explanation and Definitions of Process Requirements for the Selection
Phase

The IT investment management process begins with the project selection process. Projects
being proposed for funding are put through a "coarse-grained" screening process to (1) eliminate
proposals that fail to pass minimal acceptance criteria and (2) ensure that projects are being
reviewed at the most appropriate organizational level (department, bureau, unit, office, etc.).
Proposals that pass this screening process have their costs, benefits, and risks analyzed in-depth.
Once this is accomplished, all of the projects are compared against some common decision
criteria and ranked based on their relative benefits, costs, and risks. Using this prioritized list as
a guide, senior managers make decisions about which projects will be proposed for funding for
the upcoming year. This post-prioritization decision-making on the appropriate mixture of
projects is the essence of IT portfolio analysis. Finally, after these funding decisions have been
made, schedules for reviewing projects are established or updated.


1.1 Screening                The organization should have a process that outlines how to
Projects                     introduce projects for funding and how these projects will be
                             screened for relevancy to business/program goals and objectives and
                             technical soundness. Specifically, the organization should
Applicable criteria:
                             •   define what constitutes an IT project,
CCA 5122(a)


                                                 23
             Select   Control   Evaluate


 Process



   Data



 Decisions




CCA 5122(b)(3)                             •   identify initial requirements that projects must meet in order to
CCA 5122(b)(5)                                 be seriously considered for funding,
PRA 44 USC 3506 (h)(5)                     •   explain how screening will be conducted, and
EO 13011 Sec. 2(b)(3)                      •   establish roles and responsibilities for conducting the screening.
OMB A-11, Part 3
OMB A-130 parts 8b(1),
                                           The screening process should be established in policy guidance (to
8b(2), 8b(4), 8b(5)
OMB A-109
                                           ensure that it is conducted consistently) and used at all levels of the
OMB A-94                                   organization.
OMB Memo M-97-02
                                           As part of the initial screening process, there should be documented
                                           screening criteria (minimal requirements) that all projects are
                                           expected to meet.

                                           The screening criteria should serve three functions. They should:

                                           •   identify whether the project meets initial acceptance
                                               requirements,

                                           There should be some initial requirements that projects must
                                           meet before they are reviewed in greater detail. These initial
                                           requirements may include return-on-investment (ROI) thresholds
                                           (or minimum cost/benefit ratios), identification of the project's
                                           link to objectives in the business or strategic plans, evidence of
                                           compliance with the organization's information technology
                                           architecture, identification of business/programmatic
                                           sponsorship, and assurance that all necessary project proposal
                                           and justification steps have been performed.

                                           •   ensure that the project is being reviewed at the most appropriate
                                               organizational level, and

                                           Not all projects need to be reviewed at the same organizational
                                           level. The criteria for screening projects should be structured so a
                                           determination can be made about where in the organization a
                                           decision would best be made. Certain cost and risk thresholds
                                           can be used to determine what requires centralized (department
                                           or agency level) versus decentralized approval (bureau or unit
                                           level).

                                                              24
                                                                                                                Select   Control   Evaluate


                                                                                                    Process



                                                                                                      Data



                                                                                                    Decisions




                                  •   identify what level of management scrutiny is appropriate given
                                      the project's type, size, and risks.

                                  There should be flexible but defined rules explaining how project
                                  review and approval may vary based on the project's relative
                                  costs, benefits, and risks. Low-cost, low-risk projects should not
                                  need to have the same justification provided for them as projects
                                  of greater cost, risk, and organizational impact.

                                  On the basis of this screening process, projects will either move on
                                  for more in-depth analysis or will be sent back to the originating
                                  program group.


Key questions to ask related to screening projects
The purpose of these questions is to evaluate the presence (or absence) of defined processes for screening projects,
not to assess how well the screening processes are being executed. The use of the processes and the quality of the
information required by them will be addressed in later sections.

      Does the organization have a defined process for submitting and screening new funding
      proposals for management consideration? Is this process established in policy guidance?

      Does the process define (1) what information is to be submitted, (2) who must approve
      (screen) the information prior to formal submission, (3) how a determination will be made
      about where in the organization the project will be reviewed, and (4) the stages of
      management review?

      Are roles, responsibilities, and authority for people and offices involved in the screening
      process clearly defined?

      What information is required for submitting projects for funding? For most projects this
      information may include:
      •    business case justification, including
           • clear, designated senior management sponsorship from a program/business unit,
           • links to business/program/mission objectives that the project is helping to achieve,
              as well as an explanation of how the IT investment will directly or indirectly help
              achieve the intended outcomes associated with these objectives, and
           • clear identification of proposed benefits (both quantitative and qualitative)

                                                         25
             Select   Control   Evaluate


 Process



   Data



 Decisions




             •             cost/benefit estimates
             •             alternative and sensitivity analyses
             •             compliance with the information technology architecture
             •             risk assessment

             Do defined thresholds for benefit/cost ratios, return on investment calculations, risk
             assessments, etc. exist? Are these thresholds clearly defined and understood?

             Are all funding proposals treated the same or does the organization have different process
             requirements for funding proposals of different size, scope, etc.? Are these different
             requirements adequately documented?

             If exceptions to the screening process are allowed, are the conditions for allowing
             exceptions clearly documented?

             Does the process clearly stipulate potential actions that can be taken for projects that are
             funded without evidence of following the screening process?


1.2 Analyzing and                           The benefit, cost, and risk information of all projects (initial
Ranking All Projects                        concept, proposed, under development, operational) should be
Based on Benefit,                           analyzed and assessed in detail.
Cost, and Risk
Criteria                                    Each project should have a business case developed that provides
                                            the sponsor's justification for the project. The business case should
                                            identify the organizational needs that the project is meeting or
Applicable criteria:                        proposes to meet; provide information on the benefits, costs, and
                                            risks of the project; and establish proposed project development
CCA 5122(a)                                 time frames and delivery schedules. The information in the
CCA 5122(b)(3)                              business case should be continuously updated to ensure that it
CCA 5122(b)(5)                              always reflects the current situation.
OMB Memo M-97-02
                                            The organization should have some established group or audit
                                            function that is responsible for verifying and validating the various
                                            analyses (cost/benefit analyses including feasibility studies, risk
                                            assessments, and alternatives analyses) and information that are



                                                               26
                                                                     Select   Control   Evaluate


                                                         Process



                                                           Data



                                                         Decisions




submitted as part of a project's business case. This validation
should include

•    reviewing assumptions that were made,
•    assessing all of the alternatives that were analyzed and
     determining whether others should have been included,
•    reviewing the cost and benefit estimates to ensure that they
     were accurate and realistic,
•    evaluating the risks that were identified and determining
     whether others may be applicable, and
•    evaluating the sensitivity analyses that were conducted.

The organization should have a management information system
(MIS) or some other mechanism where all project information is
collected and maintained. Such a mechanism, if kept accurate and
up-to-date, can make data verification and validation easier by
allowing the organization to track costs, risks, etc. over time.

This mechanism for collecting and maintaining project
information will also be essential during the Control and
Evaluation phases to (1) help assess whether projects are still
aligned with mission needs and organizational objectives,
(2) determine whether projects are meeting planned performance
goals, and (3) identify possible revisions to the overall investment
management process based on previous experiences and lessons
learned.

After each project's cost, risk, and benefit information has been
examined and validated, all of the projects should be compared
against some common decision criteria in order to weigh the
relative merits of the projects and develop a prioritized listing of
projects.

The criteria used for assessing and ranking projects should consist
of elements related to three essential areas--benefits, costs, and
risks. Often organizations will establish broad categories related to
these three areas and then develop more specific subelements that
come under each broad category. For example, an organization

                   27
            Select   Control   Evaluate


Process



  Data



Decisions




                                          may establish risk as a categorical heading and then include
                                          schedule risks, cost sensitivity, technical risks, organizational risks,
                                          and risks if the project is not undertaken as subelements under the
                                          risk heading.

                                          Different organizations will break these broad categories and
                                          subelements out in different ways. For instance, some organizations
                                          may include a project's costs as one of several factors under risk,
                                          while others break project costs out as a separate category.

                                          Decisions should rarely be made based on one project factor, such
                                          as the project's estimated cost or a projection of reduced cycle
                                          time. Using an assortment of decision criteria to make decisions
                                          allows an organization to take into account and compare the
                                          different subtleties of a wide variety of projects.

                                          The organization may assign weights to each of the broad
                                          categories, as well as any subelements related to each category, in
                                          order to help prioritize those factors that the organization considers
                                          to be the most significant (e.g., a company that has limited
                                          experience developing systems may give technical risk a greater
                                          weight than projected cost). For an example of one organization's
                                          criteria for ranking projects, as well as the related weights given to
                                          each factor, see appendix II.

                                          The mixture of weights among the ranking criteria will vary from
                                          organization to organization. The weights that are given should
                                          take into account the agency's unique mission, capabilities, and
                                          limitations.

                                          The weighting schema that the organization establishes should be
                                          defined and documented. Such documentation is even more
                                          important if different weighting approaches are used for different
                                          kinds of projects (operational, infrastructure, applications
                                          development projects, etc.).

                                          To provide senior managers with an understanding of the relative
                                          costs, risks, and benefits of each project compared to the other

                                                              28
                                                                     Select   Control   Evaluate


                                                         Process



                                                           Data



                                                         Decisions




projects, the organization may develop a scoring model or decision
support tool. Such a tool compares the costs, benefits, and risks of
each project against the cost, benefit, and risk criteria and assigns a
score for each factor. The scores that the project receives for each
factor are then added up to produce a cumulative score that
establishes the project's relative worth and allows comparison
against all other projects. See figure 6 for an example of a ranked
list of IT projects.

An important point for an organization in developing such a
scoring model or decision support tool is to precisely define the
scoring elements (i.e., define what constitutes a 1 versus a 5).
The purpose behind these definitions is to ensure more consistent
or uniform objectivity in the scoring process, which helps to
eliminate widely varying interpretations and implementation.




                   29
             Select     Control   Evaluate


 Process



   Data



 Decisions




Figure 6: Example of Ranked List of IT Projects

                      Project                 Estimated   Strategic       Mission         Organizational   Risk            Benefit/        Total
                      Name                     Project    Alignment       Effectiveness   Impact                           Cost            Score
                                               Costs/                                                                      Ratio
                                                Year
                                                          25 pts. total   20 pts. total   10 pts. total    20 pts. total   25 pts. total   100 pts. total
                                                                                                                                                            A
                      Project                800K         23              18              8                18              20              87
                      XXXXXX                                                                                                                                P
                      Project
                      XXXXXX
                                             620K         23              15              9                16              15              77               P
                      Project                582K         18              14              7                14              15              68               R
                      XXXXXX
                                                                                                                                                            O
                      Project                500K         16              16              7                16              10              65
                      XXXXXX                                                                                                                                V
                      Project                1698K        15              18              6                9               15              63
                      XXXXXX                                                                                                                                E
                      Project
                      XXXXXX
                                             997K         15              15              6                14              10              60               D
                      Project                1578K        6               14              7                5               25              57
                      XXXXXX

                      Project                898K         11              10              7                11              10              49
                      XXXXXX

                      Project                898K         6               8               3                5               5               27
                      XXXXXX




                                                               The criteria for comparing and ranking projects should be used
                                                               uniformly across the organization (i.e., office-, division-, or bureau-
                                                               level decisions should be made using a set of criteria that are
                                                               similar to criteria used for agency- or department-level decisions).
                                                               Although different levels of the organization may use additional
                                                               criteria, the organization should have a set of minimum criteria that
                                                               are used enterprisewide. Using some common decision criteria
                                                               provides greater assurance that the organization is selecting projects
                                                               consistently and helps to avoid "apples versus oranges" project
                                                               comparison problems.




                                                                                              30
                                                                                                                   Select   Control   Evaluate


                                                                                                       Process



                                                                                                         Data



                                                                                                       Decisions




                                   There should also be incentives to ensure compliance with the
                                   process and to dissuade gamesmanship. The organization should
                                   identify who is responsible for enforcing the process and there
                                   should be explicit consequences for noncompliance.


Key questions to ask related to analyzing and ranking all projects based on cost,
benefit, and risk criteria
The purpose of these questions is to evaluate the organization's processes for analyzing and prioritizing projects,
not to assess how well the analysis and prioritization are being done. The use of the processes, as well as the
quality of the information, will be addressed in later sections.

      Does the organization require that the information and data submitted with funding
      proposals to be validated (accuracy, reliability, completeness)?
      •   Does the process stipulate who is responsible for performing this validation (sponsor,
          project team, independent validation and verification teams, auditors, inspector
          general, etc.)?
      •   Does the process stipulate where exceptions to validation are permitted? If
          exceptions are allowed, are other clearly defined conditions required to be met?

      Does the organization have an established, documented process for comparing and ranking
      all proposed IT-related funding?
      •     Has the organization defined explicit criteria that will be used to help compare and
            rank projects? Do these criteria include cost, risk, and benefit elements (e.g.,
            benefit/cost ratios, anticipated or actual impact on mission improvement priorities,
            risks versus benefits, etc.)? Do the criteria include both quantitative and qualitative
            criteria (e.g., return on investment calculations, risk modeling scores, capability
            assessments, alignment with critical needs, etc.)?
            • Are the decision criteria weighted? If scoring weights are attached to different
                items for management consideration, are these clearly defined and understood by
                participants? Has management consensus been established on use of the weighting
                scheme? Are these weights being applied consistently to all project proposals? If
                not, has the organization established different weighting schemas for different
                project types?
      •     Does the process explain how the decision criteria are to be applied?




                                                          31
             Select   Control   Evaluate


 Process



   Data



 Decisions




             If the organization uses a scoring model or decision support tool associated with the
             decision criteria to help measure the relative costs, benefits, and risks of each project, are
             the scoring elements precisely defined and differentiated?

             Is the process for analyzing and comparing IT projects required throughout the
             organization, regardless of where actual funding decisions/approvals are made?

             Does the process include incentives or disincentives to ensure compliance? Are roles and
             responsibilities for enforcing the process defined?

             Does the organization require that management evaluations, as well as scoring, ranking, and
             prioritization results be documented (either manually or through the use of automated
             applications such as a decision support tool)?
             •    Does the organization require that this information, together with approved project
                  data on cost, schedule, and performance, be tracked at an aggregate level? At a
                  project level?
             •    Does the organization require that this information be entered into a recognized
                  management information system? Does it require the data to be maintained in a
                  uniform fashion?


1.3 Selecting a                            The organization should have a senior management decision-making
Portfolio of Projects                      body, made up of program, IRM, and financial managers, that makes
                                           decisions about which projects to fund for the year based on its
                                           determination of where organizational needs are greatest. Such a
Applicable criteria:                       determination will usually be made by analyzing the gap between
                                           the organization's goals and objectives (as highlighted in its strategic
CCA 5122(a)                                and annual performance plans) and the organization's existing
CCA 5122(b)(1)                             capacity.
OMB Memo M-97-02
                                           The roles and responsibilities of the IT investment review group
                                           should be clearly identified and documented. The organization
                                           should also identify how this group will go about making decisions.
                                           This should include establishing how decisions will be reached, how
                                           conflicts will be handled, and how stakeholder input will be brought
                                           into the process.



                                                              32
                                                                    Select   Control   Evaluate


                                                        Process



                                                          Data



                                                        Decisions




The IT investment review group may only make decisions about
IT projects (these decisions would then be forwarded to a larger,
capital planning board), or it may make IT-related decisions as
part of the organization's capital planning decision-making
process.

The investment review group will make decisions about which
projects to propose for funding, using the list of ranked projects as
a key input. As the group goes about making these decisions, a
number of trade offs will have to be made. For instance, the group
will need to decide how much should be spent to continue
operating and maintaining existing systems, versus funding
enhancements to current systems, versus funding systems that are
currently under development, versus funding new projects, versus
funding research projects that assessing the applicability of
emerging technologies. The group must also determine the
proportions that will be spent on the various IT types (i.e., research
and development, administrative, mission critical, infrastructure,
etc.). And, the group must take into account dependencies among
projects.

The decision-making process should help address difficulties
associated with using different units of measure for analyzing
different kinds of IT projects, as well as a balancing of "soft"
versus "hard" quantitative data.

To aid the investment review group in making trade offs between
various project types and phases, the organization may maintain a
data repository that contains historical information on expenditures
in different IT investment categories (operations and maintenance,
enhancements to current systems, new systems development,
research into developing or applying emerging technologies, etc.).
By maintaining this information, the organization can review how
much was spent previously and factor this in to current spending
decisions. See figure 7 for an example of how one organization
tracks IT portfolio spending categories.




                   33
             Select   Control   Evaluate


 Process



   Data



 Decisions




Figure 7: IT Portfolio Spending by Category

                                                                  100%


                                                                       80


                                                                       60


                                                                       40


                                                                       20


                                                                        0
                                                                            Year 1 Year 2 Year 3 Year 4
                                           Research                           5%      6%      2%       4%

                                           Development                       31%     25%      9%      21%

                                           Enhancements                      23%     24%     48%      42%

                                           Operations                        41%     45%     41%      33%


                                           Total costs (in millions)         $30.4   $33.7   $ 42.3   $ 43.5




                                                        As part of the process of making trade offs and determining
                                                        spending priorities, the organization may also conduct a review (in-
                                                        house or via outside consultant/expert) of its current IT spending
                                                        portfolio to assess alignment with mission needs, priorities, strategic
                                                        direction, major process reengineering, etc. This review may
                                                        include a trend analysis to show how patterns of investment and
                                                        spending have changed, as well as an analysis to estimate how the
                                                        spending pattern may change with the proposed IT portfolio.

                                                        No matter how rigorous or structured the organization's decision-
                                                        making process is, decisions about which projects to select for
                                                        funding are ultimately managerial decisions. If senior managers

                                                                                34
                                                                                                                 Select   Control   Evaluate


                                                                                                     Process



                                                                                                       Data



                                                                                                     Decisions




                                  select projects that score low when compared to other projects
                                  (e.g., high-risk, high-return projects) the justification for these
                                  decisions should be documented and the project's progress should
                                  be closely monitored during the Control phase. Making such
                                  exceptions should be kept as minimal as possible, however, to
                                  preserve the integrity of the decision-making process.

                                  The process of reviewing and selecting IT projects should be
                                  explicitly linked with other business processes (e.g., planning,
                                  budgeting, acquisition). Most investment decisions should mirror a
                                  planning decision or business objective and should be reflected in
                                  related budgeting documents and decisions. In addition, new
                                  investment proposals should be highlighted in the agency's capital
                                  plan that is submitted to OMB.

                                  An agency's capital plan should include a statement of the
                                  agency's strategic plans, analysis of the portfolio of assets already
                                  owned or in procurement, a description of the gap between goals
                                  and objectives and existing capacity, justification for new
                                  acquisitions proposed for funding, and other related information.

                                  The investment review group's responsibilities will usually not end
                                  once it has decided upon the mix of projects that will be proposed
                                  to comprise the current year's investment portfolio. Instead, the
                                  group should meet on a regular basis (often quarterly) to discuss
                                  the status of projects and to make further project decisions. The
                                  group may also be responsible for reviewing investment portfolio
                                  decisions that were made by lower-level organizational units.


Key questions to ask related to selecting a portfolio of projects
The purpose of these questions is to evaluate the presence (or absence) of defined processes and procedures for
selecting a final set of IT projects for funding. An assessment of how well the processes are being followed, as well
as the quality of the information, will be addressed in later sections.

      Does the organization have a formal systematic process for determining priorities and
      making funding decisions? Does the process clearly establish who in the organization has
      the responsibility and authority for making final IT-related funding decisions?


                                                         35
            Select   Control   Evaluate


Process



  Data



Decisions




            •             Is the process clear in establishing this responsibility and authority for various
                          organizational levels (department/corporate, bureau, office, etc.)?

            Has the organization established an IT investment review group (or some other review
            function)?
            •    Who is on this review group?
            •    Does the process stipulate how membership on this group is determined and can be
                 changed?
            •    Does the process cover the roles, responsibilities, and authority of this group? Is it
                 clear on the role the group will play in (1) selecting, controlling, and evaluating IT
                 investments and (2) suggesting changes to organizational policies, procedures, and
                 practices?
                 • Does the group have authority to (1) approve, cancel, or delay projects, (2)
                     approve plans for mitigating risks, (3) validate expected returns, (4) place
                     constraints on investment proposals regarding project size and duration?
            •    Does the process stipulate the operating rules and procedures for this group, (i.e.,
                 explaining when it will meet, how it will function, how decisions will be made, what
                 steps will be taken to resolve conflicts, etc.)?
            •    Is it clear what projects the investment review group (or similar management group)
                 will review (e.g., all IT spending proposals or IT spending proposals that meet or
                 exceed defined decision thresholds--based on cost; level of risk; cross-functional,
                 bureau, or office impact; or involving common infrastructure needs such as
                 telecommunications, data centers, or networks)?

            Are IT decisions made as part of an overall capital planning process or are IT projects
            separated out? Does the process explain how decisions made on IT spending will be
            incorporated into the organization's overall budgeting or capital programming decision-
            making process?

            Does the process require that data on obligations, outlays, and actual expenditures be
            maintained for all IT spending? Are categories of IT-spending defined within the
            organization (e.g., hardware, infrastructure, telecommunications, operations and
            maintenance, applications development, data processing services, personnel, etc.)?




                                                                36
                                                                                                                Select   Control   Evaluate


                                                                                                    Process



                                                                                                      Data



                                                                                                    Decisions




      Has the organization conducted a review (in-house or via outside consultant/expert) of its
      current IT spending portfolio to assess alignment with mission needs, priorities, strategic
      direction, or major process reengineering? Does the organization have a process for
      documenting and disseminating results of this review? Has the rate and type of IT
      spending been aligned with management expectations?
      •    Has trend analysis been done to show how patterns of investment and spending are
           changing?
      •    Has an analysis been conducted to show how spending patterns could be affected by
           the proposed IT portfolio?

      Does the process define how unit or office-level IT decisions will be reviewed?


1.4 Establishing                  After making funding decisions, each project that was selected
Project Review                    should have a review schedule established for it, or should have its
Schedules                         current review schedule assessed and updated as needed. The time
                                  frames for these reviews will depend on various project-specific
                                  factors (amount of risk, investment size, mission importance,
Applicable criteria:              capability of the project team, etc.).

CCA 5122(a)                       It is important that these reviews be conducted on a regular,
CCA 5122(b)(1)                    scheduled basis. These reviews do not necessarily have to
CCA 5122(b)(6)                    coincide with major project milestones. Moreover, "review
FASA 41 USC 263
                                  triggers" should be established that automatically require a
FASA 10 USC 2220
OMB Memo M-97-02
                                  management review meeting. For example, a cost, schedule, or
                                  performance deviation of 10% or greater might require an
                                  immediate project review.


Key questions to ask related to establishing project review schedules
These questions deal with the processes that the organization has in place, not an evaluation of how well the
processes are actually working. Decisions and data important to this will be covered later.

      Does the process stipulate how approved projects are to be monitored by senior
      management in regular investment control meetings?
      •   Are there procedures for informing project managers of decisions about monitoring
          schedules made by the investment review group?


                                                         37
            Select   Control   Evaluate


Process



  Data



Decisions




            Is the review process clear on criteria for deciding the kinds of projects that will receive
            regular management monitoring and oversight by an investment review group versus those
            that will be monitored exclusively by management sponsors?

            Does the process allow the investment review group to call special project reviews outside
            of regular meetings if the group deems it necessary?

            Does the process require any additional certification or reviews before high-risk projects
            are allowed to proceed (e.g., risk mitigation plans, additional cost certifications, etc.)?




                                                       38
                                                                                               Select   Control   Evaluate


                                                                                   Process



                                                                                     Data



                                                                                   Decisions




                                      Select--Data
  Key data elements to be reviewed:

  2.1   Evidence that each project has met project submission requirements
  2.2   Analyses of each project's costs, benefits, and risks
  2.3   Data on the existing portfolio
  2.4   Scoring and prioritization outcomes
  2.5   Project review schedules


Explanation of Data Requirements for the Selection Phase

Good decisions require good data. Ensuring that each project meets established screening and
ranking requirements and that the project's information is accurate and up-to-date is essential
for ensuring that the most critical needs of the organization are being met by the projects and
systems that are selected. In addition, the ex post information that is generated during this
phase, such as project review schedules or risk mitigation plans, based on the selection
decisions that are made, is critical for controlling and evaluating projects during the next two
phases.


2.1 Evidence That           The efficiency of the investment management process depends
Each Project Has            initially upon how well the organization is ensuring that all projects
Met Project                 meet initial project acceptance requirements and that necessary
Submission                  project proposal and justification steps have been performed. There
Requirements                should be evidence that each project that is submitted has been
                            screened, analyzed, and evaluated according to processes and
Applicable criteria:        criteria established by the organization.

CCA 5122(a)                 The information that is analyzed may include verification that all
CCA 5122(b)(3)              requisite selection data were submitted, that answers were received
CCA 5122(b)(5)              for all relevant questions, that projects met business/program goals
CCA 5123(3)
                            and conformed to the agency's information technology architecture,
CCA 5126
                            and that projects that did not meet these requirements were not
OMB A-127, Para. 6,7
                            allowed to move on for further review and consideration. There

                                               39
            Select   Control   Evaluate


Process



  Data



Decisions




OMB A-123, Part II                        should also be evidence demonstrating that all business units
OMB Memo M-97-02                          adhered to organizational policies and procedures regarding the
                                          screening and acceptance of projects.

                                          Much of the evidence that will be reviewed will consist of cursory
                                          completeness and quality checks. For instance, if the organization
                                          has requirements that all projects over a certain cost threshold must
                                          (1) submit complete cost/benefit and risk analyses, (2) identify the
                                          business objectives that the project is meeting, and (3) provide
                                          assurance that the project conforms to the organization's technical
                                          architecture, then a review of projects that went on for further
                                          review should not identify any projects that did not meet these
                                          initial requirements. Evidence should also be available
                                          demonstrating that each project adhered to the documented
                                          process.

                                          There should also be evidence that information that was submitted
                                          was validated by a quality assurance/control function. Such
                                          validation can be performed by in-house quality control/quality
                                          assurance staff, internal audit staff (e.g., inspector general), etc.
                                          The project information should also be verified to ensure that it is
                                          accurate and reflects the most up-to-date information.

                                          All project information should be up-to-date, cost numbers should
                                          be accurate, benefits should be quantified to the extent possible,
                                          risks should be spelled out, alternatives should be identified, and
                                          sensitivity analyses should have been conducted.


Key questions to ask related to evidence that each project has met initial project
requirements
These questions should be used to help determine whether the requisite information on costs, benefits, and risks
has been submitted for funding proposals, "screened" for completeness and adherence to prescribed agency
practices, and validated.

               For IT proposals that were submitted for funding consideration, was all of the required
               data/information prepared and submitted in accordance with the prescribed process?



                                                             40
                                                                                                 Select   Control   Evaluate


                                                                                       Process



                                                                                        Data



                                                                                    Decisions




     Is there evidence that the data/information (cost, schedule, performance, risk) for
     submitted projects has been validated--either independently or using a self-assessment
     process?

     Did the information/data presented in the proposals come from agency-recognized
     information systems (automated or otherwise)?

     Is the information/data easily accessible for further review and consideration?


2.2 Analyses of Each        Each project that is submitted should have a business case prepared
Project's Costs,            that provides justification for the project. Included in the business
Benefits, and Risks         case should be identification of the project's functional requirements
                            and estimates of the project's life-cycle costs, benefits, and risks (to
                            the extent possible), as well as the corresponding analyses that
                            were conducted to develop the estimates.

Applicable criteria:        Making accurate cost savings estimates and benefit
                            determinations requires having at least a rudimentary
CCA 5122(a)                 understanding of the baseline costs and benefits from existing IT
CCA 5122(b)(3)              capabilities.
CCA 5122(b)(5)
CCA 5126
                            A key analysis that should almost always be submitted with project
OMB A-94
OMB A-130, parts 8b(1),
                            proposals is a cost/benefit analysis. A complete cost/benefit
8b(2)                       analysis should
OMB Memo M-97-02
                            •    identify and quantify benefits and costs,
                            •    identify assumptions and constraints that were used when
                                 developing these figures,
                            •    evaluate alternatives using net present value, and
                            •    perform risk and sensitivity analyses.

                            The amount of rigor and types of analyses that are conducted will
                            depend, in part, on the size of the investment and the amount of
                            risk. It may not be economical to conduct an in-depth cost-benefit
                            analysis for a low-cost, low-risk project that only affects a specific
                            division or office or a limited number of users. The organization
                            should have a process that outlines what project data are

                                               41
            Select   Control   Evaluate


Process



  Data



Decisions




                                          required given each project's type, cost, and risks; variation in the
                                          quality or type of data should not be ad hoc.

                                          Listed below are some of the cost, risk, and benefit elements that an
                                          organization should keep in mind as it develops project estimates.

                                          Costs (recurring and non-recurring)

                                          •    Up-front costs, such as hardware and software purchases,
                                               costs to design and develop the project, transition costs, etc.
                                          •    On-going costs, such as salaries, software upgrades, training,
                                               supplies, operations and maintenance, disposal, etc.
                                          •    Indirect costs, such as initial productivity losses, computer
                                               support (network management, data administration, hotlines),
                                               etc.

                                          Risks

                                          •    Project risks, such as the size of the investment, project size
                                               and longevity (is the project designed in modules, does the
                                               project rely on the implementation of other systems, do other
                                               systems rely on this project), has the project group managed
                                               other projects of similar risk and complexity.
                                          •    Organizational risks, such as mission risks if the project does
                                               not proceed, program management's commitment to the
                                               project, political expectations for the project, whether the
                                               project is legislatively mandated, etc.
                                          •    Technical risks, such as skills required, hardware and
                                               software dependencies, application software, etc.

                                          Benefits (will usually consist of both tangible and intangible
                                          benefits)

                                          •    Tangible benefits include benefits that can be explicitly
                                               quantified. Such benefits may include reducing costs,
                                               increasing productivity (e.g., reducing errors, eliminating
                                               duplication or needless work steps, etc.), decreasing cycle


                                                             42
                                                                                                                Select   Control   Evaluate


                                                                                                    Process



                                                                                                      Data



                                                                                                    Decisions




                                        time, or improving service quality (e.g., timeliness,
                                        convenience, access, reliability, etc.).
                                  •     Intangible benefits include benefits that may be easy to
                                        identify, but that are difficult to quantify. These benefits may
                                        include faster, more efficient decision-making; greater data
                                        accuracy; improved data security; reduced customer burden; or
                                        increased organizational knowledge.

                                  In identifying and measuring IT benefits, it is important to
                                  always remember the business function or process that is being
                                  supported by the technology. For instance, the benefits that are
                                  gained from implementing EDI technology are derived from the
                                  increased capability and efficiency that the technology provides to
                                  the organization and its customers.

                                  All of the information in the business case should be as up-to-date
                                  and accurate as possible. If the analyses are to yield meaningful
                                  results, it is essential that the project team carefully formulate
                                  assumptions, identify feasible alternatives, and provide realistic cost
                                  and benefit estimates.

                                  Most agencies have criteria or methodologies detailing how
                                  cost/benefit analyses are to be conducted and what should be
                                  included. In addition, OMB Circular A-94 provides guidance
                                  and discount rates for conducting cost/benefit analyses.


Key questions to ask related to the analyses of each project's costs, benefits, and risks
These questions are concerned with the review of the data and corresponding analyses submitted for project
justifications. In many cases, determining answers to these questions will require that a sample of business cases
be taken.


      Are project cost data fully identified--direct, indirect, ongoing? Do the data include full life-
      cycle costs? Did these cost data come from a recognized agency financial system?




                                                         43
            Select   Control   Evaluate


Process



  Data



Decisions




               Have benefits of the investment been identified using quantitative and/or qualitative
               data/information that relate directly to mission support and performance improvement?
               Are expected cost savings, productivity gains, and improvements in quality identified and
               timeframes specified as to when these should occur?

               Have all foreseeable risks been identified? These risks may include technical, managerial,
               capability, procurement, organizational impact, stakeholder, etc. Have all concerns and
               potential problem issues been resolved or accounted for in a risk mitigation strategy for the
               project?

               Have business owners been involved in constructing and certifying the accuracy of the data
               that are submitted?

               Were baseline cost and performance data used as a basis for analysis? Is this information
               reliable and accurate? If baseline data were not available, were the estimates generated
               using a prescribed approach or method?

               For projects that are requesting continued or additional funding (for a new phase, new
               module, or as a result of increased costs), is there evidence that the newly submitted data
               reflect any changes that may have occurred to the cost, benefit, or risk information?

               Has project schedule information been reviewed in light of competing priorities; skills,
               capabilities, and availability of agency staff; contractor expertise and experience; etc.

               Was the cost and return information that was submitted constructed using accepted
               methods and techniques (prescribed by the agency, OMB guidance, legislative provisions,
               and/or accepted industry practice)?


2.3 Data on the                           Because all projects (ongoing, under-development, etc.) go through
Existing IT Portfolio                     the Selection process (usually on an annual basis), portfolio data
                                          from previous years should be available to assess and compare
                                          previously selected projects. The spending, cost, and obligation
Applicable criteria:                      data in this portfolio should be up-to-date and categorized in ways
                                          that are most meaningful to agency management.
CCA 5122(b)(1)
CCA 5122(b)(3)
CCA 5122(b)(5)

                                                            44
                                                                                                                Select   Control   Evaluate


                                                                                                    Process



                                                                                                      Data



                                                                                                    Decisions




CCA 5122(b)(6)                    An agency's cost accounting system should be able to distinguish
CCA 5126                          between what has been obligated to date and what is still
OMB A-130, part 8b(2)             available, as well as identify what the incurred costs to date were
                                  for. In addition, the system may be able to split spending into
                                  more specific categories, such as development, operations,
                                  maintenance, etc. (Activity-based cost tracking, for example,
                                  should provide this detail.)


Key questions to ask related to data on the existing portfolio
The purpose of these questions is to determine whether information/data on the organization's existing IT portfolio
of spending and investment is used as part of the management review process. This helps to ensure that funding
decisions are being made in the context of overall spending directions.

      Does the organization maintain data on its current IT spending portfolio (e.g., are major
      categories of spending and investment defined and tracked--such as operations and
      maintenance, applications and systems development, hardware acquisitions,
      telecommunications, personnel, contracted services, data administration, research and
      development, etc.)?

      Are the costs and returns of the IT spending portfolio presented on an aggregate basis
      (past, current, future)? On a project basis?

      Was the portfolio information (IT spending categories, cost/benefit estimates, average
      development costs, etc.) derived from recognized agency management information systems?
      Are standard definitions and reporting elements used to maintain consistency and
      uniformity?


2.4 Scoring and                   There are several pieces of information that should arise out of the
Prioritization                    Selection phase, based on the actual decisions that are made. This
Outcomes                          information includes

Applicable criteria:              •     the initial project scores and ranked list of projects,
                                  •     the investment review group's scores based on any additional
CCA 5122(b)(3)                          decision-support tools,
CCA 5122(b)(5)                    •     the investment group's final list of projects that it is proposing
                                        to make up the investment portfolio,

                                                         45
            Select   Control   Evaluate


Process



  Data



Decisions




                                          •    documented justification for selecting projects that scored
                                               below accepted thresholds (e.g., high-risk, high-return
                                               projects), and
                                          •    funding information, as well as acquisition and development
                                               schedules, for all projects that were selected.

                                          The organization should also be maintaining net cost and benefit
                                          information on the complete portfolio of IT investments.

                                          Finally, all of the projects that were selected for funding should be
                                          included in the Agency Capital Plan that is submitted to OMB.
                                          Information that is submitted in this plan should include baseline
                                          cost, schedule, and performance goals for each project.

                                          In addition to the Capital Plan, decisions that are made on the mix
                                          of existing and new projects should be clearly identified in the
                                          agency's annual performance plans. Actions described in the
                                          Capital Plan to implement the funding, procurement, and
                                          management of the IT projects should also be articulated in these
                                          performance plans.


Key questions to ask related to scoring and prioritization outcomes
These questions help determine whether senior managers consult, review, further assess information/data used to
make decisions (final or recommended to the agency head) about annual IT spending.

               Are summary data presented on each project's costs, benefits, risks for senior management
               (investment review group, etc.) to consider?

               Does the management review group conduct scoring exercises to vote on the relative
               strengths and weaknesses of proposals? Are these scoring exercises recorded and
               documented? Are the criteria that are used as the basis for the scoring instruments defined
               and used consistently?

               Can the costs of the approved list of projects for funding be tracked to available funds
               and/or reflected in the budget requests?




                                                             46
                                                                                              Select   Control   Evaluate


                                                                                  Process



                                                                                    Data



                                                                                  Decisions




     When management approves funding for projects that fall outside accepted thresholds (high
     risks, high project costs, noncompliance with the architecture, etc.), is an
     explanation/rationale provided for this decision? Are additional management and project
     reporting requirements stipulated?


2.5 Project Review         All projects that are selected for funding should have project review
Schedules                  schedules, risk management plans, and project-specific performance
                           measures established. All of this information will be particularly
Applicable criteria:       critical for assessing performance, identifying risks, and making
                           decisions during the Control and Evaluation phases.
CCA 5122(b)(6)
                           The timing of reviews, as well as the number of reviews that will
                           be conducted, will depend on the investment size of the project,
                           the amount of risk, the capability of the project team, etc.

                           In addition, the investment review group may identify additional
                           project management or investment review reporting requirements
                           (data, information, analysis), beyond what is specified by existing
                           processes, for projects that it determines are particularly high-risk.
                           These additional requirements should be clearly documented and
                           communicated to the responsible project team. The project team
                           should also be given explanation detailing how this information--and
                           its assessment by senior management--may influence project
                           continuation, delay, or cancellation.

                           At some point the project team should develop an outline or
                           strategy describing how any necessary acquisitions will be handled.
                           Key tenants of a sound acquisition strategy are that it appropriately
                           allocates risk between the agency and contractor, effectively uses
                           competition, ties contract payment to accomplishments, and takes
                           maximum advantage of commercial technology.




                                              47
            Select   Control   Evaluate


Process



  Data



Decisions




Key questions to ask related to project review schedules
The purpose of these questions is to determine whether any additional data requirements (beyond that expected by
the normal process used for investment control meetings) were decided upon by senior management and whether
these requirements were communicated to the project teams or sponsors.

               Once projects are approved for funding by an investment review group and/or agency head,
               are any additional project management or investment review reporting requirements (data,
               information, analysis) established for high-risk, high-cost projects (beyond what may be
               specified by existing processes)?
               •    If so, have these requirements been clearly documented and communicated to the
                    responsible project team?
               •    Is it clear why this data/information is being requested and what it will be used for?
               •    Has an explanation been given to the project team explaining how this information--
                    and its assessment by senior management--may influence project continuation, delay,
                    or cancellation?

               Did each project that was approved have an outline or strategy developed establishing how
               any necessary acquisitions would be handled? Is this strategy appropriate given the project
               type?




                                                        48
                                                                                                Select   Control   Evaluate


                                                                                    Process



                                                                                      Data



                                                                                    Decisions




                                  Select--Decisions
 Key decision elements to be reviewed:

 3.1 Determining whether projects met process-stipulated requirements
 3.2 Deciding upon the mixture of projects in the overall IT investment portfolio




Explanation of Decisions Made During the Selection Phase

The purpose of the Selection phase is to put the organization in the best possible position to
make decisions about which IT proposals or projects to fund. Getting to this final decision
requires that initial decisions be made about whether proposed projects should be moved on for
further consideration. It then requires decisions to be made about the relative merits of each
individual project. This is followed by the most important decisions, in which trade-offs are
made between the various projects and systems in order to develop the IT investment portfolio
that will be funded for the upcoming year.


3.1 Determining             All new projects should have a decision made about whether the
Whether Projects            project meets all minimal project requirements, at what
Met Process-                organizational level the project should be reviewed, and the level of
stipulated                  analytical rigor necessary for decisions. While these screening
Requirements                decisions should be relatively straight-forward, driven primarily by
                            project-level data sufficiency, they should not be thought of as
                            simply a cursory exercise. The overall efficiency and effectiveness
Applicable criteria         of the entire Selection phase depends to a large extent upon these
                            initial screening decisions.
CCA 5122(a)
CCA 5122(b)(1)              The organization should also have a process for determining where
CCA 5122(b)(3)              in the organization a funding decision should be made. The
CCA 5122(b)(5)
                            efficiency of the investment management process is significantly
OMB A-130, parts 8b(2),
8b(4), 8b(5)
                            affected by how well the organization identifies which projects
OMB A-11, part 3            should be reviewed where. Senior decisionmakers should not spend


                                               49
            Select   Control   Evaluate


 Process



  Data



Decisions




                                          their time in lengthy, in-depth reviews of projects that could have
                                          been easily assessed and decided upon at lower organizational
                                          levels.


Key questions to ask related to determining whether projects met process-stipulated
requirements
The purpose of these questions is to determine whether actual decisions are being made that adhere to project
screening processes and data prescribed by the organization's IT investment review process. Does the organization
make decisions using the process and data that it has put in place?

              Are decisions being made about project readiness for review using the data/information
              requirements established by a project screening process?
              •    Are project submissions being reviewed consistently (using the process and
                   information required)?
              •    Are screening decisions being recorded?
              •    Is there evidence of projects being rejected? Were explanations for submission
                   rejections documented and communicated to the business sponsor?

              If exceptions are being made to screening criteria, is the explanation documented and
              forwarded with the project proposal?


3.2 Deciding Upon                         Decisions made at this stage are the most important of all. The
the Mixture of                            projects that are proposed to make up the investment portfolio for
Projects in the                           the year should represent the best match with organizational needs
Overall IT                                and business objectives or, in instances where exceptions were
Investment Portfolio                      made, an explanation should be provided detailing reasons for the
                                          exception.

Applicable criteria                       In making the selection decisions, senior managers should be taking
                                          into account tradeoffs between the various projects and systems
CCA 5122(a)                               that are going to be funded. Making these tough choices requires
CCA 5122(b)(1)                            the organization to develop and maintain an understanding that not
CCA 5122(b)(3)                            every project or system can be funded. Spending more for
CCA 5122(b)(5)
                                          operational systems may mean that there is less money for research
OMB A-130, parts 8b(2),
                                          and development. The relative merits of each project should be
8b(4), 8b(5)
OMB A-11, part 3                          rigorously assessed and analyzed in order to prioritize and select


                                                             50
                                                                                                               Select   Control   Evaluate


                                                                                                    Process



                                                                                                     Data



                                                                                                   Decisions




OMB Memo M-97-02                 those projects that best match the most critical business needs of
                                 the organization.

                                 In addition, projects selected for the portfolio should have decisions
                                 made about how often they will be reviewed and how associated
                                 risks are going to be managed.


Key questions to ask related to selecting the composition of the current portfolio
The purpose of these questions is to determine whether senior managers are making decisions following an
established process and based on reliable and accurate data. Is the organization making decisions using the process
and data that have been put in place, and do these decisions reflect an understanding of the necessary tradeoffs
that should be made among projects.

     Do the systems that were selected for the current portfolio represent the best match with
     mission needs? Do all of the selected projects support objectives or goals established in
     the organization's strategic plan or annual performance plans? Are all of the selected
     projects justified on the basis of their relative costs, benefits, and risks? Are there any
     other factors that appear to have influenced the executives' decisions?

     For ongoing projects, have projected versus actual data on costs and interim results been
     used to make decisions about project continuation?

     Were project decisions made at the appropriate organizational level?

     Determine the ratio of funding between the various project types (new, proposed, under
     development, operational, etc.) that were selected. Does this ratio appear to effectively
     support the organization's business plan and objectives?




                                                        51
            Select   Control   Evaluate


Process



  Data



Decisions




                                                   Control--Process
            Key process elements to be reviewed:

            4.1         Consistently monitoring projects
            4.2         Involving the right people
            4.3         Documenting all actions and decisions
            4.4         Feeding lessons learned back in to the Selection phase



Explanation and Definitions of Process Requirements for the Control Phase

Achieving maximum benefits from a project, while minimizing risks, requires that the project be
consistently monitored and managed for successful results. During the Control process, agency
executives should be actively engaged in monitoring all of the projects in the investment
portfolio, making decisions and taking actions to change the course of a project when necessary,
and incorporating their experiences back in to the Selection phase to further refine and improve
the process.

4.1 Consistently                           Each project should be reviewed at key milestones in its life cycle
Monitoring Projects                        (a project review schedule should have been approved when the
                                           initial funding decision was made). The focus of these reviews
Applicable criteria                        should expand as projects move from initial design and pilot
                                           through full implementation and as the dollar amounts that are
CCA 5122(b)(1)                             expended increase (see figure 8).
CCA 5122(b)(6)
CCA 5125(c)(2)                             A low-cost, small-scale research and development project being
CCA 5127
                                           conducted to determine the applicability of a systems technology
PRA 44 USC 3506 (h)(5)
                                           to a business process requirement might receive limited review
FASA 10 USC 2220
FASA 41 USC 263                            other than assessing whether the general approach is sound and
EO 13011 Sec. 2(b)(3)                      feasible. However, projects that are preparing for limited field or
OMB A-11, part 3                           full-scale implementation should be reviewed in-depth--including
OMB A-130, parts 8b(2),                    cost and performance to date--to ensure that the project delivers
8b(3)


                                                              52
                                                                                                                                                 Select   Control   Evaluate


                                                                                                                                     Process



                                                                                                                                       Data



                                                                                                                                     Decisions




                                     promised benefits within cost and risk limitations and to correct
                                     any problems before significant dollars are expended.

Figure 8: Matching Investment Control With Project Phases




                                                                                                                Control



                                                                                  Control
                                                                                                       Select

      $                                             Control                                                               Evaluate

                                                                         Select
                     Control
                                                                                            Evaluate
                                          Select
                                                              Evaluate
            Select
                               Evaluate




                Prototype                          Pilot                   Limited                              Production
                                                                         Implementation


                                     In addition, as the reviews are conducted, the context of the
                                     program that the system or project supports should be factored in.
                                     For instance, a project may exceed performance expectations, but if
                                     it is contributing to a program that is failing or is no longer needed,
                                     then little is gained for the organization.

                                     The project reviews should assess several aspects of the project and
                                     its development. Below are examples of assessment categories that
                                     should be considered as part of the project reviews:

                                     •        Deliverables--results achieved to date versus expected results
                                     •        Methodology--problems that have arisen concerning the
                                              systems development process (including contractor
                                              management issues)

                                                                         53
            Select   Control   Evaluate


Process



  Data



Decisions




                                          •    Technical--technical issues or problems concerning such
                                               factors as hardware, software development, or
                                               telecommunications
                                          •    Schedule--estimated time frames versus actuals, including
                                               schedule slippages and/or compressions
                                          •    Costs--estimated costs versus actual costs spent or obligated to
                                               date, any changes in funding, and the impact of these changes
                                          •    Business/program alignment--evaluation of the project's mission
                                               improvement effectiveness and relationship to business
                                               objectives
                                          •    Risk--risks that were previously identified are being
                                               appropriately mitigated, new risks have been evaluated and
                                               steps taken to address them

                                          The organization should have some standard policies that help
                                          ensure that these different categories are assessed uniformly across
                                          the organization; however, the measures that are used to evaluate
                                          each project will be specific to that project. For instance, the
                                          organization may have a requirement that all projects have their
                                          schedules reviewed, but the schedules that are reviewed will be
                                          different for each project.

                                          The problem with many progress reviews is that they focus almost
                                          exclusively on cost and schedule concerns. While these factors are
                                          important, the prime focus of progress reviews should be on
                                          ensuring that benefits are being accomplished, that risks are
                                          being managed, and that the project is still meeting strategic
                                          needs. As noted earlier, "review triggers," such as updated
                                          benefit/cost ratios or ROI thresholds, done in conjunction with
                                          schedule and spending checks, can help the organization
                                          determine when actions need to be taken.

                                          The organization should have a documented process detailing how
                                          reviews will be conducted, what data and project information is
                                          required, and how decisions will be made based on the results of
                                          the project reviews. This process should include identifying roles
                                          and responsibilities for making decisions, as well as rules for how
                                          the project decisions will be made.

                                                             54
                                                                                                              Select   Control   Evaluate


                                                                                                  Process



                                                                                                    Data



                                                                                                  Decisions




                                Some organizations use a traffic-light method to help make
                                project decisions. Projects are given red, yellow, or green lights
                                depending on how the project rated against expected performance
                                measures. Yellow lights indicate that management action is
                                necessary to avoid potential problems affecting project outcomes.
                                Red lights indicate that major problems have already occurred.
                                (As with all reporting and scoring mechanisms, it is critical that
                                the organization define the conditions associated with element.)

                                The organization should also have an independent audit team,
                                quality assurance group, or independent validation and verification
                                (IV&V) contractor who is responsible for ensuring that project
                                information is valid and verifying that corrective actions have been
                                taken. In addition, the organization should have procedures in place
                                to ensure that information from this quality assurance function is
                                integrated in to the project review process.

                                Finally, the organization should have mechanisms in place to ensure
                                that project teams are complying with the control process. This
                                may include incentives for raising problems to senior managers and
                                disincentives for noncompliance.

                                Project reviews, while helping to ensure accountability, should not
                                be totally viewed as a "gotcha" opportunity, in which project
                                managers are punished when problems are identified. Rather,
                                the reviews should be considered opportunities for raising
                                problems early, when they may be easier to address, rather than
                                allowing the problems to be buried, creating a risk that they will
                                arise later when costs are higher and the potential impact is
                                greater.


Key questions to ask related to consistently monitoring projects
The purpose of these questions is to determine whether fundamental elements of a management control process are
defined and documented. Evaluation of the data and actual implementation of the process are covered in later
sections. The process outlined here is for a management level, not project level, review; however, many of the
same decision elements are applicable to both.




                                                      55
            Select   Control   Evaluate


Process



  Data



Decisions




                 Does the organization have a defined, documented, and repeatable process for monitoring
                 and reviewing IT projects? Does this process define what the focus of the investment
                 reviews will be? Some key elements that may be included in the review include the
                 following:
                 •    project status, including where the project stands in relation to the development of
                      other projects
                 •    business sponsor evaluation of the project
                 •    estimated vs. actual costs to date
                 •    estimated schedule vs. actual schedule
                 •    actual outcomes from modular testing, pilots, prototypes, or limited site testing vs.
                      estimates
                 •    technical performance as well as estimated impact on program or business
                      performance
                 •    updates on risk mitigation efforts, including identification of any new risks and steps
                      to address
                 •    technical issues or problems that have arisen
                 •    contractor performance and delivery
                 •    review of the methodology or systems development process
                 •    new unexpected issues affecting project progress or outcomes

                 Does the process stipulate what project data and information must be submitted for
                 management evaluation?

                 Does the process indicate how data and information being presented for management
                 review are to be verified and validated? Are roles and responsibilities for conducting this
                 verification and validation spelled out?

                 Does the process define a specific group (or groups) of managers that are responsible for
                 conducting IT investment control meetings?
                 •   Are procedural rules for how the investment review group will operate and make
                     decisions clearly defined and documented?
                 •   Are the roles, responsibilities, and authority of this group(s) clearly defined?
                 •   Is the purpose of the investment review group clearly stated?
                 •   Are the types of decisions that will be made at the IT investment control meeting
                     defined (e.g., project continuation, delay, cancellation, termination, acceleration, etc.)?




                                                            56
                                                                                                  Select   Control   Evaluate


                                                                                      Process



                                                                                        Data



                                                                                      Decisions




     If investment review processes are used across different units of an organization (e.g.,
     department, bureau, or office levels), do the different units have consistent policies,
     practices, and procedures for monitoring projects and reaching decisions?

     Does the process make accommodations for flexible reviews (frequency, required report
     submissions, etc.) for different kinds of projects (high risk/low return vs. low risk/low
     return)?

     Does the process define who is accountable for acting on review decisions? Is it clear the
     types of actions that are under different people's responsibilities (e.g., project team, CIO
     staff, business sponsor, financial staff)?

     Does the process define how open action items are to be addressed? Does the process
     define roles and responsibilities for making these decisions, as well as criteria for
     evaluating actions that are taken and determining whether the open item has been
     resolved?

     What mechanisms are in place to help ensure compliance with the review process? Are
     there disincentives in place for noncompliance? Who is responsible for overseeing the
     process? How stringently is the process maintained?


4.2 Involving the           Senior managers (particularly program managers) should be actively
Right People                involved in the ongoing project reviews and are responsible for
                            making decisions about whether to continue, accelerate, modify, or
Applicable criteria         cancel a project. While members of the development team can, and
                            should, be part of the decision-making process, they should not
CCA 5122(b)(1)              have unilateral responsibility or authority to make all project
CCA 5125(c)(2)              decisions. In addition, site executives and project managers should
PRA 44 USC 3506 (a)(4)      take part in devising and approving the solution to any problems
                            that are identified.




                                                57
            Select   Control   Evaluate


Process



  Data



Decisions




Key questions to ask related to involving the right people
The purpose of these questions is to help determine who is involved in making project decisions.

                 Who is involved in ongoing project reviews and decisions? Do the review groups include
                 staff from program, IT, and financial offices? Is there membership by a quality assurance
                 or some other outside assessment group?

                 Are project managers or site executives included in devising and approving actions to
                 address deficiencies that were identified?


4.3 Documenting All                       All of the information in the business case, including the various
Actions and                               analyses that were conducted to justify the project, should be
Decisions                                 updated to reflect the current state as project implementation
                                          continues and dollar amounts increase.

Applicable criteria                       Some leading organizations estimate that often they cannot
                                          accurately estimate costs or quantify benefits until almost 40
CCA             5122(b)(3)                percent of the way into the project.
CCA             5122(b)(5)
CCA             5122(b)(6)                The organization should have a uniform mechanism (e.g.,
CCA             5123(3)
                                          management information system) for collecting, automating, and
CCA             5126
                                          processing data on expected versus actual outcomes. Specifically,
                                          this mechanism should

                                          •    provide the cost and performance data needed to monitor and
                                               evaluate investments individually and strategically,
                                          •    provide feedback on the project's adherence to strategic
                                               initiatives and plans, and
                                          •    allow for the review of unexpected costs or benefits that
                                               resulted from investment decisions.

                                          Data in this system should be easily accessible to both the program
                                          team and senior managers.

                                          Collecting and maintaining project information is important, not
                                          only from a project review standpoint, but also from the
                                          standpoint of establishing an organizational memory. Decisions

                                                             58
                                                                                                                Select   Control   Evaluate


                                                                                                    Process



                                                                                                      Data



                                                                                                    Decisions




                                 in all three phases of the investment cycle (Select, Control, and
                                 Evaluate) will depend on this information being accessible and
                                 up-to-date.


Key questions to ask related to documenting all actions and decisions
The purpose of these questions is to help determine whether the organization has processes and procedures for
capturing project-related information. An assessment of the information and actual implementation of the
processes will be covered in later sections.

     Does the organization have procedures for ensuring the accuracy and reliability or project
     information? Does the organization define the various pieces of information that are to be
     updated and maintained? This information may include the following:
     •    project-related decisions that are made
     •    actions that are to be taken, as well as criteria or measures for evaluating
          improvement
     •    project outcomes
     •    cost/benefit analyses and other associated project information
     •    business case information

     Does the process stipulate where these data are to be maintained (e.g., official agency
     information system with uniform data standards and entry procedures)?


4.4 Feeding Lessons              Information learned during the Control phase should be fed back in
Learned Back in to               to the Selection phase to help make future selection decisions and
the Selection Phase              to modify and enhance the screening and selection decision criteria.
                                 To make this easier, there should be some mechanism in place for
                                 aggregating decisions and actions in order to identify patterns of
Applicable criteria              problems or, conversely, patterns of excellence.

CCA 5122(b)(1)                   Document the warning signs that, with hindsight, preceded the
CCA 5122(b)(6)                   problem, identify what remedial steps were taken, and what the
CCA 5123(3)                      outcome of this approach was. Such documentation will help to
                                 make future acquisition decisions and identify recurring problems
                                 on existing programs.




                                                       59
            Select   Control   Evaluate


Process



  Data



Decisions




Key questions to ask related to feeding lessons learned back in to the Selection phase
The purpose of these questions is to help determine whether the organization has processes and procedures for
aggregating project information and revising the investment management process based on lessons that are learned.
An assessment of the information and actual implementation of the processes will be covered in later sections.

                 Does the organization have a process for evaluating current decision-making processes and
                 suggesting changes to these processes based on lessons that are learned from investment
                 control reviews?
                 •    Does the process account for and distinguish between senior management decision-
                      making changes and project-level management changes?
                 •    Does the process specify someone who is accountable for identifying lessons that are
                      learned from investment control reviews?

                 Is there a process for refining or updating the selection criteria (both screening and
                 ranking) based on lessons that are learned?

                 Does the organization have a process for aggregating data/information across all major IT
                 projects (or spending categories) in order to compile an overall organizational track record
                 on costs and benefits attributable to IT?
                 •    Is someone (or office) charged with this specific responsibility?
                 •    Does the organization have procedures for presenting this information to the IT
                      investment review group? For presenting it to all agency executives?




                                                            60
                                                                                                Select   Control   Evaluate


                                                                                    Process



                                                                                      Data



                                                                                    Decisions




                                    Control--Data
  Key data elements to be reviewed:

  5.1 Measures of interim results
  5.2 Updated analyses of each project's costs, benefits, schedule, and risks



Explanation and Definitions of Data Requirements for the Control Phase

Because the Selection phase usually occurs only once a year during the annual budget process,
project information for that phase tends to be collected and assessed on a periodic basis. In
contrast, information in the Control phase is continuously collected, updated, and fed to agency
decisionmakers. The data in the Control phase should consist of such items as comparisons of
actual results achieved to date versus estimates and an assessment of benefits achieved as part
of project pilots or prototypes. Data collected during this phase will also consist of ex post
documentation such as executive decisions and changes made to projects to address risks or
better meet business requirements. The type and depth of data that are collected and
maintained in this phase should be commensurate with the type and size of the project.


5.1 Measures of             As projects move from one phase of the project's life cycle to the
Interim Results             next, and as the dollars that are expended increase, interim results
                            should be compared against estimates to ensure that the project is
                            progressing as expected and to indicate when actions should be
Applicable criteria         taken as problems arise or requirements change.

CCA   5112(c)               The organization should have project-specific measures established
CCA   5122(b)(1)            to help analyze actuals versus estimates, ensure that the project is
CCA   5122(b)(5)            meeting business requirements, and identify where improvements
CCA   5122(b)(6)
                            may be needed. These measures will consist of items such as cost
CCA   5123(3)
CCA   5125(c)(2)
                            and schedule information, quantitative and qualitative benefit
CCA   5126                  measures, status of deliverables, risk elements, etc. The measures
CCA   5127                  should be updated as actual costs, risks, and benefits are identified.


                                               61
            Select   Control   Evaluate


Process



  Data



Decisions




GPRA 31 USC 1115                          Using these measures, the organization should identify and monitor
GPRA 31 USC 1116                          interim results that are achieved. The following are examples of the
CFO 31 USC 902(a)(3)                      kinds of data that should be analyzed:
OMB A-127, Para. 6,7
OMB A-123, Part II                        •   Accumulation of actual cost data and comparisons to estimated
                                              cost levels.
                                          •   Evidence that results for the phase (or results to date) have
                                              been compared against initial estimates for cost, schedule,
                                              performance, risk, and return.
                                          •   Documentation of the change between the current number and
                                              scope of requirements and the original requirements baseline
                                              established for the project.
                                          •   Documentation of the comparison between the current
                                              business conditions and assumptions and the project's initial
                                              assumptions and context.
                                          •   After the release of each new increment, each project
                                              participating in the increment should be analyzed to determine
                                              what interim benefits have been achieved in comparison to the
                                              previous increment.
                                          •   Documentation of differences between the actual performance
                                              of the software organization or contractor and their claims at
                                              the beginning of the project (e.g., schedule, costs, functionality,
                                              technical solutions, etc.).
                                          •   Aggregate data covering costs, benefits, and project
                                              performance for all IT projects in the investment portfolio.


Key questions to ask related to measures of interim results
The focus of these questions is on the actual data being examined by senior management as part of the investment
control process. In order to answer these questions, a sample of ongoing IT projects could be selected and used as
the unit of analysis.

               Are specific measures of performance being used to track costs, schedule, benefits, and
               risks? Are these measures updated throughout each project's life-cycle as costs, benefits,
               and risks become better known?

               Are data being used to track actual project performance (interim results) against estimates
               that were used to justify the project?


                                                            62
                                                                                                             Select   Control   Evaluate


                                                                                                 Process



                                                                                                   Data



                                                                                                 Decisions




     Are gaps or differences between estimates and actuals being analyzed and explanatory
     factors documented for positive or negative variances?

     Is there documentation to support that interim cost, schedule, benefit, and risk information
     has been validated or verified? If risks have changed, is this supported by documented
     data or analyses?


5.2 Updated                      The cost, benefit, schedule, and risk information that was included
Analyses of Each                 in the business case, including the various analyses that were
Project's Costs,                 conducted to justify the project, should be updated as project
Benefits, Schedule,              implementation continues and as dollar amounts increase. For
and Risks                        instance, it may have been difficult to precisely estimate costs and
                                 benefits when the project was first proposed, but such
                                 quantification may be improved as prototype and pilot project
Applicable criteria              results become available.

CCA 5122(b)(6)                   Information and analyses in the business case should also be
CCA 5123(3)                      updated to provide justification for adding additional functional
CCA 5125(c)(2)                   requirements to the project. This justification should weigh the
CCA 5126
                                 costs of adding the requirement late in the development process
OMB A-130 parts 8b(1),
                                 versus the anticipated benefits that are expected from the added
8b(2), 8b(3)
                                 functionality.

                                 Older versions of these analyses should be maintained for later
                                 comparisons and to feed lessons learned back in to the Selection
                                 phase.


Key questions to ask related to updated analyses of each project's costs, benefits,
schedule, and risks
The following questions address whether the organization is updating project cost, benefit, schedule, and risk
information as information changes and the project proceeds through development. To answer these questions, a
sample of ongoing projects could be selected as the unit of analysis.

     In investment control meetings, has the information in project business cases been updated
     to reflect the current state (including project costs to date, current risks and mitigation
     plans, interim benefit or performance results achieved, etc.)?


                                                       63
            Select   Control   Evaluate


Process



  Data



Decisions




               Are project-level data (current and historical) being maintained and updated using agency
               approved databases/information systems?

               Have any business assumptions or environmental factors (political, funding, stakeholder
               support) changed? If so, have the impacts of these factors on project outcomes been
               evaluated?

               If the project is behind schedule, have risk mitigation plans been updated and explanatory
               factors thoroughly evaluated?

               If contractor assistance is being utilized, have contractor performance reports been
               conducted and results made available for management consideration?




                                                         64
                                                                                                  Select   Control   Evaluate


                                                                                      Process



                                                                                        Data



                                                                                      Decisions




                                 Control--Decisions
  Key decision elements to be reviewed:

  6.1 Deciding whether to cancel, modify, continue, or accelerate a project
  6.2 Aggregating data and reviewing collective actions taken to date



Explanation of Decisions Made During the Control Phase

The primary focus of the Control phase should be on making project management decisions.
Actions should be taken quickly to address problems as they are identified and senior managers
should be actively involved in making decisions about all of the projects in the investment
portfolio. While many of these decisions will be implicit (the project is right on course, no
problems have been identified, requirements have remained the same and, thus, the decision will
usually be to continue the project as is), it is critical, nonetheless, that a conscious decision be
made about the future of each project.

6.1 Deciding                 As each project is reviewed at various stages in its life-cycle,
Whether to Cancel,           decisions should be made about the future of the project. These
Modify, Continue, or         decisions will be unique for each particular project and should be
Accelerate a Project         based on the particular merits of the project. In addition, some
                             explanation or documentation of the decision should be included.
Applicable Criteria          Even implicit decisions should have some documentation to show
                             that a conscious decision was made to continue the project.
CCA 5122(b)(1)
CCA 5122(b)(6)               Projects that have deficiencies or problems identified (actuals
CCA 5123(3)                  exceed estimated levels, risks are increasing, requirements have
CCA 5125(c)(2)
                             changed, etc.) should have a decision made by senior managers
CCA 5126
CCA 5127
                             about what to do with the project. Decisions will usually involve
FASA 10 USC 2220             one of four alternatives:
FASA 41 USC 263
EO 13011, Sec. 2(b)(3)       •    modify the project
OMB A-11, part 3             •    cancel the project
OMB A-130, part 8b(3)        •    continue the project as is

                                                65
            Select   Control   Evaluate


Process



  Data



Decisions




                                          •    accelerate the project's development

                                          Decisions may also be made to suspend funding or make future
                                          funding releases conditional on corrective actions being taken.
                                          These decisions should be documented, along with an explanation
                                          or criteria stating how funding can be reobtained. The decisions
                                          should also be reflected in budget information. For instance, if a
                                          project's development is halted while the feasibility of an alternative
                                          is assessed, budgetary spending information should reflect such a
                                          halt in funding. There should also be an explanation or documented
                                          criteria stating what must occur before funding is reinstated.

                                          In addition, depending on what decisions are made about a project,
                                          future "cascading" actions resulting from these decisions should be
                                          clearly identified and delineated. For instance, halting the
                                          development of a project will impact a number of other areas,
                                          including project management, personnel and staffing decisions,
                                          budget decisions and spending priorities, etc. These cascading
                                          actions should also be reviewed and monitored to ensure that
                                          money is not continuing to be spent and that all development
                                          activities have ceased.

                                          An independent review should be conducted prior to funding being
                                          reinstated to ensure that all corrective actions have been taken and
                                          to determine whether additional changes or modifications are still
                                          needed.


Key questions to ask related to deciding whether to cancel, modify, continue, or
accelerate a project
The purpose of these questions is to help determine whether management decisions and actions are being taken
using investment control processes and requisite project data. A sample of projects that have progressed through a
management investment control process can be used to help answer these questions.




                                                             66
                                                                                            Select   Control   Evaluate


                                                                                Process



                                                                                  Data



                                                                                Decisions




Is the organization reviewing projects and using data analysis to make decisions about the
future of each project? Were the decisions that were made reasonable given the situation
and condition of the project? For those projects whose status remained stable, was a
conscious decision made that the project should proceed or was it implicitly assumed that
the project would continue?

If problems were identified, was a decision made about what actions should be taken?
Who made this decision? Is there some explanation or justification given for this decision?
Were the actions that were taken appropriate to the problem (i.e., was the problem an IT
problem or a business/process problem)?

What evidence is there, if any, that actions have been taken based on the results of project
reviews? For instance, if the organization determined that new requirements have been
introduced, what actions were taken to respond to these additional requirements? Were
these actions documented? Did these actions fully address the requirements?

If decisions are made that affect a project's funding, such as suspending project funds or
cancelling a project, is there evidence in budget documents and spending information that
reflects this decision? Are there criteria identifying what must be done for funding to
resume?

Are future "cascading" actions resulting from project decisions clearly identified and
delineated?

Is a management group following agency policies, procedures, and practices in making
regular decisions about the continuation of major IT projects?

Are project data being used to make decisions and take action on IT projects that are
subjected to investment reviews?

Are decisions being made at the right time (i.e., as prescribed by the agency process or as
agreed to by senior management as part of project approval)?

Are decisions regarding projects being executed? Is accountability and follow-up clearly
defined?




                                           67
            Select   Control   Evaluate


Process



  Data



Decisions




              Is an independent review conducted afterward to ensure that actions were taken and to
              make further changes as necessary?

              If projects are allowed to proceed when investment review data and analyses raise serious
              questions about the project, has documentation been provided detailing how management
              reached its decision?


6.2 Aggregating                           A review of past activities and decisions made about a particular
Data and Reviewing                        project can influence both current and future managerial decisions.
Collective Actions                        This is a primary reason why aggregating information is important.
Taken to Date                             Aggregating allows trends to be more easily identified. Looking at
                                          projects across an agency or bureau, for example, can help pinpoint
Applicable criteria                       those divisions that have had repeated success at developing and
                                          managing IT projects, and those that have had more trouble. This
CCA 5122(b)(1)                            in turn can be used as inputs for decisionmakers when weighing
CCA 5122(b)(6)                            organizational capability risks and determining project review
CCA 5123(3)                               schedules.
CCA 5125(c)(2)
FASA 10 USC 2220
                                          Aggregating can also help as the organization looks to refine and
FASA 41 USC 263
EO 13011, Sec. 2(b)(3)                    improve the screening and selection criteria and performance
OMB A-11, part 3                          measures. Data can be aggregated by project, or can be grouped
OMB A-130 part 8b(3)                      along unit, divisional, bureau, or agency lines.

                                          Problems that are identified from this analysis may be serve as an
                                          indication of specific endemic weaknesses with project
                                          management, contractor oversight, or cost-estimation practices that
                                          need revision and corrective actions. In addition, positive trends
                                          that are identified can provide valuable lessons for highlighting and
                                          reinforcing organizational strengths.




                                                             68
                                                                                                              Select   Control   Evaluate


                                                                                                  Process



                                                                                                    Data



                                                                                                  Decisions




Key questions to ask related to aggregating data and reviewing the collective actions
taken to date
The following questions are focused on determining management's awareness of trends-positive and negative--that
the aggregate set of IT project investment reviews reveal.

     Has the organization aggregated data in order to assess organizational performance and to
     identify patterns or trends? At what levels--unit, division, agency, departmental--has
     information been aggregated? Is this information being fed back in to decisionmakers to
     help make future decisions?




                                                       69
            Select   Control   Evaluate


Process



  Data



Decisions




                                                Evaluate--Process
          Key process elements to be reviewed:

          7.1 Conducting post-implementation reviews (PIR) using a standard methodology
          7.2 Feeding lessons learned back in to the Selection and Control phases



Explanation and Definitions of Process Requirements for the Evaluation
Phase

The Evaluation phase "closes the loop" on the IT investment management process by comparing
actuals against estimates in order to assess performance and identify areas where future
decision-making can be improved. Lessons that are learned during the Evaluation phase should
be geared towards modifying future Selection and Control decisions. Central to this process is
the post-implementation review with its evaluation of the historical record of the project.


7.1 Conducting Post-                      Once a project has reached a final end point (e.g., the project is
implementation                            fully implemented, the project has been cancelled, etc.), a post-
Reviews (PIR) Using                       implementation review (or post-investment review) should be
a Standard                                conducted. This review will usually occur about 3 to 12 months
Methodology                               after a project has reached its final end point and should be
                                          conducted by a group other than the project development team to
                                          ensure that it is conducted independently and objectively.
Applicable criteria
                                          Organizations often spend significant time and resources focused
CCA 5112(c)                               on selecting IT projects, but less attention is given to evaluating
CCA 5122(b)(1)                            projects after they are implemented. Yet the information gained
CCA 5122(b)(6)                            from PIRs is critical for improving how the organization selects,
CCA 5125(c)(2)
                                          manages, and uses its IT resources.
CCA 5126
CCA 5127
PRA 44 USC 3506 (h)(5)                    Each PIR that is conducted should have a dual focus--it should
PRA 44 USC 3514                           (1) provide an assessment of the implemented project, including an
(a)(2)(D)                                 evaluation of the development process, and (2) indicate the extent


                                                            70
                                                                                             Select   Control   Evaluate


                                                                                 Process



                                                                                   Data



                                                                                 Decisions




EO 13011 Sec. 2(b)(3)     to which the organization's investment decision-making processes
GPRA 31 USC 1115          are sustaining or improving the success rate of IT projects. The
GPRA 31 USC 1116          following are three essential areas that should be evaluated as part
CFO 31 USC 902(a)(3)      of a complete PIR:
OMB A-130, parts 8b(1),
8b(3)
                          (1) Customers

                          Surveys should be conducted to determine users' satisfaction with
                          the end product. There should also be a focused look at how well
                          the project supports the organization's various business processes.

                          Many of the intangible benefits that were identified at the outset
                          will relate to how customers and end users feel about the final
                          project.

                          (2)   Mission/Program Impact

                          A close look should be taken to determine whether the implemented
                          system has achieved its intended impact, and whether this impact is
                          still aligned with mission goals. An assessment should also be made
                          of other project-specific aspects, such as an estimate of cost savings
                          that have been achieved, compliance with the information
                          technology architecture, evaluations of the information product
                          (accuracy, timeliness, adequacy, and appropriateness of
                          information), and identification of additional maintenance or
                          security issues.

                          (3) Technical Capability

                          Finally, an evaluation should be made of the technical aspects of the
                          project, both current and future. Such an evaluation may focus on
                          such factors as the competency of the workforce to use the new
                          system and employee satisfaction or retention, the extent to which
                          advanced technology was used, and the methodological expertise of
                          the development team.

                          To ensure that each project is evaluated consistently, the
                          organization should have a documented methodology for conducting

                                             71
            Select   Control   Evaluate


Process



  Data



Decisions




                                          PIRs. This methodology, which should be used at all organizational
                                          levels, should spell out roles and responsibilities for conducting
                                          reviews and for taking actions based on the results. PIRs should be
                                          required on a regular basis to ensure that completed projects are
                                          reviewed in a timely manner. The organization should also have
                                          policies or procedures that document how information from the
                                          PIRs is to be relayed back to decisionmakers.

                                          Finally, because there is a great deal of knowledge that can be
                                          gained from failed projects, evaluations should also be conducted
                                          for projects that were cancelled prior to being fully implemented.
                                          Although project accountability is important, these evaluations
                                          should focus on identifying what went wrong with the project, in
                                          order to learn from mistakes and minimize the chances of their
                                          being repeated.


Key questions to ask related to conducting post-implementation reviews using a
standard methodology
The purpose of these questions is to determine whether a defined, documented process is used to conduct post-
implementation reviews of specific IT projects. These questions can help determine whether an organized process
exists to conduct these reviews. Data and actual use of the process for making decisions will be covered in later
sections.

               Does the organization have a defined, documented process for conducting post-
               implementation reviews (PIR) of IT projects?
               •    Is the purpose(s) of the PIR process clearly explained and communicated?
               •    Is the process clear about when PIRs are to be conducted? Are PIRs required on a
                    regular basis to ensure that completed projects are reviewed in a timely manner?
               •    Does the process delineate roles, responsibilities, and authorities for people and
                    offices involved in conducting the PIRs? Does the process help ensure that these
                    assessments are objective?
               •    Does the process stipulate how conclusions and recommendations resulting from PIRs
                    are to be communicated to and reviewed by senior management?




                                                             72
                                                                                                 Select   Control   Evaluate


                                                                                     Process



                                                                                       Data



                                                                                     Decisions




     Does the organization have a standardized methodology for conducting PIRs? Does this
     methodology, at a minimum, include assessments of customer satisfaction,
     mission/programmatic impact, and technical performance/capability? Is the methodology
     required at all levels of the organization?

     Does the process define how the outcomes of PIRs are to be addressed? Are roles and
     responsibilities defined for taking action to address any concerns or problems that are
     identified?

     What steps does the organization require to ensure that PIRs are conducted independently
     and objectively? Are the results of the PIRs validated or verified?

     Are the causes of project and process problems identified as part of the PIR?


7.2 Feeding Lessons         All of the PIR information gained in the Evaluation phase should be
Learned Back in to          collected and maintained with project information gathered during
the Selection and           the other two phases. Developing this complete library of project
Control Phases              information helps to establish an organizational memory in which
                            both successes and failures can be used for learning.

Applicable criteria         There should be some mechanism or process to ensure that
                            information is being aggregated and fed back in to improve the
CCA 5122(b)(6)              investment management process. For instance, the cost, risk, and
CCA 5123(3)                 benefit criteria (including the weights given to each) for the
CCA 5125(c)(2)              Selection phase may be refined to ensure greater implementation
OMB A-130, part 8b(3)
                            success of future projects. The organization should also determine
                            whether there may be different or more appropriate cost, benefit,
                            and risk measures that could be established that would help better
                            monitor projects.




                                               73
            Select   Control   Evaluate


Process



  Data



Decisions




Key questions to ask related to feeding lessons learned back in to the Selection and
Control phases
The purpose of these questions is to determine whether the organization uses a defined process for deciding how to
take action to institutionalize knowledge, experience, and lessons learned from PIRs that are conducted.

               Does the organization's process or methodology for conducting PIRs include provisions for
               (1) changing or improving existing management decision-making processes and (2)
               strengthening project-level management?

               Does the organization have a mechanism for tracking (over time and for different kinds of
               projects) and aggregating the results of PIRs that are conducted?
               •    Are the results of PIRs collected and maintained (in a manual or automated database)?
                    Is the information reported in an timely manner? Are the results easily accessible?
               •    Has the organization identified roles and responsibilities for collecting and analyzing
                    PIR report information?

               Does the organization have procedures for regularly reporting PIR results to senior
               management? Is this information reliable, accurate, and easily accessible?

               Does the organization have procedures for regularly assessing the PIR process for
               completeness, quality, and contribution to project level and executive decision-making?




                                                         74
                                                                                                 Select   Control   Evaluate


                                                                                      Process



                                                                                       Data



                                                                                     Decisions




                                     Evaluate--Data
  Key data elements to be reviewed:

  8.1 Measurements of actual vs. projected performance
  8.2 Documented "track record" (project and process)




Explanation and Definitions of Data Requirements for the Evaluation
Phase

Data collected during the Evaluation phase will be primarily historical in nature focusing on the
outcome of a project as compared to executives' expectations for the project. In addition, ex
post information that is developed should include modifications made to the Selection and
Control phases, as well as the institutionalized lessons-learned information. This information
should be used to revise the Selection and Control phases and to help make future investment
decisions.


8.1 Measurements of         The focus of the PIR should be on evaluating a project's actual
Actual vs. Projected        results compared to estimates in terms of cost, schedule,
Performance                 performance, and mission improvement outcomes. An attempt
                            should also be made to determine the causes of major differences
Applicable criteria         between planned and end results. And the PIR should be used to
                            help identify any inappropriate systems development and project
CCA 5122(b)(5)              management practices.
CCA 5122(b)(6)
CCA 5123(3)                 The PIR should provide a wide range of information regarding both
CCA 5125(c)(2)
                            the project and the process for developing and implementing the
CCA 5126
                            project. Specific information includes the following:
GPRA 31 USC 1115
GPRA 31 USC 1116
CFO 31 USC 902(a)(3)        •    an assessment of the project's effectiveness in meeting the
OMB A-130, part 8b(3)            original objectives,


                                               75
             Select   Control   Evaluate


 Process



   Data



 Decisions




                                           •    an identification of benefits that have been achieved, an
                                                assessment of whether they match projected benefits, and a
                                                determination of reasons for any discrepancies.
                                           •    an evaluation of whether original business assumptions used to
                                                justify the project were valid,
                                           •    a comparison of actual costs incurred against projected costs,
                                           •    a determination of how well the project met time schedules
                                                and implementation dates,
                                           •    management and user perspectives on the project, and
                                           •    an evaluation of issues that still require attention.

                                           Outputs of the PIR should include user evaluations of the
                                           effectiveness of the project, actual costs broken out by category,
                                           measurements used to calculate benefits, a comparison matrix of
                                           actuals to estimates, and business-as-achieved documentation.


Key questions to ask related to measurements of actual vs. projected performance
The purpose of these questions is to determine the adequacy of data being used in conjunction with post-
implementation reviews. To answer these questions, a sample of completed PIRs might be evaluated.

              Is the organization collecting projected versus actual cost, benefit, and risk data as part of
              the post-implementation reviews?
              •    Has the cost, benefit, and risk information that was used for initial project justification
                   been preserved? Have updates that have been made to costs, benefits, or risks been
                   noted? Have these updates also been preserved?
              •    Have project benefits that were obtained been quantified? If not, are qualitative
                   measures being used to determine impact?
              •    Have the cost data been verified or validated? Are these data contained in a
                   recognized agency financial management/accounting database?

              Does the PIR include assessments of customer satisfaction (end-users, business or program
              unit sponsor, etc.)?

              Does the PIR include assessments of technical capability (e.g., conformance to recognized
              systems development methodology, architecture compliance, contractor performance and
              oversight)?


                                                              76
                                                                                                                 Select   Control   Evaluate


                                                                                                     Process



                                                                                                       Data



                                                                                                     Decisions




8.2 Documented                   The organization should be maintaining documentation of all
"Track Record"                   decisions, changes, actions, and results that occurred throughout
(Project and                     the project's life cycle, as well as other relevant project information,
Process)                         such as the business case and updated cost/benefit analyses. The
                                 organization should also be tracking recommendations (for both
Applicable criteria              improving the project and refining the overall investment
                                 management process) that arise out of the PIRs.
CCA   5122(a)
CCA   5122(b)(6)                 This "track record" will be invaluable for helping the organization
CCA   5125(c)(2)                 refine and improve its processes as more and more information is
CCA   5126
                                 collected and aggregated.


Key questions to ask related to documented "track record" (project and process)
The purpose of these questions is to determine whether aggregate data are being collected, analyzed, and
maintained to support improvements in project management and executive-level decision-making.

      Has the organization conducted trend analyses (descriptive or statistical) using the results
      of PIRs that have been conducted?
      •    Do these analyses attempt to correlate actual project performance results with factors
           that may have caused these results (positive and negative)?
      •    Are the results of these analyses presented to management as a regular part of the
           investment decision-making process? Are special reports issued to executive
           management?

      Are recommendations for specific projects and senior management decision-making
      processes presented in PIRs?
      •    Do these recommendations cover changes to process, data, or decision-making
           procedures used for both the Selection and Control phases?
      •    Are these recommendations well documented and supported by existing analyses?




                                                        77
            Select   Control   Evaluate


Process



  Data



Decisions




                                              Evaluate--Decisions
         Key decision elements to be reviewed:

         9.1 Assessing the project's impact on mission performance and determining
         future prospects for the project
         9.2 Revising the Selection and Control phases based on lessons learned




Explanation of Decisions Made During the Evaluation Phase

A number of key decisions will be made during the Evaluation phase, including an assessment of
how well the project met its intended objectives, a determination of what changes or
modifications to the project are still needed, and an identification of ways to modify or improve
the overall investment management process to better maximize results and minimize risks. In
addition, the organization may assess the overall performance of its IT investments in improving
mission performance. To make these decisions, agency executives must gauge the degree to
which past decisions have influenced the outcome of IT projects, understand why these
decisions had the effect that they did, and determine how changing the processes for making
decisions could create a better outcome for current IT projects and future IT proposals.


9.1 Assessing the                         The results and recommendations that arise out of the PIRs,
Project's Impact on                       combined with other project information, are a critical input for
Mission Performance                       senior decisionmakers to use to assess the project's impact on
and Determining                           mission performance. In making this assessment, senior managers
Future Prospects for                      will need to ask a number of questions about the project, including
the Project                               the following:

                                          •    How effective was the project in meeting the original
Applicable criteria                            objectives? Are these objectives still valid?
                                          •    Were the original business assumptions used to justify the
CCA 5122(a)(1)                                 project valid?
CCA 5122(b)(3)                            •    What is the current status of the system?
CCA 5122(b)(5)

                                                            78
                                                                                                                  Select   Control   Evaluate


                                                                                                      Process



                                                                                                        Data



                                                                                                      Decisions




CCA   5123(3)                     •     Are further changes or modifications necessary?
CCA   5123(4)
CCA   5123(5)                     Even after a project has been implemented, decisions should be
CCA   5125(c)(2)                  made on a regular basis about the status of the project. Senior
CCA   5126                        managers should regularly question whether (1) the current system
                                  meets organizational needs, (2) the system should be modified to
                                  better meet these needs, (3) a new system is needed to best meet
                                  these needs, or (4) the needs could best be met by outsourcing the
                                  work.

                                  In addition, because operation and maintenance (O&M) costs, for
                                  such activities as hardware upgrades, system software changes, and
                                  ongoing user training, can consume a significant amount of IT
                                  resources (some have estimated that ownership costs, operations
                                  and maintenance costs, and disposition costs can consume as much
                                  as 80 percent of a project's total life-cycle costs), a plan should be
                                  developed for the continued support and operation of each IT
                                  project.


Key questions to ask related to assessing the project's impact on mission performance
and determining future prospects for the project
The purpose of these questions is to help assess decisions that the organization has made regarding its evaluation of
the impact that projects have had on mission performance as well as the continued direction being given for these
projects.

      What decisions have been made by senior management (or investment review group)
      regarding whether implemented projects met defined criteria for successful outcomes?

      Were corrective actions for specific projects included in senior management's decisions?
      •   Were timetables and steps for implementing these changes established as part of the
          decision?
      •   Were follow-up management reviews established? Has a clear purpose for these
          reviews been defined?

      Has a plan been developed detailing how future O&M and disposition costs will be
      addressed?



                                                         79
            Select   Control   Evaluate


Process



  Data



Decisions




                Are decisions that are being made on specific projects cognizant of the potential (or actual)
                impact the decision may have on other related projects?

                Have decisions regarding the status of projects been finalized?
                •   Have expected changes been communicated to the project manager?


9.2 Revising the                          An organization's investment management process will usually not
Selection and                             remain static, but will evolve and change over time as the
Control Phases                            organization learns more and more about what has been successful
Based on Lessons                          and what still needs to be improved. Modifications that may be
Learned                                   made to the process include the following:

Applicable criteria                       •    changing the mixture of members on the organization's
                                               decision-making investment review group
CCA 5112(c)                               •    changing the Selection phase decision-making criteria (both
CCA 5122(b)(1)                                 screening and ranking criteria)
CCA 5122(b)(3)                            •    changing the Control phase criteria used for monitoring the
CCA 5122(b)(5)
                                               progress of projects
CCA 5123(3)
CCA 5125(c)(2)
                                          •    modifying the time frames for reviewing projects during the
CCA 5126                                       Control phase
GPRA 31 USC 1116                          •    modifying the triggers for identifying projects for review
OMB A-127, Para. 6,7                      •    modifying the PIR methodology
OMB A-123, Part II
                                          The results from one project will often not provide enough
                                          information to allow significant modification to be made to the
                                          agency's IT decision-making processes. However, a significant,
                                          recurring system development problem found across multiple
                                          projects over time would be cause for refining or even
                                          significantly revising the organization's decision-making
                                          processes and criteria.

                                          The causes for differences between planned and actual results
                                          should be determined and corrective actions to the overall IT
                                          management process, decision criteria, etc. should be identified and
                                          documented. Once the causes for differences between planned and
                                          actual results have been determined, steps should be taken to


                                                            80
                                                                                                              Select   Control   Evaluate


                                                                                                  Process



                                                                                                    Data



                                                                                                  Decisions




                                 address these causes in order to ensure greater success in the
                                 future.

                                 All alterations or updates that are made to the Selection and Control
                                 phases, based on the results of PIRs, should be documented.


Key questions to ask related to revising the Selection and Control Phases based on
lessons learned
The following questions are focused on assessing whether the organization has made changes to the IT investment
management process based on lessons that were learned.

     What decisions have been made to modify existing organizational IT investment
     management processes?
     •   Have these decisions been communicated to staff?
     •   Are changes to existing processes, operating procedures, and data requirements
         aligned with conclusions and recommendations documented in PIRs?
     •   Has the organization clearly established and communicated when these changes to
         existing management processes will take effect?




                                                       81
                                         SECTION 4

                                 RELATED LEGISLATION AND
                                EXECUTIVE BRANCH GUIDANCE

LEGISLATION

Congress has passed several pieces of legislation that lay the groundwork for agencies to
establish an investment approach for managing IT. The following are five key pieces of
legislation that put in place various requirements related to the IT investment management
process:

     The   Clinger-Cohen Act of 1996 (CCA)
     The   Paperwork Reduction Act of 1995 (PRA)
     The   Federal Acquisition Streamlining Act of 1994 (FASA)
     The   Government Performance and Results Act of 1993 (GPRA)
     The   Chief Financial Officers Act of 1990 (CFO)

Notes:     Brief summaries of the legislative provisions are cited; readers should consult
           copies of the statutes themselves for actual language contained in the law.

           The phases of the IT investment process that are most related to each legislative
           provision follow each provision. The phase symbols are as follows:

           Selection phase --

           Control phase --

           Evaluation phase --


The Clinger-Cohen Act of 1996 (CCA) (formally the Information Technology
Management Reform Act, Division E of Public Law 104-106)

The Clinger-Cohen Act requires federal agencies to focus more on the results achieved
through IT investments while streamlining the federal IT procurement process. Specifically
this act introduces much more rigor and structure into how agencies approach the selection
and management of IT projects. Among other things, the head of each agency is required to
implement a process for maximizing the value and assessing and managing the risks of the
agency's IT acquisitions.

Specific sections of the Clinger-Cohen Act related to IT investments include:




                                               82
CCA 5112(b) - The OMB Director is to promote and be responsible for improving
the acquisition, use, and disposal of IT to improve the productivity, efficiency, and
effectiveness of federal programs.



CCA 5112(c) - At the same time the President submits the budget to the
Congress, the OMB Director is to submit a report to the Congress on the net
program performance benefits achieved as a result of agencies' major capital
investments in information systems and how the benefits relate to the achievement
of agency goals.



CCA 5112(c) - The OMB Director is to develop, as part of the budget process, a
process for analyzing, tracking, and evaluating the risks and results of all major
capital investments for information systems by the federal agencies. This process is
to cover the life of each system and include explicit criteria for analyzing the
projected and actual costs, benefits, and risks associated with the investments.



CCA 5113(b)(1) - The OMB Director shall evaluate the IRM practices of executive
agencies with respect to the performance and results of IT investments.
(b)(4) - The Director shall implement periodic reviews of selected information resources
management activities of the executive agencies through the budget process.



CCA 5113(b)(5) - Specific actions that may be taken by the Director of OMB to
enforce agency accountability for its information resources management and
investments made in information technology include (1) recommending increases
or reductions in the agency's IRM budget, (2) using administrative controls to
restrict the availability of agency funds, and (3) designating an executive agent to
contract with private sources for the agency's management and acquisition of
information technology.




                                              83
CCA 5122 - Agency heads are to design and implement a process for
maximizing the value and assessing and managing the risks of their IT
acquisitions; the process (among other things) is to provide for the selection of
investments using minimum criteria on whether to undertake an investment
(including quantitatively expressed projected net, risk-adjusted return on investment, and
specific quantitative and qualitative criteria for comparing and prioritizing
alternative information systems projects) and to provide a means for senior
management to obtain timely information regarding progress (at established
milestones) in terms of cost, capability of the system to meet requirements,
timeliness, and quality.



CCA 5122(b)(2) - The IT investment process of executive agencies is to be
integrated with the processes for making budget, financial, and program
management decisions.



CCA 5123(3) - Agency heads shall ensure that performance measurements are
prescribed for IT used by or to be acquired for the agency; the performance measurements
are to measure how well the IT supports agency programs.



CCA 5123(5) - Agency heads are to analyze the missions of the agency and,
based on the analysis, revise the agency's mission-related and administrative
processes (as appropriate) before making significant investments in IT used to
support those missions.



CCA 5125(b)(1) - The agency CIO is responsible for providing advice and other
assistance to agency heads and other senior managers to ensure that IT is
acquired and information resources are managed for the agency in a manner that
implements the policies and procedures of this act, consistent with the Paperwork
Reduction Act, and the priorities of the agency head.



CCA 5125(b)(2) - The agency CIO is responsible for developing, maintaining, and
facilitating the implementation of a sound and integrated IT architecture for the
agency; the architecture is an integrated framework for evolving or maintaining
existing IT and acquiring new IT to achieve the agency's strategic and IRM goals.

                                             84
CCA 5125(c)(2) - The agency CIO is to monitor the performance of IT programs
of the agency, evaluate the performance of those programs on the basis of
applicable performance measures, and advise the agency head regarding whether
to continue, modify, or terminate the program or project.



CCA 5126 - The agency head, in consultation with the CIO and Chief Financial
Officer (or comparable official) is to establish policies and procedures that (1)
ensure that accounting, financial, and asset management systems and other
information systems are designed, developed, maintained, and used effectively to
provide financial or program performance data for the agency's financial
statements; (2) ensure that financial and related program performance data are
provided on a reliable, consistent, and timely basis to agency financial
management systems; and (3) ensure that the financial statements support the
assessment and revision of agency processes and performance measurement.



CCA 5127 - The agency head shall identify in the agency's IRM plan (required by
PRA) major IT acquisition programs, or phase or increment of such program,
that has significantly deviated from the cost, performance, or schedule goals
established for the program.



CCA 5202 - The head of the agency should, to the maximum extent practicable,
use modular contracting for the acquisition of major information technology
systems. (This section also describes key characteristics of "modular contracting,"
including (1) that the system be acquired in successive acquisitions of
interoperable increments complying with common or commercially accepted IT
standards, (2) that contracts should be awarded within 180 days after the date on
which the solicitation is issued, and (3) the information technology should be
delivered within 18 months after the date on which the solicitation resulting in
award of the contract was issued.)




                                             85
Paperwork Reduction Act of 1995 (PRA)
(Public Law 104-13)

The Paperwork Reduction Act of 1995 (PRA) is the "umbrella" IT legislation for the federal
government, with other statutes elaborating on the goals contained within PRA. PRA
requires agencies to use information resources to improve the efficiency and effectiveness of
their operations and fulfillment of their missions.

Specific sections of PRA related to IT investments include:

PRA 44 USC 3502(7) - Definition of IRM: the process of managing information resources to
accomplish agency missions and improve agency performance.


PRA 44 USC 3506(a)(4) - Agency program officials, in consultation with the Chief
Information Officer and Chief Financial Officer (or comparable official), are to define
program information needs and develop strategies, systems, and capabilities to
meet those needs.



PRA 44 USC 3506(b)(2) - Each agency is to develop and maintain a strategic IRM plan on
how IRM activities help accomplish agency missions (the plan is to include plans
for reducing information burdens imposed on the public, for enhancing public
access to and dissemination of government information, and for meeting the
information technology needs of the government).



PRA 44 USC 3506(b)(3)(A) - each agency is to maintain an ongoing process to ensure that
IRM operations and decisions are integrated with organizational planning, budget,
financial management, human resources management, and program decisions.



PRA 44 USC 3506(b)(3)(C) - agencies are to establish goals for IRM to improve the
productivity, efficiency, and effectiveness of agency operations and methods for
measuring progress in achieving the goals.




                                              86
PRA 44 USC 3506(h)(5) - Agencies are to maximize the value and assess and manage the
risks of major information system initiatives through a process that (a) integrates
budget, financial, and program management decisions and (b) is used to select,
control, and evaluate the results of the initiatives.




PRA 44 USC 3514(a)(2)(D) - the OMB Director is to provide an annual report to the
Congress on (among other things) the extent to which agencies have improved program
performance and the accomplishment of agency missions through IRM.




Government Performance and Results Act of 1993 (GPRA)
(Public Law 103-62)

GPRA requires agencies to set goals, measure performance, and report on their
accomplishments. As such, an agency's IT investments should directly support the
accomplishment of these goals

The specific sections of GPRA related to IT investments include:

GPRA 5 USC 306 - By September 30, 1997, agency heads are to submit to OMB and the
Congress a strategic plan for their program activities, including a comprehensive
mission statement covering major agency functions and operations.



GPRA 31 USC 1115 - Starting with fiscal year 1999, agencies are to prepare annual
performance plans covering each program activity set forth in the budget. The plans are to
establish performance goals in objective, quantifiable, and measurable form and performance
indicators to be used in measuring relevant outputs, service levels, and outcomes of each
program activity.




                                             87
GPRA 31 USC 1116 - No later than March 31, 2000, and annually thereafter, agency heads are
to prepare and submit to the President and the Congress program performance reports
setting forth performance indicators and comparing actual program performance against the
performance goals.




The Federal Acquisition Streamlining Act of 1994 (FASA)
(Public Law 103-355)

Title V of FASA requires agencies to define cost, schedule, and performance goals
for federal acquisition programs (to include IT projects) and monitor these
programs to ensure that they remain within prescribed tolerances. If a program falls out of
tolerance, FASA requires the agency head to review, take necessary actions, and, if
necessary, terminate the program.

Specific sections of FASA related to IT investments include:

FASA 10 USC 2220 - The Secretary of Defense shall approve or define the cost,
performance, or schedule goals for major defense acquisition programs and for
each phase of the acquisition cycle; whenever the Secretary determines that major
defense acquisition programs are not achieving, on average, 90 percent of the
cost, performance, and schedule goals established, the Secretary shall ensure that
there is timely review of the program and identify suitable actions (including
termination) to be taken with respect to the program.



FASA 41 USC 263 - The head of each executive agency shall approve or define
the cost, performance, and schedule goals for major agency acquisition programs
(congressional policy is that each agency should achieve, on average, 90 percent
of the cost and schedule goals established for major and non-major programs of
the agency without reducing the performance or capabilities of the items being
acquired); whenever necessary to implement the congressional policy,
agency heads are to determine whether there is a continuing need for the
programs that are significantly behind schedule, over budget, or not in compliance
with the performance or capability requirements and identify suitable actions,
including termination, to be taken.




                                             88
Chief Financial Officers Act of 1990 (CFO)
(Public Law 101-576)

Having accurate financial data is critical to understanding the costs and
assessing the returns on IT investments. The CFO Act focuses on the need to significantly
improve the financial management and reporting practices of the federal government

Specific sections of the CFO Act related to IT investments include:

CFO Act 31 USC 501 - Purpose statements: (1) provide for improvement, in
each agency of the federal government, of systems of accounting, financial
management, and internal controls to ensure the issuance of reliable financial
information and to deter fraud, waste, and abuse of government resources; and (2)
provide for the production of complete, reliable, timely, and consistent financial
information for use by the Executive Branch and the Congress in the financing,
management, and evaluation of federal programs.



CFO Act 31 USC 902(a)(3) - The agency Chief Financial Officer shall develop and
maintain an integrated agency accounting and financial management system,
including financial reporting and internal controls, that provides for (1) complete,
reliable, consistent, and timely information which is prepared on a uniform basis
and that is responsive to the financial information needs of agency management;
(2) the development and reporting of cost information; (3) the integration of
accounting and budgeting information; and (4) the systematic measurement of
performance.



EXECUTIVE BRANCH GUIDANCE

In addition to these legislative provisions, OMB and the White House have also issued several
pieces of guidance related to the acquisition and management of information resources. This
executive branch guidance includes

OMB Circular A-11
OMB Circular A-94
OMB Circular A-109
OMB Circular A-123
OMB Circular A-127
OMB Circular A-130
Executive Order 13011 Sec. 2(b)(3)
OMB Memorandum M-97-02

                                              89
OMB Circular A-11 provides detailed instructions and guidance on the preparation and
submission of agency budget requests and related materials, including program performance
information. Part 2 of the Circular provides specific instructions on the preparation and
submission of agency strategic plans, as required by GPRA. Part 3 provides guidance on the
planning, budgeting, and acquisition management of major fixed assets and requires agencies
to provide information on all major fixed asset projects included in their budget submissions
to OMB.



OMB Circular A-94 provides general guidance for conducting cost-benefit and
cost-effectiveness analyses. This guidance serves as a checklist for determining whether an
agency has considered and included all necessary elements for sound cost-benefit and
cost-effectiveness analyses. The circular also provides specific guidance on the discount
rates to be used in evaluating federal programs whose benefits and costs are distributed over
time.



OMB Circular A-109 establishes policies for acquiring major systems. Major systems are
defined as those programs that are critical to fulfilling an agency mission, entail the
allocation of relatively large resources, and warrant special management attention. Among
other requirements, the circular requires that an agency acquiring a major system to (1)
ensure that the system fulfills a mission need, (2) make appropriate trade-offs among
investment costs, ownership costs, schedules, and performance characteristics, (3) ensure
adequate system testing and evaluation, (4) accomplish system acquisition planning, built on
an analysis of agency missions, (5) tailor an acquisition strategy for each program, (6)
maintain a capability to predict, review, assess, negotiate, and monitor system life-cycle
costs, and (7) assess cost, schedule, and performance experience against predictions for
consideration at key decision points.



OMB Circular A-123 provides guidance on establishing, assessing, correcting, and reporting
on management and internal controls. Part II provides a definition of management controls
and establishes guidance for designing management structures that help ensure
accountability for results as federal agencies develop and execute strategies for implementing
or reengineering agency programs and operations.




                                             90
OMB Circular A-127 prescribes policies and standards for developing, operating, evaluating,
and reporting on financial management systems. Part 6 of the circular lays out policy
guidance for establishing governmentwide financial systems and compatible agency systems.
Specifically, these systems are to provide complete, reliable, consistent, timely, and useful
financial management information on federal government operations to enable central
management agencies, individual operating agencies, divisions, bureaus, and other subunits to
carry out their fiduciary responsibilities; deter fraud, waste, and abuse of federal government
resources; and facilitate efficient and effective delivery of programs through relating financial
consequences to program performance. Part 7 defines the specific requirements that
financial management systems should have in place to meet the policy requirements
established in part 6.



OMB Circular No. A-130 provides uniform governmentwide information resources
management policies as required by the Paperwork Reduction Act. Section 7 of the circular
describes basic considerations and assumptions, while sections 8(a) and 8(b) describe
information management policy and information systems and IT management policy,
respectively.

Section 8b(1)--Evaluation and Performance Measurement. Agencies are to promote the
appropriate application of federal information resources by (1) seeking opportunities to
improve the effectiveness and efficiency of government programs through work process
redesign and the judicious application of information technology; (2) preparing and updating
a cost-benefit analysis for each information system as necessary throughout its life cycle; (3)
conducting cost-benefit analyses to support ongoing management oversight processes; and
(4) conducting post-implementation reviews of information systems to validate estimated
benefits and document effective management practices.



8b(2)--Strategic Information Resources Management (IRM) Planning: Agencies are to
establish and maintain (1) strategic information resources management planning that
addresses how the management of information resources promotes the fulfillment of an
agency's mission; (2) information planning that promotes the use of information throughout
its life cycle to maximize the usefulness of the information, minimize the burden on the
public, and preserve the appropriate integrity, availability, and confidentiality of information;
and (3) operational information technology planning that links information technology to
anticipated program and mission needs, reflects budget constraints, and forms the basis for
budget requests. An agency's IRM planning is also to coordinate with other agency planning
processes including strategic, human resources, and financial resources.




                                               91
8b(3)--Information Systems Management Oversight: Agencies are to establish information
system management oversight mechanisms that (1) ensure that each information system
meets agency mission requirements; (2) provide for periodic review of information systems;
(3) ensure that the official who administers a program supported by an information system is
responsible and accountable for the management of that information system throughout its
life cycle; (4) provide for the appropriate training for users of federal information resources;
(5) ensure that federal information system requirements do not unduly restrict the
prerogatives of state, local, and tribal governments; (6) ensure that major information
systems proceed in a timely fashion towards agreed-upon milestones in the information
system's life cycle, meet user requirements, and deliver intended benefits to the agency and
affected publics; and (7) ensure that financial management systems conform to the
requirements of OMB Circular A-127.



8b(4)--Use of Information Resources: Agencies are to create and maintain
management and technical frameworks for using information resources that document
linkages between mission needs, information content, and information technology
capabilities. These frameworks should guide both strategic and operational IRM planning.
They should also address steps necessary to create an open systems environment.

Among other requirements, agencies are to (1) develop information systems in a manner that
facilitates necessary interoperability, application portability, and scalability of computerized
applications across networks of heterogeneous hardware, software, and communications
platforms, (2) ensure that improvements to existing information systems and the
development of planned information systems do not unnecessarily duplicate information
systems available within the same agency, from other agencies, or from the private sector,
and (3) establish a level of security for all information systems that is commensurate with
the risk and magnitude of the harm resulting from the loss, misuse, or unauthorized access to
or modification of the information contained in these information systems.



8b(5)--Acquisition of Information Technology: Agencies are to acquire information
technology in a manner that makes use of full and open competition, that maximizes return
on investment, and that considers the need for accommodations of accessibility for
individuals with disabilities to the extent that needs for such access exist. In addition,
off-the-shelf software from commercial sources is to be acquired, unless the cost-
effectiveness of developing custom software to meet mission needs is clear and has been
documented. Finally, all information technology is to be acquired in accordance with OMB
Circular A-109, where appropriate.




                                              92
Executive Order 13011, "Federal Information Technology," highlights the need for executive
agencies to significantly improve the management of their information systems, including the
acquisition of information technology, by implementing the relevant provisions of PRA, the
Clinger-Cohen Act, and GPRA. Agencies are to refocus their information technology
management to directly support their strategic missions, implement an investment review
process that drives budget formulation and execution for information systems, and rethink
and restructure the way they perform their functions before investing in information
technology to support that work. Agency heads are to strengthen the quality and decisions
of employing information resources to meet mission needs through integrated analysis,
planning, budgeting, and evaluation processes.

Section 2(b)2 makes agency heads responsible for establishing mission-based performance
measures for information systems investments that are aligned with agency performance
plans prepared pursuant to GPRA. Section 2(b)3 makes agency heads responsible for
establishing agencywide and project-level management structures and processes that are
responsible and accountable for managing, selecting, controlling, and evaluating investments
in information systems. Agency heads also have the authority to terminate information
systems when appropriate.



OMB Memorandum M-97-02 "Funding Information Systems Investments": This memo
establishes eight decision criteria that OMB will use, starting with fiscal year 1998 budget
proposals, to evaluate major information system investments proposed for submission in the
President's budget. The first four decision criteria describe criteria related specifically to
capital planning. The fifth criterion establishes the critical link between planning and
implementation--the information architecture--which aligns technology with mission goals.
The last three criteria establish risk management principles that are intended to help provide
assurance that the proposed investment will succeed.




                                              93
 APPENDIX I                                                                           APPENDIX I

                                         APPENDIX I

                               PROPOSED STRUCTURE FOR
                                 PRESENTING RESULTS

 The following reporting structure is based on a previously completed evaluation of one
 organization's IT investment decision-making.


                STRENGTHS                                        WEAKNESSES


OVERALL

Implemented partial investment management         Meager evidence of business results from
process:                                          past and existing IT investments.

• Created IRB and executive involvement           Critical flaws in implementation still exist:
• Established investment decision criteria
• Started post-implementation reviews             • lack of quality data used in investment
                                                    decisions
Plans established to address remaining            • inadequate quantitative metrics (ROI,
investment process elements.                        benefit/cost, and risk analysis)
                                                  • low proportion of post-implementation
Significance: A foundation has been put in          reviews to new investments
place for a complete investment process.          • lack of smaller, more manageable project-
                                                    level investment controls
                                                  • inadequate tradeoffs of maintenance of
                                                    existing systems to new investments

                                                  Process is not institutionalized.

                                                  Significance: Until process is complete,
                                                  congressional funding of IT capital
                                                  investment faces unnecessary risks. Until
                                                  institutionalized, investment management
                                                  will be sporadic and variable in character
                                                  across a massive IT modernization
                                                  program.




                                             94
 APPENDIX I                                                                            APPENDIX I


                STRENGTHS                                            WEAKNESSES


SELECTION

Good start on criteria, includes right                 Few current IT investments have the
categories for decision criteria (cost, risk,          minimal data requirements (ROI,
benefit) and requirements for business case            benefit/cost, risk analysis).
and benefit/cost analyses.
                                                     Significance: Poor quality information is
Significance: Ensuring a more                        producing poor quality decisions;
comprehensive analysis of projects than used         acceptance of unjustified projects and
in the past.                                         misjudging of project costs, benefits, and
                                                     risks.
Have involved executive management team,
including field representatives and                    Project-level cost and benefit data, in
departmental executives.                               many cases, do not exist.

Significance: Selected top-level management          Significance: Perception of control does
is engaged and learning from using an                not match reality (large-size projects create
investment-oriented, decision-making                 risk and instability).
approach.
                                                     Heavy reliance on qualitative, judgmental
IRB is meeting frequently and making                 data and analysis at the expense of more
decisions.                                           quantitative measures, especially ROI.

Significance: IRB is becoming a visible              Significance: Decisions are weighted more
decision-making entity with vested authority         toward personal judgments without
and responsibility for IT investment                 adequate consideration of critical,
decisions.                                           objective, independently verifiable facts
                                                     about measurable mission impact.
All modernization projects placed under a
single authority.                                    Inadequate tradeoffs among maintenance
                                                     and new development projects.
Significance: Single focal point for
accountability for resource allocations of           Significance: Inefficient allocation of
capital IT investments.                              resources.

                                                        = key deficiencies




                                                95
 APPENDIX I                                                                            APPENDIX I



                STRENGTHS                                         WEAKNESSES


CONTROL

Assessments of cost, schedule, and                  IT projects remain too large in size and
performance are being made through program          scope.
control meetings.
                                                    Significance: Increases cost and schedule
Significance: Mechanism in place to engage          risks beyond existing management and
program sponsors on status of technology            technical capabilities of the organization.
modernization projects.
                                                    More rigor needed in decisions being made
Targeting go/no-go decisions at different           at critical milestones ("tight gates" with no
stages of project life-cycle.                       exceptions).

Significance: Regular evaluations of cost,          Significance: Some projects allowed to
schedule, and some performance at critical          proceed despite limitations in quality of
project milestones.                                 analysis.

Consideration of internal and external audit        Lack of systematic risk management
findings.                                           processes.

Significance: Attempts being made to be             Significance: Organization taking risks
responsive to technical and managerial              beyond capability to address or manage
problems key to long-term improvement.              them.

                                                    Better coordination needed between BPR
                                                    efforts and IT investment process.

                                                    Significance: Business line reengineering
                                                    efforts are not aligned with major
                                                    modernization components which raises
                                                    questions about long-term process
                                                    improvement.




                                               96
 APPENDIX I                                                                       APPENDIX I



                STRENGTHS                                       WEAKNESSES


EVALUATION

Post-implementation review approach has            Lack of proportional management
been revised and is being used on projects.        attention and emphasis on completing
                                                   post-implementation reviews, despite
Significance: Provides an evaluation               having already made heavy existing and
mechanism to compare expected results with         new investments.
actual outcomes.
                                                 Significance: Track record does not exist
Two post-implementation reviews now              to justify new large-scale IT investments.
complete, with four more scheduled for near-
term completion.                                 Complete evaluations (cost, schedule, AND
                                                 completion performance) not being used at
Significance: Creating factual, verifiable       each project milestone.
milestone evidence on results of IT
investments.                                     Significance: Systems being deployed with
                                                 incomplete evidence on costs versus
Initial efforts being made to learn from post-   benefits.
implementation reviews.
                                                   Learning not widespread or quickly
Significance: Beginning to show propensity         translated into revised decision-making
to start learning from past mistakes.              processes.

                                                 Significance: Slow learning rate equals
                                                 slow improvement rate.



                                                   = key deficiencies




                                         97
APPENDIX II                                                                    APPENDIX II

                                      APPENDIX II

                  EXAMPLE OF ONE ORGANIZATION'S CRITERIA
                   FOR COMPARING AND RANKING PROJECTS


  A. RISK (relative weight = 20 points) This category measures the risk resulting from
  uncertainty. The more risk carried by the project or system, the lower the score (except
  for the risk of not doing the project).

  A.1. Schedule Risk (4 of 20 points) Evaluate the probability this project can be
  completed on schedule. Score from zero to four points based on where the project best
  fits on a scale from very risky to low risk. Key characteristics of both ends of the scale
  are as follows:

    Zero Points: Very risky. Execution of project is likely to slip; acquisition strategy
    indicates contract may not be awarded in time to meet schedule or obligate budget
    year dollars. Project staff is limited in size and/or experience and project is complex.
    An accelerated project schedule was imposed rather than developed from project
    planning.

    Four Points: Low risk. Execution of project is not likely to slip; acquisition strategy
    should result in timely contract award such that funds can be obligated as planned.
    Adequate project staff is available and has requisite experience to execute the project;
    project is not complex. Project schedule has not been accelerated to meet artificial
    deadlines.

  A.2. Cost Sensitivity. (4 of 20 points) Evaluate the sensitivity or quality of the cost
  estimates. Score from zero to four points based on the scale described below.

    Zero Points: Very risky. Project is complex and cost estimates appear to require
    additional refinement. Software development is required and represents more than 50
    percent of the predicated cost.

    Four Points: Low risk. Cost estimates are well supported. Little software
    development required or a software cost estimating technique has been used to
    produce a reasonably reliable cost estimate.




                                              98
APPENDIX II                                                                     APPENDIX II

  A.3. Technical Risk. (4 of 20 points) Evaluate the risk to complete the system from a
  technical point of view. Score from zero to four points based on the scale described
  below.

    Zero Points: Very risky. Hardware and/or software solution does not conform to
    organization's technical architecture and/or there is little or no experience with this
    technology in the organization. Hardware, software, or support is not now available
    commercially and requires development specifically for the organization.

    Four Points: Low risk. Planned hardware and software conform to organization's
    technical architecture and there is successful experience in using this technology in the
    organization. Hardware, software, and support are commercially available and do not
    have to be developed for use in the organization.

  A.4. Organizational Risk. (4 of 20 points) Assess the risk that the proposed system
  will fail due to organizational disruption. Score from zero to four points based on the
  scale described below.

    Zero Points: Very risky. System implementation requires significant organizational
    change, process redesign and/or people's jobs to be done differently and the project is
    not proactively seeking to mitigate this risk.

    Four Points: Low risk. System has little impact on the organization or the project is
    mitigating this risk through training and/or investment in a business process redesign
    effort which builds commitment to the system.

  A.5. Risk of Not Doing It. (4 of 20 points) Assess the risk to the organization of not
  proceeding with this project. Score from zero to four points based on the scale described
  below.

    Zero Points: Low risk. This in a incremental improvement to an existing system. The
    impact of this system can be achieved by other means.

    Four Points: Very risky. This system is important to provide future opportunities for
    cost savings and/or much improved customer service. If this system is not built or is
    delayed for a year, the organization will probably fail to meet customer demands in the
    near future.




                                              99
APPENDIX II                                                                     APPENDIX II

  B. ORGANIZATIONAL IMPACT (relative weight = 10 points) Measures the impact
  on organizational personnel of the system. The more favorable the impact on the
  organization the higher its score.

  B.1. Personnel and Training. (3 of 10 points) Assess the impact of the system on the
  knowledge, skill, and training of organizational personnel if the system in implemented.
  Score from zero to three based on the scale below.

    Zero Points: System is likely to require significant new skills to operate and support
    and the project does not appear to mitigate this impact through appropriate training,
    changes in rating qualifications, etc.

    Three Points: System is an improvement to an existing system and will require
    relatively little new skill and/or knowledge to operate or support. If it is a new system,
    it will introduce valuable new skills and knowledge to the organization and the project
    will mitigate any adverse impact through appropriate training, planning for rating
    qualification changes, etc.

  B.2. Scope of Beneficiaries/ Cross-Functional. (4 of 10 points) Assess a higher
  score (zero to four) the broader the scope of beneficiaries.

    Zero Points: Limited number of beneficiaries. This system will be used by only one
    office in headquarters, a single area, or district. Not a cross-functional system.

    Four Points: System is cross-functional and serves a number of offices, areas, and/or
    districts. System will be used by a large number of organizational units. System will
    be used by the public.

  B.3. Quality of Work Life. (3 of 10 points) Measures the improvement in quality of
  work life expected for the systems. Score higher (zero to three points) the more work
  life improvement is expected.

    Zero Points: Little if any positive impact on the quality of work life. System may
    increase the work required (e.g., additional data entry).

    Three Points: Positive contribution to the quality of work life will clearly result. For
    example, the system will improve medical care for dependents or allow a job to be
    done much faster such that job satisfaction will increase.




                                             100
APPENDIX II                                                                    APPENDIX II

  C. MISSION EFFECTIVENESS (relative weight = 20 points) Measures the impact
  of the system on both external and internal customers. It is a measure of the system's
  ability to improve the performance of support or operational programs. This
  improvement should be measured in quantitative terms, but not in dollars. The economic
  (dollar) impact is captured in the benefit/cost ratio. However, the same benefits might be
  measured here in a different manner. For example, improvements might be expressed in
  terms of accomplishing a task sooner (hours or minutes), delivering a service with fewer
  mistakes, increasing the availability of a computer system for customer use (hours per
  month saved in time for system backups), or a number of similar terms. The more the
  project or system improves mission effectiveness the higher the score.

  C.1. Improve Internal Program Services. (10 of 20 points). Assess the expected
  improvement in service to internal customers. For example the system might improve the
  timeliness of financial reporting throughout the organization. Score (zero to ten) higher,
  the more that service will be improved in response to a problem expressed by users of
  the service.

    Zero Points: System does not appear to meet a problem defined by an internal
    customer. Little improvement in important customer service criteria, such as
    timeliness, quality, or availability is expected. An improvement is described but not
    quantified.

    Ten Points: A significant improvement expected in areas such as timeliness, quality, or
    availability, and the improvement is quantified. The improvement also addresses an
    important problem or area of service improvement defined by the customer.

  C.2. Improved Service to the Public. (10 of 20 points). Assess the expected
  improvement in service to the public. Score (zero to ten) higher, the more improvement
  is anticipated in response to a requirement defined by the public.

    Zero Points: System appears to provide little or no direct improvement in service to
    the public. Systems may make a small improvement in timeliness, quality, or
    availability, but there is no documented need for such an improvement. The
    improvement is not quantified.

    Ten Points: System significantly improves service to the public in a mission where
    need is demonstrated or provides a new type of service to meet changing customer
    demands. The improvement is quantified.




                                             101
APPENDIX II                                                                   APPENDIX II

  D. STRATEGIC ALIGNMENT. (relative weight = 25 points) Measures to what
  extent the proposed investment supports strategic organizational objectives. Scoring is
  based primarily on explicit documentation of the need for the IRM system in planning
  documents. The more the project or system is aligned with program/strategic goals, the
  higher the score.

  D.1. Business Model. (7 of 25 points). Assess the degree of alignment with the
  organization's business model.

    Zero Points: Proposed project or system does not support organizational
    products/services or processes identified in the business model.

    One to Four Points: Proposed system is specifically mentioned in the 5-year IRM plan
    and supports organizational products/services or processes identified in the business
    model. (Score one to two points if the system supports products/services or processes
    in the business model, but is not listed in the 5-year IRM plan.)

    Five to Seven Points: Proposed system is specifically mentioned in the 5-year IRM plan
    and supports products/services or processes identified in the business model, and the
    project has been coordinated with all offices identified by the model for the respective
    processes the system supports.

  D.2. Level of Interest. (12 of 25 points). Assess the level of interest by senior
  managers (at agency and departmental level) and/or the Congress. Score (zero to twelve)
  higher the greater the level of interest.

    Zero Points: No expressed support for this system by senior managers or the
    Congress.

    Twelve Points: System has strong support from the Congress, departmental senior
    managers, and/or the head of the agency. System is specifically mentioned in
    determinations.

  D.3. Business Process Redesign. (6 of 25 points). Assess the degree this system
  enables the organization to do business in a better way. Score (zero to six) higher the
  greater the expected improvement.

    Zero Points: This system automates an existing business process with little
    improvement of the process (i.e., helps to do the same thing faster).

    Six Points: System enables a significant improvement in the way business is
    conducted.


                                             102
APPENDIX II                                                                     APPENDIX II

  E. BENEFIT-COST IMPACT(S) (relative weight = 25 points). Measures the value
  of the system in dollar terms. The system benefit/cost ration is the key indicator. This
  ration is developed using the standard benefit-cost guidance and spreadsheet promulgated
  by the organization. The standard guidance ensures all system studies include a common
  set of costs and approach benefits definition in a similar manner. The standard
  spreadsheet assists system sponsors in the benefit/cost calculation. The higher the
  benefit/cost ratio, the better the score.

    Zero Points: Any benefit/cost ratio less than one (i.e., costs exceed the benefits).

    One Point: Benefit/cost ratio of one.

    Five Points: Benefit/cost ratio of 1.5 to 1.75.

    Ten Points: Benefit/cost ratio of 1.76 to 1.99.

    Fifteen Points: Benefit/cost ratio of 2.0 to 2.99.

    Twenty Points: Benefit/cost ratio of 3.0 to 3.99.

    Twenty-five Points: Benefit/cost ratio of 4.0 or greater.




                                               103
                                       Glossary
Activity: a named process, function, or task that occurs over time and has recognizable
results. Activities use up resources to produce products and services. Activities combine
to form business processes.

Activity-Based Costing: a set of accounting methods used to identify and describe costs
and required resources for activities within processes.

Agency Capital Plan: document that identifies existing and proposed capital assets and
that provides justification for new capital funding. Included in the capital plan should be
a statement of the agency's strategic plan, a description of assets already owned by the
agency or in procurement, an analysis detailing the performance gap between existing
capabilities and the goals and objectives highlighted in the strategic plan, justification for
new capital acquisitions proposed for funding, and other related information.

Alignment: the degree of agreement, conformance, and consistency among
organizational purpose, vision and values; structures, systems, and processes; and
individual skills and behaviors.

Annual Performance Plan: document, covering each program activity identified in an
agency's budget, that describes the actions and goals that the organization will undertake
during the year to work towards the long-term goals established in the organization's
strategic plan. Specifically, the annual performance plan establishes the agency's
performance goals for the year, describes strategies the agency will use to meet these
goals, and identifies performance measures to measure or assess the relevant service
levels, outcomes, or outputs that are to be achieved and to compare actual program
results with the established performance goals.

Annual Program Performance Report: report submitted with an agency's budget
submission that compares actual agency performance to the annual goals established in
the agency's annual performance plan.

Baselining: obtaining data on the current process that provide the metrics against which
to compare improvements and to use in benchmarking.

Benchmark: a measurement or standard that serves as a point of reference by which
process performance is measured.

Benchmarking: a structured approach for identifying the best practices from industry
and government, and comparing and adapting them to the organization's operations. Such
an approach is aimed at identifying more efficient and effective processes for achieving
intended results, and suggesting ambitious goals for program output, product/service
quality, and process improvement.

                                             104
Benefit: term used to indicate an advantage, profit, or gain attained by an individual or
organization.

Best Practices: the processes, practices, or systems identified in public and private
organizations that performed exceptionally well and are widely recognized as improving a
organization's performance and efficiency in specific areas. Successfully identifying and
applying best practices can reduce business expenses and improve organizational
efficiency.

Business Case: a structured proposal for business improvement that functions as a
decision package for organizational decision-makers. A business case includes an analysis
of business process performance and associated needs or problems, proposed alternative
solutions, assumptions, constraints, and a risk-adjusted cost-benefit analysis.

Business Process: a collection of related, structured activities--a chain of events--that
produce a specific service or product for a particular customer or customers.

Business Process Reengineering: in government, a systematic disciplined improvement
approach that critically examines, rethinks, and redesigns mission-delivery processes and
subprocesses within a process management approach. In a political environment, the
approach achieves radical mission performance gains in meeting customer and
stakeholder needs and expectations.

Business Vision: description of what senior management wants to achieve with the
organization in the future. Business vision usually refers to the medium to long term and
is often expressed in terms of a series of objectives.

Capital Asset: Tangible property, including durable goods, equipment, buildings,
installations, and land.

Cost/benefit Analysis: a technique used to compare the various costs associated with
an investment with the benefits that it proposes to return. Both tangible and intangible
factors should be addressed and accounted for.

Customer: groups or individuals who have a business relationship with the organization--
those who receive and use or are directly affected by the products and services of the
organization. Customers include direct recipients of products and services, internal
customers who produce services and products for final recipients, and other organizations
and entities that interact with an organization to produce products and services.

Cycle Time: the time that elapses from the beginning to the end of a process or
subprocess.




                                            105
Decision Criteria: documented set of factors that are used to examine and compare the
costs, risks, and benefits of various IT projects and systems. These decision criteria
consist of (1) screening criteria, which are used to identify whether new projects meet
initial acceptance requirements and ensure that the project is reviewed at the most
appropriate organizational level, and (2) criteria for assessing and ranking all projects.
These ranking criteria weigh and compare the relative costs, risks, and benefits of each
project against all other projects.

Discount Rate: The interest rate used in calculating the present value of expected yearly
benefits and costs.

Discount Factor: The factor that translates expected benefits or costs in any given
future year into present value terms. The discount factor is equal to 1/(1 + i)t where i is
the interest rate and t is the number of years from the date of initiation for the program
or policy until the given future year.

Financial System: an information system, comprised of one or more applications, that
is used for any of the following: collecting, processing, maintaining, transmitting, and
reporting data about financial events; supporting financial planning or budgeting activities;
accumulating and reporting cost information; or supporting the preparation of financial
statements.

Information Engineering: an approach to planning, analyzing, designing, and
developing an information system with an enterprisewide perspective and an emphasis on
data and architectures.

Information Management: the planning, budgeting, manipulating, and controlling of
information throughout its life cycle.

Information Resources Management (IRM): the process of managing information
resources to accomplish agency missions. This term encompasses information itself, as
well as related resources, such as personnel, equipment, funds, and information
technology.

Information System: the organized collection, processing, transmission, and
dissemination of information in accordance with defined procedures, whether automated
or manual. Information systems include non-financial, financial, and mixed systems.

Information Technology (IT): the hardware and software operated by an organization
to accomplish a federal function, regardless of the technology involved (e.g., computers,
telecommunications, etc.).




                                            106
Information Technology Architecture: an integrated framework for evolving or
maintaining existing IT and acquiring new IT to achieve the agency's strategic and IRM
goals. A complete IT architecture should consist of both logical and technical
components. The logical architecture provides the high-level description of the agency's
mission, functional requirements, information requirements, system components, and
information flows among the components. The technical architecture defines the specific
IT standards and rules that will be used to implement the logical architecture.

Intangible Benefit: benefits produced by an investment that are not immediately
obvious and/or measurable.

IT Investment Management Approach: an analytical framework for linking IT
investment decisions to an organization's strategic objectives and business plans. The
investment management approach consists of three phases--select, control and evaluate.
Among other things, this management approach requires discipline, executive
management involvement, accountability, and a focus on risks and returns using
quantifiable measures.

Investment Review Board (IRB): decision-making body, made up of senior program,
financial, and information managers, that is responsible for making decisions about IT
projects and systems, based on comparisons and trade-offs between competing projects
and an emphasis on meeting mission needs and improving organizational performance.

Life-cycle Cost: The overall estimated cost for a particular program alternative over the
time period corresponding to the life of the program, including direct and indirect initial
costs plus any periodic or continuing costs for operation and maintenance.

Mixed System: an information system that supports both financial and
non-financial functions.

Model: a representation of a set of components of a process, system, or subject area. A
model is generally developed for understanding, analysis, improvement, and/or
replacement of the process.

Modular Design: information system project design that breaks the development of a
project into various pieces (modules) that each solve a specific part of the overall mission
problem. These modules should be as narrow in scope and brief in duration as
practicable. Such design minimizes the risk to an organization by delivering a net benefit
that is separate from the development of other pieces.




                                            107
Net Present Value (NPV): the future stream of benefits and costs converted into
equivalent values today. This is done by assigning monetary values to benefits and costs,
discounting future benefits and costs using an appropriate discount rate, and subtracting
the sum total of discounted costs from the sum total of discounted benefits.

Outcome: the ultimate, long-term, resulting effect--both expected and unexpected--of the
customer's use or application of the organization's outputs.

Performance Gap: the gap between what customers and stakeholders expect and what
each process and related subprocesses produces in terms of quality, quantity, time, and
cost of services and products.

Performance Measurement: the process of developing measurable indicators that can
be systematically tracked to assess progress made in achieving predetermined goals and
using such indicators to assess progress in achieving these goals.

Post-implementation Review (PIR): an evaluation tool that compares the conditions
prior to the implementation of a project (as identified in the business case) with the
actual results achieved by the project.

Return on Investment (ROI): a figure of merit used to help make capital investment
decisions. ROI is calculated by considering the annual benefit divided by the investment
amount.

Risk Analysis: a technique to identify and assess factors that may jeopardize the success
of a project or achievement of a goal. This technique also helps define preventive
measures to reduce the probability of these factors from occurring and identify
countermeasures to successfully deal with these constraints when they develop.

Sensitivity Analysis: analysis of how sensitive outcomes are to changes in the
assumptions. The assumptions that deserve the most attention should depend largely on
the dominant benefit and cost elements and the areas of greatest uncertainty of the
program or process being analyzed.

Stakeholder: an individual or group with an interest in the success of an organization in
delivering intended results and maintaining the viability of the organization's products and
services. Stakeholders influence programs, products, and services. Examples include
congressional members and staff of relevant appropriations, authorizing, and oversight
committees; representatives of central management and oversight entities such as OMB
and GAO; and representatives of key interest groups, including those groups that
represent the organization's customers and interested members of the public.




                                            108
Strategic Plan: document used by an organization to align its organization and budget
structure with organizational priorities, missions, and objectives. According to
requirements of GPRA, a strategic plan should include a mission statement, a description
of the agency's long-term goals and objectives, and strategies or means the agency plans
to use to achieve these general goals and objectives. The strategic plan may also identify
external factors that could affect achievement of long-term goals.

Strategic Planning: a systematic method used by an organization to anticipate and
adapt to expected changes. The IRM portion of strategic planning sets broad direction
and goals for managing information and supporting delivery of services to customers and
the public and identifies the major IRM activities to be undertaken to accomplish the
desired agency mission and goals.

Sunk Cost: a cost incurred in the past that will not be affected by any present or future
decision. Sunk costs should be ignored in determining whether a new investment is
worthwhile.

Tangible Benefit: a benefit produced by an investment that is immediately obvious and
measurable.

Value-Added: those activities or steps that add to or change a product or service as it
goes through a process; these are the activities or steps that customers view as important
and necessary.




                                           109