United States General Accounting Office _________________________________________________________________________________ Accounting and Information Management GAO Division _________________________________________________________________________________ February 1997 Assessing Risks and Returns: Version 1 A Guide for Evaluating Federal Agencies' IT Investment Decision-making _________________________________________________________________________________ GAO/AIMD-10.1.13 PREFACE Investments in information technology (IT) can have a dramatic impact on an organization's performance. Well-managed IT investments that are carefully selected and focused on meeting mission needs can propel an organization forward, dramatically improving performance while reducing costs. Likewise, poor investments, those that are inadequately justified or whose costs, risks, and benefits are poorly managed, can hinder and even restrict an organization's performance. Despite making a huge investment in IT, many government operations are still hampered by inaccurate data and inadequate systems. Too often, federal IT projects have cost too much, produced too little, and failed to significantly improve mission performance. Yet there is also general agreement that the government's ability to improve its service and performance will depend heavily upon how well IT can be integrated into fundamental business/mission needs. Outdated computer systems must be replaced; inefficient, paper-oriented processes must be automated; accurate financial data must be developed and maintained; and an ever-increasing amount of information must be stored and managed. Several recent management reforms, including revisions to the Paperwork Reduction Act (PRA), the Clinger-Cohen Act,1 the Government Performance and Results Act (GPRA), and the Chief Financial Officers (CFO) Act, have introduced requirements emphasizing the need for federal agencies to significantly improve their management processes, including how they select and manage IT resources. For instance, a key goal of the Clinger-Cohen Act is that agencies should have processes and information in place to help ensure that IT projects are being implemented at acceptable costs, within reasonable and expected time frames, and are contributing to tangible, observable improvements in mission performance. Moreover, these agency processes should be institutionalized throughout the organization, and should be used for all IT-related decisions. The ultimate goal of these various legislative reforms is for agencies to make better decisions that will measurably increase the performance of the organization. PURPOSE AND USE OF THIS GUIDE This evaluation guide was developed to provide a structure for evaluating and assessing how well a federal agency is selecting and managing its IT resources and to identify specific areas where improvements can be made. To accomplish this, the guide focuses on assessing an organization from three levels: 1 The fiscal year 1997 Omnibus Consolidated Appropriations Act, Pub. L. 104-208, renamed both Division D (the Federal Acquisition Reform Act) and E (the Information Technology Management Reform Act) of the 1996 DOD Authorization Act, Pub. L. 104-106, as the "Clinger-Cohen Act of 1996." 1 • the processes that the organization is using to select, manage, and evaluate its investments in information technology, • the data (cost, benefit, and risk) that are being used to make IT decisions, and • the IT decisions that are being made using the defined processes and data. It may be unusual, especially early on, to find an agency that meets all of the requirements outlined in this guide. Managing IT projects and systems as investments is a relatively new focus within the government. But while an entire IT investment management structure may not be in place, evidence of certain aspects of the investment management process may be found, although these pieces may be incomplete or conducted on an infrequent basis. To take this gradation into account, we are proposing the use of a reporting method that highlights both positives and negatives in an agency's investment decision-making, and that also targets specific critical areas that should be improved. An example of this reporting structure, based on a previously completed evaluation of one organization's IT investment decision-making, is provided in appendix 1. In addition, while the overall evaluation approach of this guide is focused on evaluating enterprise or agencywide processes and decisions, information and questions in the guide can also be tailored for reviews of individual IT projects or systems development activities. And the guide can be used for evaluations of past activities (after-the-fact audits) or assessments of current policies. Users should determine what information is needed based on the type of evaluation that is being conducted, and then adapt the criteria and questions to meet their specific needs. HOW THIS GUIDE WAS DEVELOPED This guide was developed based on information we have gained while analyzing the management practices of several leading private and public sector organizations.2 We have since updated this information to ensure that it remains up-to-date, and we have examined the IT investment decision-making processes of additional private, state, and local organizations. In addition, to determine how well these practices translated to the federal government, we evaluated the IT investment management practices of a small group of federal agencies and compared these practices to those of leading organizations.3 2 Executive Guide: Improving Mission Performance Through Strategic Information Management and Technology (GAO/AIMD-94-115, May 1994). 3 Information Technology Investment: Agencies Can Improve Performance, Reduce Costs, and Minimize Risks (GAO/AIMD-96-64, Sept. 30, 1996). 2 We also assisted the Office of Management and Budget (OMB) in its development of a guide for federal agencies to use to evaluate IT investments.4 The information provided in this guide is consistent with the IT investment approach outlined in OMB's document and provides additional discussion of particular topics. Finally, many of the concepts outlined in this guide have been incorporated into OMB's Capital Programming Guide, which provides guidance on the planning, budgeting, acquisition, and management of different kinds of capital assets, including information technology.5 The decision-making approach for information technology outlined in this guide may be incorporated into federal agencies' capital planning processes in many different ways. For instance, the approach may (a) be incorporated into an overall capital planning process, such as that outlined by OMB in the Capital Programming Guide, (b) provide input on IT-related projects to a more comprehensive capital planning process, or (c) be used for technology decision-making in the absence of an overall capital programming and planning process. Obviously, federal agencies will be at different stages of maturity and implementation of their overall capital programming processes. Over time, we would expect greater convergence between these capital processes and the IT investment decision-making process discussed in this document. CAVEATS TO THE USER (1) This guide incorporates provisions from major federal legislation and executive branch guidance addressing IT investment decision-making. This guide conforms to requirements contained in several pieces of legislation, including PRA, the Clinger-Cohen Act, GPRA, the CFO Act, and the Federal Acquisition Streamlining Act (FASA), as well as requirements found in a number of key OMB circulars, bulletins, and policy documents. However, the IT investment approach described in this guide is a complex process and agencies have been given a great deal of latitude in how they go about selecting and managing their IT resources. But while specific steps may differ from agency to agency, key fundamental components of an investment decision-making process should be in place at every agency. (2) This guide is intended to be a flexible evaluation approach that is applicable governmentwide. When evaluating an organization's6 IT decision-making, it is important to take into account the organization's operating environment, as well as its goals and missions. 4 Evaluating Information Technology Investments, A Practical Guide, Executive Office of the President, Office of Management and Budget, November 1995. 5 Capital Programming Guide (Exposure Draft, Dec. 10, 1996), Executive Office of the President, Office of Management and Budget. 6 Throughout this guide we use the term "organization" to refer to the entity being used as the unit of analysis (i.e., department, agency, bureau, office, etc.). To ensure that all related processes are included, it will be important that the scope of the evaluation be clearly identified. 3 An organization with all IT functions and services centrally organized may implement decision processes differently from an organization with a highly decentralized IT structure. Likewise, an organization with a primary research and development mission may go about evaluating IT differently from an agency focused exclusively on program delivery. Because of the wide variance among organizations and the complexity of the investment management process, this guide introduces a flexible, multitiered evaluation approach that allows for organizational diversity and flexibility. However, while accounting for variation and differences, there are still common elements that should be present in any organization's IT investment management process. This guide focuses on these common elements. STRUCTURE OF THIS GUIDE This guide is laid out in four sections. Section 1 gives an overview of the IT investment management process. Section 2 provides an in-depth explanation of the evaluation framework that this guide proposes for evaluating an agency's IT investment decision-making, as well as assumptions about what the organization has already accomplished. Section 3 provides the specific criteria and evaluation questions that can be used to help guide your review. Finally, Section 4 lists all relevant legislation and executive branch policy documents that are associated with the IT investment management process. CONTACTS This guide was developed by the Information Resources Management Policies and Issues Group under the direction of Christopher W. Hoenig, Director, Information Management and Technology Issues. If you have any questions about this guide or the IT investment management approach outlined here, please contact Dave McClure, Senior Assistant Director, at (202) 512-6257 (email@example.com), Shane Hartzler, Business Process Analyst, at (202) 512-6296 (firstname.lastname@example.org), or John Rehberger, Information Systems Analyst, at (202) 512-3687 (email@example.com). An electronic version of this guide is available from GAO's World-Wide Web server at the following Internet address: http://www.gao.gov/policy/itguide/index.htm. The automated version is also available under the "Tools" section of AIMD's Pathways intranet at http://gaoweb/issues/pathways.aimd/aimd.htm. Gene L. Dodaro Brian P. Crowley Assistant Comptroller General Assistant Comptroller General Accounting and Information Management for Policy Division 4 TABLE OF CONTENTS Preface 1 Purpose and Use of This Guide 1 How This Guide Was Developed 2 Caveats to the User 3 Structure of This Guide 4 Contacts 4 Table of Contents 5 Section 1: Overview of the IT Investment 6 Management Process Critical Success Factors 10 Legislative Requirements 11 Section 2: Framework for Assessing Agencies' IT 13 Investment Decision-making Assumptions About Related Management Decisions 18 and Processes Linkages to IT Performance Management 20 Potential Information Sources 20 Section 3: Criteria and Questions 22 Section 4: Related Legislation and Executive Branch 82 Guidance Legislation 82 Executive Branch Guidance 89 Appendix I: Proposed Structure for Presenting 94 Results Appendix II: Example of One Organization's Criteria 98 for Comparing and Ranking Projects Glossary 104 5 SECTION 1 OVERVIEW OF THE IT INVESTMENT MANAGEMENT PROCESS An IT investment management process is an integrated approach to managing IT investments that provides for the continuous identification, selection, control, life-cycle management, and evaluation of IT investments. This structured process provides a systematic method for agencies to minimize risks while maximizing the return of IT investments. To be most successful, an IT investment management process should have elements of three essential phases--select, control, and evaluate. However, each phase should not be viewed as a separate step. Rather, each is conducted as part of a continual, interdependent management effort. Information from one phase is used to support activities in the other two phases. The following figure illustrates the three phases of an IT investment management process and the relationships between the various phases. Figure 1: Fundamental Phases in the IT Investment Management Process Control What are you doing to ensure that the projects will deliver the benefits projected? Select How do you know you have selected the best projects? Evaluate Based on your evaluation, did the systems deliver what you expected? Process Information 6 An analysis of the existing portfolio of IT investments, commonly done as part of a systems or applications disposition process, helps to ensure that senior managers are informed of current costs, benefits, and risks associated with the exisiting portfolio. This disposition analysis and planning process also helps ensure that an accurate systems and applications inventory is maintained and it forms the basis for a systems retirement and replacement strategy. Such a strategy can provide a solid foundation for keeping, stopping, transforming, or replacing information systems and technology. In addition, the disposition analysis can assist in technical infrastructure planning and helps to ensure the integrity of the organization's information technology architecture. The IT investment management process begins with the Selection phase. In this phase, the organization determines priorities and makes decisions about which projects will be funded during the year.7 A starting point for the Selection phase is the screening process, in which projects being submitted for funding are compared against a uniform set of screening criteria and thresholds in order to determine whether the projects meet minimal requirements and to identify at what organizational level the projects should be reviewed. The costs, benefits, and risks of all IT projects--proposed, under development, operational, etc.--are then assessed (usually by an independent group) and the projects are compared against each other and ranked or prioritized. As part of this process, weighting factors may be attached to the ranking criteria. These ranking criteria should, at a minimum, include cost, risk, and benefit factors, as well as an assessment of how well the project meets mission needs. Finally, a senior management decision-making body makes decisions about which projects to select for funding based on mission needs and organizational priorities. The systems and projects that are selected for funding make up the portfolio of IT investments.8 The Selection phase helps ensure that the organization (1) selects those IT projects that will best support mission needs and (2) identifies and analyzes a project's risks and proposed benefits before a significant amount of project funds are spent. A critical aspect of this phase is management understanding and participation and decision-making that is driven by accurate, up-to-date data and an emphasis on using IT to enhance mission performance. Once selected, all of the projects in the portfolio are consistently controlled and managed. Progress reviews, in which the progress of projects are compared against projected cost, schedule, and expected mission benefits, are conducted at key milestones in each project's life cycle. The type and frequency of these reviews are usually determined based on the analyses of risk, complexity, and cost that went into selecting the project. If a project is late, 7 The Selection phase can be conducted quarterly, annually, or to coincide with whatever time frame is used by the organization itself or, as is the case in the public sector, what is mandated by law. 8 The senior management decision-making body may or may not have "final" approval of the funding request. In most cases, the portfolio of IT investments selected by the decision-making body will be forwarded to the agency head for final approval and inclusion in the agency budget request. In addition, OMB reviews and congressional budget decisions ultimately serve as final approval steps. 7 over cost, or not meeting performance expectations, senior executives decide whether it should be continued, modified, or canceled, and actions are quickly taken to mitigate the effects of changes in risks and costs. The Control phase helps ensure that as a project is developed and investment costs rise, that the project continues to meet mission needs, and if it does not or if problems have arisen, mitigating steps are quickly taken to address the deficiencies. Decisions made at the Control phase may include cancelling the project, modifying it to better meet mission requirements, accelerating development of the project, or continuing its development as planned. Finally, once projects have been fully implemented, actual versus expected results are evaluated to (1) assess the project's impact on mission performance, (2) identify any changes or modifications to the project that may be needed, and (3) revise the investment management processes based on lessons that were learned. This IT investment decision-making approach is a fluid and dynamic process. Figure 2 illustrates how this process can work when IT spending for all projects--new proposals and ongoing projects--is decided each year as part of an annual budget process. Both proposed and ongoing projects enter in to an IT investment disposition analysis, which examines the existing inventory of systems and applications to review existing costs, benefits, and risks associated with all IT investments. Selection decisions are made based on an analysis of where needs are greatest and in line with the organization's systems retirement and replacement plans and implementation strategy. Projects that are terminated or delayed as part of selection decisions are evaluated immediately to allow the organization to assess the impact on future proposals and to quickly benefit from lessons that are learned. Figure 2 also illustrates how projects selected as part of an IT portfolio of investments enter in to an investment control process for the remainder of the year. Investment control meetings are conducted on a regular basis throughout the year. These meetings may coincide with episodic project events that automatically trigger a management review (i.e., deviations in cost, schedule or performance outside of accepted thresholds) or with critical life-cycle milestones. Post-implementation reviews (PIRs) may also be conducted for projects that are completed or canceled during the year. The results of these control meetings and post-implementation reviews provide input into the following year's Selection process. 8 Figure 2: Example of an Annual IT Investment Decision-making Process Feedback Evaluation of projects stopped or delayed IT investment Analysis and SELECTION New IT disposition ranking of all IT proposals analysis & of IT portfolio projects planning CONTROL of on-going projects via On-going reviews based on critical milestones or IT projects conditions that "trigger" immediate management attention EVALUATION Feedback of projects Feedback implemented or canceled The selection, control, and evaluation decision-making processes described above can be applied to almost any organization, even one that is highly decentralized. In the federal government, for instance, major cabinet departments often have several agencies under their purview. Separate IT investment decision-making processes can exist at both the departmental and agency levels, provided that the organization can identify which IT projects and resources are shared (and should be reviewed at the departmental level) and which are unique to each individual agency. See figure 3 for an illustration of how decision-making can be applied at various organizational levels. 9 Figure 3: Levels of IT Investment Decision-making S Select Corporate decisions IT Project Portfolios Change in Decision-making C Control C Authority Projects that are: E Evaluate -- high-dollar amount or S high-risk; -- cross-departmental; or -- common infrastructure E projects. Business unit 1 decisions Business unit 2 decisions Business unit 3 decisions C C C S S S E E E Criteria for determining projects that should bump up to the highest review level may include (1) high-dollar, high-risk projects (the risk and dollar thresholds for identifying these projects should be predetermined by the organization), (2) cross-functional projects (two or more organizational units will benefit from the project), or (3) common infrastructure support (e.g., hardware and telecommunications). Projects that meet these particular threshold criteria are strong candidates for discussion, review, and decisions at a departmentwide level. Determining where in the organization a project should be reviewed is something agencies have been given freedom and flexibility to decide. The key to making good decisions is having clearly defined roles, responsibilities, and criteria for determining the types of projects that will be reviewed at the different organizational levels. CRITICAL SUCCESS FACTORS To be successful, an agency's IT investment management processes should generally include the following elements: • Key organizational decisionmakers are committed to the process and are involved throughout each project's life cycle. 10 · Projects are assessed jointly by program, financial, and IT managers. • The investment management process is repeatable, efficient, and conducted uniformly and completely across the organization. · The process includes provisions for continually selecting, managing, and evaluating projects in the investment portfolio. • Decisions are made consistently throughout the organization. · Decisions at any level of the organization are made using uniform decision criteria. · Decisions are driven by accurate and up-to-date cost, risk, and benefit information. · Decisions are made from an overall mission focus (there is an explicit link with the goals and objectives established in the organization's strategic plan or annual performance plans and with the organization's information technology architecture). • Accountability and learning from previous projects is reinforced. • The emphasis is on optimizing the portfolio mix in order to manage risk and maximize the rate of return. • The process incorporate all IT investments, but recognizes and allows for differences between various project types (mission critical, administrative, infrastructure) and phases (new, under development, operational, etc.). LEGISLATIVE REQUIREMENTS In recent years, several pieces of legislation have been passed that have required federal agencies to examine and change their current operations and management practices in order to improve performance and achieve greater mission outcomes. The following is a brief description of the five major management reforms. The Paperwork Reduction Act of 1995 (PRA) (Public Law 104-13) PRA requires agencies to use information resources to improve the efficiency and effectiveness of their operations and fulfillment of their missions. It emphasizes achieving program benefits and meeting agency goals through the effective use of IT. As such, it is the "umbrella" IT legislation for the federal government with other statutes elaborating on the goals contained within PRA. The Clinger-Cohen Act of 1996 (formally the Information Technology Management Reform Act, Division E of Public Law 104-106) The Clinger-Cohen Act requires federal agencies to focus on the results they are achieving through IT investments. Specifically, the act introduces much more rigor and structure into how agencies approach the selection and management of IT projects. Among other things, the head of each agency is required to implement a process for maximizing the value and assessing and managing the risks of the agency's IT acquisitions. 11 Government Performance and Results Act of 1993 (GPRA) (Public Law 103-62) GPRA requires agencies to set goals, measure performance, and report on their accomplishments. A key tenant of GPRA is that agencies will develop strategic plans, as well as annual performance plans that are linked to the strategic plans, that establish the organization's goals and objectives as well as strategies for achieving these goals. With these plans in place, an agency can begin to assess whether its activities, core processes, and resources are aligned to support its mission and achieve desired outcomes. GPRA also requires agencies to establish performance measures and benchmarks in order to begin identifying gaps between actual and desired performance levels and mission outcomes. The Federal Acquisition Streamlining Act of 1994 (FASA) (Public Law 103-355) Title V of FASA requires agencies to define cost, schedule, and performance goals for federal acquisition programs (including IT projects) and to monitor these programs to ensure that they remain within prescribed tolerances. If a program falls out of tolerance (failure to meet 90 percent of cost, schedule, and performance goals), FASA gives the agency head the authority to review, and if necessary terminate, the program. Chief Financial Officers Act of 1990 (CFO) (Public Law 101-576) The CFO Act focuses on the need to significantly improve the financial management and reporting practices of the federal government. Having accurate financial data is critical to understanding the costs and assessing the returns on IT investments. 12 SECTION 2 FRAMEWORK FOR ASSESSING AGENCIES' IT INVESTMENT DECISION-MAKING As agencies begin taking actions to improve their IT investment decision-making to conform with the various legislative requirements, their focus should be on three particular areas that will have direct bearing on whether real, positive change is produced. Specifically, agency heads need to (1) institutionalize management processes, (2) regularly validate cost, benefit, and risk data used to support IT investment decisions, and (3) focus on measuring and evaluating results. Informed management decisions can only occur if accurate, reliable, and up-to-date information is included in the decision-making process. Project cost data must be tracked and easily accessible. Benefits must be defined and quantitatively and qualitatively measured in outcome-oriented terms. And risks must be quantified and mitigated to better ensure project success. Early signs of success will be examples where the contributions of IT to gains in productivity, reductions in cost and cycle-time, and increases in service delivery quality and satisfaction are quantitatively documented and independently reported. Agencies must also have hard numbers and facts on what was spent on IT and what the agency achieved with the investment. These evaluations, wherever possible, must focus on the measurable contribution that IT is making towards improving mission performance. Early signs of success will be examples of measurable impact or where IT projects with questionable results were cancelled or delayed as a result of IT investment control processes. In evaluating an agency's investment management practices and decisions and assessing how well they are responding to these critical areas, three sets of questions should be answered: • Does the agency have decision-making and management processes in place to select IT projects and systems, control and monitor these projects throughout their life cycle, and evaluate results and revise the processes based on lessons learned? • Are IT decisions being driven by cost, risk, and benefit information, and is this information accurate and up-to-date? Are project justifications updated as funding is spent and interim benefits assessed? • And most important, is the organization making decisions that maximize benefits while minimizing risks? Is the organization ensuring that the projects being funded are meeting the most critical needs of the organization? Are the selected projects being monitored and controlled, and are actions being taken to address problems as they are identified or as requirements change? We designed this assessment guide to help evaluators determine the answers to these questions, and in doing so, begin to offer tangible advice and targeted recommendations on how federal agencies can better select, control, and evaluate their IT projects. This guide is 13 structured so that evaluators will assess the Selection, Control, and Evaluation phases from three review levels--process, data, and decisions. See figure 4 for an illustration of the IT investment evaluation model. A short description and examples of each review level are also provided below. Figure 4: Dimensions of the IT Investment Evaluation Approach Repeatability Efficiency Completeness Pr Select Control Evaluate oc es s Da ta De cis io ns Process This is an assessment of the investment management processes9 that the organization is following to select IT investments, control and monitor progress of these investments, and evaluate final results. The central question to be answered is: "Does the organization have defined, documented processes for selecting, controlling, and evaluating its IT investments?" The goal in assessing an agency's processes is to identify to what extent the organization has a structure in place for managing and evaluating IT investments. • There should be documented evidence (in guidance or policy) that an IT investment management process is in place (consisting of selection, control, and evaluation pieces), that it is repeatable and implemented consistently throughout the organization, and that decision-making roles, responsibilities, and authority have been clearly defined. 9 Process is assumed to include requisite policies, practices, and procedures that are used to manage and make decisions about IT investments. 14 An important point to remember when making an assessment of existing processes is that the evaluation should be focused solely on the organization's policies, practices, and procedures, not on actual decisions. Having institutionalized management processes, honed to work in the culture of the organization, is critical to producing consistently good results. The investment processes should accurately reflect the way the organization actually functions and makes decisions. Data An IT investment process cannot operate without accurate, reliable, and up-to-date data on project costs, benefits, and risks. It is the basis for informed decision-making. In addition, documentation of management decisions are essential to begin to assemble a track record of results. Evaluating the data involved in the IT investment management process requires evaluating two different types of data: ex ante - the information that is being used as inputs to the IT investment process (e.g., the cost/benefit/risk analyses that are used to justify the selection and continued funding of projects, the performance measures that are used to monitor a project's progress, etc.). ex post - information that is produced based on decisions that are made (e.g., project review schedules and risk mitigation plans should be developed once a decision is made to fund a project). • All projects (proposed, under development, operational, etc.) should have complete and accurate project information--cost and benefit data, risk assessments, links to business/program goals and objectives, and performance measures, as well as up-to-date project-specific data, including current costs, implementation plans, staffing plans, and performance levels. In addition, the organization should have qualitative and quantitative project requirements and decision criteria in place to help screen IT projects, assess and rank projects, and control and evaluate the projects as they move through the various phases of their life cycle. • All management actions and decisions that are made should be documented and maintained. Moreover, some decisions require that additional information be produced. For instance, after a project is selected, project-specific review schedules and risk mitigation plans should be developed. Decisions One of the most important goals of this guide is enabling evaluators to assess the effectiveness of the organization's IT investment process and the extent to which it is contributing to the improved mission performance of the organization. After evaluating the processes that the organization uses to select, control, and evaluate IT investments and the data that are used to make decisions, evaluators will be in a much better position to reach 15 conclusions about the specific decisions that the organization is making. The central focus of analysis is on whether management decisions and actions are being taken using the investment control processes and requisite project data. • The IT investment portfolio should represent a mixture of those projects that best meet the mission needs of the organization. Projects in the portfolio should be consistently monitored and decisions should be made at key milestones to ensure that the project is continuing to have its expected business or programmatic impact with a focus on minimizing risk and maximizing return. Completed projects are evaluated to compare actual performance levels to estimated levels and to feed lessons learned back in to the Selection and Control phases. Three Critical Factors In addition to these three phases, there are three critical attributes--repeatability, efficiency, and completeness--that cut across each phase and that should be assessed at each review level. Repeatability focuses on the extent to which the processes, data, or decisions being reviewed are conducted consistently over time and across different organizational units (recognizing that processes should naturally evolve as lessons are learned and improvements are made). • Projects are selected uniformly across more than one budget cycle, project reviews are conducted for all projects at established intervals, an evaluation methodology is in place and is used to assess all fully implemented projects. • The organization has identified all necessary information (cost/benefit/risk analyses, proposed schedule, user and business requirements, etc.) for making decisions, and this information is maintained, updated, and used to drive all project decisions. Specific, quantifiable decision criteria have been established and are used at all decision levels. • Projects are selected and managed based on established criteria or documented justifications. Two essential aspects of repeatability are whether (1) roles, responsibilities, and authority have been defined and documented and (2) uniform decision criteria are in place. Efficiency focuses on how well management processes, the generation of project data, and decision-making is working. The focus is on the overall quality (the accuracy, reliability, and timeliness) of the investment approach. In addition, the same data generated to support IT investment selection, control, and evaluation should be used to manage IT projects through their life-cycle. 16 • All projects are subjected to a similar investment management process (consisting of Select, Control, and Evaluate phases) and this process is documented so that everyone knows the steps that are conducted and the analyses that are required. • Cost, risk, and benefit data, both qualitative and quantitative, are accurate and are updated as information is gained. Project information is readily available and an organizational track record is maintained. Project results and lessons learned are tracked and aggregated in order to further refine and improve decision-making. • Decisions are being made at the right level. Senior managers' limited time is being utilized to the best extent possible. Actions are quickly taken to address deficiencies. Completeness focuses on the extent to which all phases of the process (Select, Control, and Evaluate) are being followed and whether use of the IT investment process is institutionalized across the organization. Traditionally, federal agencies have focused their management efforts on selecting IT projects, but once selected, the projects are never revisited and assessed to determine whether requirements or risks have changed and to ensure that the project is continuing to meet mission needs. As the evaluation is conducted and questions are asked, a concerted effort should be made to keep these three critical factors in mind. Evaluators should continually ask themselves whether the processes, data, or decisions that they are assessing are repeatable, efficient, and complete. Using this initial evaluation approach, we developed key elements for each of the nine areas (i.e., Select--Process, Control--Process, Evaluate--Process, etc.). These key elements represent the most critical factors that each organization should have in place. Figure 5 shows the IT investment evaluation approach with the additional key elements. 17 Figure 5: The IT Investment Evaluation Approach With Key Elements Repeatability Efficiency Completeness Pr Select Control Evaluate oc es Selection processes include Control processes include Evaluation processes include s screening projects consistently monitoring projects conducting post-implementation analyzing and ranking all projects involving the right people reviews (PIRs) using a standard based on benefit, cost, and risk documenting all major actions and methodology criteria decisions feeding lessons learned back in to the selecting a portfolio of projects feeding lessons learned back in to Selection and Control phases establishing project review the Selection phase Da schedules ta Selection data include Control data include Evaluation data include evidence that each project has met measures of interim results measurements of actual vs. projected project submission requirements updated analyses of each project's performance analyses of each project's costs, costs, benefits, schedule, and risks documented "track record" (project benefits, and risks and process) data on the existing portfolio scoring and prioritization outcomes De cis project review schedules io Selection decisions include Control decisions include Evaluation decisions include ns determining whether projects met deciding whether to cancel, modify, assessing the project's impact on process-stipulated requirements continue, or accelerate a project mission performance and determining deciding upon the mixture of aggregating data and reviewing future prospects for the project projects in the overall IT investment collective actions taken to date revising the Selection and Control portfolio phases based on lessons learned ASSUMPTIONS ABOUT RELATED MANAGEMENT DECISIONS AND PROCESSES In developing this guide, we made several assumptions about what the organization being reviewed has already accomplished. These assumptions correspond closely with OMB's policy guidance that requires agencies to determine, before committing resources to any new capital asset, whether (1) the asset will support functions that are mission critical, (2) any 18 other government or private entity can provide the service better, and (3) agency business processes have first been reengineered to provide optimal performance at the lowest cost.10 First, because an essential factor in selecting and managing IT projects is ensuring that the projects support organizational goals and objectives, it is critical that the organization have a clear statement of its mission. Mission goals and objectives, as well as strategies for accomplishing them, should be explained in the agency's strategic and annual performance plans.11 In addition, the agency should have an established information technology architecture12 that systems and projects are expected to follow. Second, before making a significant investment in an IT project, managers should determine the extent to which existing resources allow their programs to meet the goals and objectives spelled out in the agency's strategic and annual performance plans. To help make this determination, the agency should have an accurate inventory of its existing systems and applications, including an identification of their related costs and organizational benefits. Finally, a key hazard in acquiring information technology is that the new system will automate outmoded, inefficient business processes. The Clinger-Cohen Act requires agency heads to revise mission-related and administrative processes (as appropriate) before making a significant investment in IT systems to support them. Thus, an assessment of current processes (process mapping, baselining, benchmarking) should be completed before any decision is made about acquiring technology. (For guidance on assessing how well an organization is addressing key tasks and risks associated with business process re- engineering, see GAO's forthcoming evaluation guide entitled Business Process Reengineering Assessment Guide). 10 See Policy Memorandum M-97-02, "Funding Information Systems Investments," October 25, 1996, Office of Management and Budget; and "Capital Programming Guide," (Exposure Draft, December 10, 1996), Office of Management and Budget. 11 OMB memorandum 96-22, "Implementation of the Government Performance and Results Act of 1995" (April 11, 1996) required agencies to submit to OMB by the fall of 1996 draft mission statements, general long-term goals and objectives, and annual performance goals. In addition, GPRA requires agency heads to submit completed strategic plans to OMB and the Congress by September 30, 1997, and to establish and begin submitting annual performance plans by fiscal year 1999. 12 Section 5125(d) of the Clinger-Cohen Act defines an information technology architecture as an integrated framework for evolving or maintaining existing IT and acquiring new IT to achieve the agency's strategic and information resources management goals. A complete IT architecture should consist of both logical and technical components. The logical architecture provides the high-level description of the agency's mission, functional requirements, information requirements, system components, and information flows among the components. The technical architecture defines the specific IT standards and rules that will be used to implement the logical architecture. 19 LINKAGES TO IT PERFORMANCE MANAGEMENT A key requirement of GPRA is that federal agencies are to begin establishing performance measures that identify the primary business outcomes on which the agency is focused on achieving. In addition, the Clinger-Cohen Act specifically requires agencies to justify IT investments in terms of improvements in agency operations and performance. Moreover, agency managers are expected to use performance measures as the basis for continuing, modifying, or terminating IT projects. Because the investments that an organization makes in IT should be aligned with the business or mission results that it is trying to achieve, the IT performance measures that are used to monitor progress should reflect this focus on results. IT performance measures should be inherently tied to a project's expected benefits. A key question for evaluators to ask is: "Are the expenditures on IT products and services reasonable given the expected/actual improvements in mission performance?" POTENTIAL INFORMATION SOURCES When reviewing an agency's IT investment decision-making and management processes, information will come from a wide variety of sources. These sources will include the following: Agency-generated policy documents The organization should have documented policies or procedures that outline how various aspects of the IT investment management process are to be conducted. These documents may include the following: • Charters or operating procedures for decision-making bodies • Organizational charts • Project submission requirements • Methodologies (for conducting post-implementation reviews, cost/benefit analyses, risk analyses, cost estimation, systems development) • Operating policies and procedures (including concept of operations) • Cost and risk models • Criteria for justifying and making decisions about projects (both a listing of these criteria and definitions), • Return on investment (ROI) guidance/calculations (thresholds), • Risk assessment methods, tools, and guidance • Business case justification guidance • Strategic business/mission plans • Strategic IT/IRM plans • Information or systems architecture policy 20 Reporting requirements Federal agencies are required by various pieces of legislation and executive branch guidance to submit information on agency plans and actions. Information required includes the following: • Budget submissions (OMB Circular A-11: Section 43 exhibits, 300 A and C exhibits; OMB Policy Memo M-97-02 --IT Funding Requirements) • GPRA strategic plans • GPRA annual performance plans • GPRA annual program performance reports • Clinger-Cohen IT performance reports • PRA organizational, strategic, and IRM mission plans, • PRA IRM strategic and operational plans Project-specific data Finally, the development and management of IT projects requires project-specific information to be generated and maintained. This information may include: • project plans and progress reports, including requirements documents, design/development schedules, a software development plan, a test plan, interface documents, a conversion plan, etc. • system design alternatives documentation • work and information flow diagrams • requirements analysis documentation • performance results/assessments (post-implementation reviews) • business case information In addition, agency information may include documentation of management decisions and actions, such as memos and minutes of high-level decision-making bodies. Agency budget justification documents submitted to OMB and to congressional appropriations committees may also contain valuable information. 21 SECTION 3 CRITERIA AND QUESTIONS This section explains the common elements associated with each phase of an IT investment decision-making approach. Each element is cross-indexed to applicable criteria contained in federal laws and executive branch policies and guidance (abbreviations used for these laws and policies, as well as brief summaries for each citation, are included in Section 4). The evaluation questions provided in this section are intended to help guide evaluators in their assessment by providing a minimum set of questions for each area; however, they are not meant to be an exhaustive list. Depending on the unit of analysis for IT investment decision-making--cabinet-level department, independent agency, a bureau within a department or agency, or a single program office--evaluators may find that additional questions are still needed. The questions that are provided cover explicit criteria contained in legislation and executive branch policy guidance and are linked to critical elements associated with each investment decision-making phase. In many instances, though, existing legislative and policy criteria are not specific in explaining how agencies are to implement requisite processes. This provides agencies flexibility in how they implement their IT investment management processes. However, regardless of the approach, common elements associated with this type of decision-making framework should be in place. The questions focus on these common elements. 22 Select Control Evaluate Process Data Decisions Select--Process Key process elements to be reviewed: 1.1 Screening projects 1.2 Analyzing and ranking all projects based on benefit, cost, and risk criteria 1.3 Selecting a portfolio of projects 1.4 Establishing project review schedules Explanation and Definitions of Process Requirements for the Selection Phase The IT investment management process begins with the project selection process. Projects being proposed for funding are put through a "coarse-grained" screening process to (1) eliminate proposals that fail to pass minimal acceptance criteria and (2) ensure that projects are being reviewed at the most appropriate organizational level (department, bureau, unit, office, etc.). Proposals that pass this screening process have their costs, benefits, and risks analyzed in-depth. Once this is accomplished, all of the projects are compared against some common decision criteria and ranked based on their relative benefits, costs, and risks. Using this prioritized list as a guide, senior managers make decisions about which projects will be proposed for funding for the upcoming year. This post-prioritization decision-making on the appropriate mixture of projects is the essence of IT portfolio analysis. Finally, after these funding decisions have been made, schedules for reviewing projects are established or updated. 1.1 Screening The organization should have a process that outlines how to Projects introduce projects for funding and how these projects will be screened for relevancy to business/program goals and objectives and technical soundness. Specifically, the organization should Applicable criteria: • define what constitutes an IT project, CCA 5122(a) 23 Select Control Evaluate Process Data Decisions CCA 5122(b)(3) • identify initial requirements that projects must meet in order to CCA 5122(b)(5) be seriously considered for funding, PRA 44 USC 3506 (h)(5) • explain how screening will be conducted, and EO 13011 Sec. 2(b)(3) • establish roles and responsibilities for conducting the screening. OMB A-11, Part 3 OMB A-130 parts 8b(1), The screening process should be established in policy guidance (to 8b(2), 8b(4), 8b(5) OMB A-109 ensure that it is conducted consistently) and used at all levels of the OMB A-94 organization. OMB Memo M-97-02 As part of the initial screening process, there should be documented screening criteria (minimal requirements) that all projects are expected to meet. The screening criteria should serve three functions. They should: • identify whether the project meets initial acceptance requirements, There should be some initial requirements that projects must meet before they are reviewed in greater detail. These initial requirements may include return-on-investment (ROI) thresholds (or minimum cost/benefit ratios), identification of the project's link to objectives in the business or strategic plans, evidence of compliance with the organization's information technology architecture, identification of business/programmatic sponsorship, and assurance that all necessary project proposal and justification steps have been performed. • ensure that the project is being reviewed at the most appropriate organizational level, and Not all projects need to be reviewed at the same organizational level. The criteria for screening projects should be structured so a determination can be made about where in the organization a decision would best be made. Certain cost and risk thresholds can be used to determine what requires centralized (department or agency level) versus decentralized approval (bureau or unit level). 24 Select Control Evaluate Process Data Decisions • identify what level of management scrutiny is appropriate given the project's type, size, and risks. There should be flexible but defined rules explaining how project review and approval may vary based on the project's relative costs, benefits, and risks. Low-cost, low-risk projects should not need to have the same justification provided for them as projects of greater cost, risk, and organizational impact. On the basis of this screening process, projects will either move on for more in-depth analysis or will be sent back to the originating program group. Key questions to ask related to screening projects The purpose of these questions is to evaluate the presence (or absence) of defined processes for screening projects, not to assess how well the screening processes are being executed. The use of the processes and the quality of the information required by them will be addressed in later sections. Does the organization have a defined process for submitting and screening new funding proposals for management consideration? Is this process established in policy guidance? Does the process define (1) what information is to be submitted, (2) who must approve (screen) the information prior to formal submission, (3) how a determination will be made about where in the organization the project will be reviewed, and (4) the stages of management review? Are roles, responsibilities, and authority for people and offices involved in the screening process clearly defined? What information is required for submitting projects for funding? For most projects this information may include: • business case justification, including • clear, designated senior management sponsorship from a program/business unit, • links to business/program/mission objectives that the project is helping to achieve, as well as an explanation of how the IT investment will directly or indirectly help achieve the intended outcomes associated with these objectives, and • clear identification of proposed benefits (both quantitative and qualitative) 25 Select Control Evaluate Process Data Decisions • cost/benefit estimates • alternative and sensitivity analyses • compliance with the information technology architecture • risk assessment Do defined thresholds for benefit/cost ratios, return on investment calculations, risk assessments, etc. exist? Are these thresholds clearly defined and understood? Are all funding proposals treated the same or does the organization have different process requirements for funding proposals of different size, scope, etc.? Are these different requirements adequately documented? If exceptions to the screening process are allowed, are the conditions for allowing exceptions clearly documented? Does the process clearly stipulate potential actions that can be taken for projects that are funded without evidence of following the screening process? 1.2 Analyzing and The benefit, cost, and risk information of all projects (initial Ranking All Projects concept, proposed, under development, operational) should be Based on Benefit, analyzed and assessed in detail. Cost, and Risk Criteria Each project should have a business case developed that provides the sponsor's justification for the project. The business case should identify the organizational needs that the project is meeting or Applicable criteria: proposes to meet; provide information on the benefits, costs, and risks of the project; and establish proposed project development CCA 5122(a) time frames and delivery schedules. The information in the CCA 5122(b)(3) business case should be continuously updated to ensure that it CCA 5122(b)(5) always reflects the current situation. OMB Memo M-97-02 The organization should have some established group or audit function that is responsible for verifying and validating the various analyses (cost/benefit analyses including feasibility studies, risk assessments, and alternatives analyses) and information that are 26 Select Control Evaluate Process Data Decisions submitted as part of a project's business case. This validation should include • reviewing assumptions that were made, • assessing all of the alternatives that were analyzed and determining whether others should have been included, • reviewing the cost and benefit estimates to ensure that they were accurate and realistic, • evaluating the risks that were identified and determining whether others may be applicable, and • evaluating the sensitivity analyses that were conducted. The organization should have a management information system (MIS) or some other mechanism where all project information is collected and maintained. Such a mechanism, if kept accurate and up-to-date, can make data verification and validation easier by allowing the organization to track costs, risks, etc. over time. This mechanism for collecting and maintaining project information will also be essential during the Control and Evaluation phases to (1) help assess whether projects are still aligned with mission needs and organizational objectives, (2) determine whether projects are meeting planned performance goals, and (3) identify possible revisions to the overall investment management process based on previous experiences and lessons learned. After each project's cost, risk, and benefit information has been examined and validated, all of the projects should be compared against some common decision criteria in order to weigh the relative merits of the projects and develop a prioritized listing of projects. The criteria used for assessing and ranking projects should consist of elements related to three essential areas--benefits, costs, and risks. Often organizations will establish broad categories related to these three areas and then develop more specific subelements that come under each broad category. For example, an organization 27 Select Control Evaluate Process Data Decisions may establish risk as a categorical heading and then include schedule risks, cost sensitivity, technical risks, organizational risks, and risks if the project is not undertaken as subelements under the risk heading. Different organizations will break these broad categories and subelements out in different ways. For instance, some organizations may include a project's costs as one of several factors under risk, while others break project costs out as a separate category. Decisions should rarely be made based on one project factor, such as the project's estimated cost or a projection of reduced cycle time. Using an assortment of decision criteria to make decisions allows an organization to take into account and compare the different subtleties of a wide variety of projects. The organization may assign weights to each of the broad categories, as well as any subelements related to each category, in order to help prioritize those factors that the organization considers to be the most significant (e.g., a company that has limited experience developing systems may give technical risk a greater weight than projected cost). For an example of one organization's criteria for ranking projects, as well as the related weights given to each factor, see appendix II. The mixture of weights among the ranking criteria will vary from organization to organization. The weights that are given should take into account the agency's unique mission, capabilities, and limitations. The weighting schema that the organization establishes should be defined and documented. Such documentation is even more important if different weighting approaches are used for different kinds of projects (operational, infrastructure, applications development projects, etc.). To provide senior managers with an understanding of the relative costs, risks, and benefits of each project compared to the other 28 Select Control Evaluate Process Data Decisions projects, the organization may develop a scoring model or decision support tool. Such a tool compares the costs, benefits, and risks of each project against the cost, benefit, and risk criteria and assigns a score for each factor. The scores that the project receives for each factor are then added up to produce a cumulative score that establishes the project's relative worth and allows comparison against all other projects. See figure 6 for an example of a ranked list of IT projects. An important point for an organization in developing such a scoring model or decision support tool is to precisely define the scoring elements (i.e., define what constitutes a 1 versus a 5). The purpose behind these definitions is to ensure more consistent or uniform objectivity in the scoring process, which helps to eliminate widely varying interpretations and implementation. 29 Select Control Evaluate Process Data Decisions Figure 6: Example of Ranked List of IT Projects Project Estimated Strategic Mission Organizational Risk Benefit/ Total Name Project Alignment Effectiveness Impact Cost Score Costs/ Ratio Year 25 pts. total 20 pts. total 10 pts. total 20 pts. total 25 pts. total 100 pts. total A Project 800K 23 18 8 18 20 87 XXXXXX P Project XXXXXX 620K 23 15 9 16 15 77 P Project 582K 18 14 7 14 15 68 R XXXXXX O Project 500K 16 16 7 16 10 65 XXXXXX V Project 1698K 15 18 6 9 15 63 XXXXXX E Project XXXXXX 997K 15 15 6 14 10 60 D Project 1578K 6 14 7 5 25 57 XXXXXX Project 898K 11 10 7 11 10 49 XXXXXX Project 898K 6 8 3 5 5 27 XXXXXX The criteria for comparing and ranking projects should be used uniformly across the organization (i.e., office-, division-, or bureau- level decisions should be made using a set of criteria that are similar to criteria used for agency- or department-level decisions). Although different levels of the organization may use additional criteria, the organization should have a set of minimum criteria that are used enterprisewide. Using some common decision criteria provides greater assurance that the organization is selecting projects consistently and helps to avoid "apples versus oranges" project comparison problems. 30 Select Control Evaluate Process Data Decisions There should also be incentives to ensure compliance with the process and to dissuade gamesmanship. The organization should identify who is responsible for enforcing the process and there should be explicit consequences for noncompliance. Key questions to ask related to analyzing and ranking all projects based on cost, benefit, and risk criteria The purpose of these questions is to evaluate the organization's processes for analyzing and prioritizing projects, not to assess how well the analysis and prioritization are being done. The use of the processes, as well as the quality of the information, will be addressed in later sections. Does the organization require that the information and data submitted with funding proposals to be validated (accuracy, reliability, completeness)? • Does the process stipulate who is responsible for performing this validation (sponsor, project team, independent validation and verification teams, auditors, inspector general, etc.)? • Does the process stipulate where exceptions to validation are permitted? If exceptions are allowed, are other clearly defined conditions required to be met? Does the organization have an established, documented process for comparing and ranking all proposed IT-related funding? • Has the organization defined explicit criteria that will be used to help compare and rank projects? Do these criteria include cost, risk, and benefit elements (e.g., benefit/cost ratios, anticipated or actual impact on mission improvement priorities, risks versus benefits, etc.)? Do the criteria include both quantitative and qualitative criteria (e.g., return on investment calculations, risk modeling scores, capability assessments, alignment with critical needs, etc.)? • Are the decision criteria weighted? If scoring weights are attached to different items for management consideration, are these clearly defined and understood by participants? Has management consensus been established on use of the weighting scheme? Are these weights being applied consistently to all project proposals? If not, has the organization established different weighting schemas for different project types? • Does the process explain how the decision criteria are to be applied? 31 Select Control Evaluate Process Data Decisions If the organization uses a scoring model or decision support tool associated with the decision criteria to help measure the relative costs, benefits, and risks of each project, are the scoring elements precisely defined and differentiated? Is the process for analyzing and comparing IT projects required throughout the organization, regardless of where actual funding decisions/approvals are made? Does the process include incentives or disincentives to ensure compliance? Are roles and responsibilities for enforcing the process defined? Does the organization require that management evaluations, as well as scoring, ranking, and prioritization results be documented (either manually or through the use of automated applications such as a decision support tool)? • Does the organization require that this information, together with approved project data on cost, schedule, and performance, be tracked at an aggregate level? At a project level? • Does the organization require that this information be entered into a recognized management information system? Does it require the data to be maintained in a uniform fashion? 1.3 Selecting a The organization should have a senior management decision-making Portfolio of Projects body, made up of program, IRM, and financial managers, that makes decisions about which projects to fund for the year based on its determination of where organizational needs are greatest. Such a Applicable criteria: determination will usually be made by analyzing the gap between the organization's goals and objectives (as highlighted in its strategic CCA 5122(a) and annual performance plans) and the organization's existing CCA 5122(b)(1) capacity. OMB Memo M-97-02 The roles and responsibilities of the IT investment review group should be clearly identified and documented. The organization should also identify how this group will go about making decisions. This should include establishing how decisions will be reached, how conflicts will be handled, and how stakeholder input will be brought into the process. 32 Select Control Evaluate Process Data Decisions The IT investment review group may only make decisions about IT projects (these decisions would then be forwarded to a larger, capital planning board), or it may make IT-related decisions as part of the organization's capital planning decision-making process. The investment review group will make decisions about which projects to propose for funding, using the list of ranked projects as a key input. As the group goes about making these decisions, a number of trade offs will have to be made. For instance, the group will need to decide how much should be spent to continue operating and maintaining existing systems, versus funding enhancements to current systems, versus funding systems that are currently under development, versus funding new projects, versus funding research projects that assessing the applicability of emerging technologies. The group must also determine the proportions that will be spent on the various IT types (i.e., research and development, administrative, mission critical, infrastructure, etc.). And, the group must take into account dependencies among projects. The decision-making process should help address difficulties associated with using different units of measure for analyzing different kinds of IT projects, as well as a balancing of "soft" versus "hard" quantitative data. To aid the investment review group in making trade offs between various project types and phases, the organization may maintain a data repository that contains historical information on expenditures in different IT investment categories (operations and maintenance, enhancements to current systems, new systems development, research into developing or applying emerging technologies, etc.). By maintaining this information, the organization can review how much was spent previously and factor this in to current spending decisions. See figure 7 for an example of how one organization tracks IT portfolio spending categories. 33 Select Control Evaluate Process Data Decisions Figure 7: IT Portfolio Spending by Category 100% 80 60 40 20 0 Year 1 Year 2 Year 3 Year 4 Research 5% 6% 2% 4% Development 31% 25% 9% 21% Enhancements 23% 24% 48% 42% Operations 41% 45% 41% 33% Total costs (in millions) $30.4 $33.7 $ 42.3 $ 43.5 As part of the process of making trade offs and determining spending priorities, the organization may also conduct a review (in- house or via outside consultant/expert) of its current IT spending portfolio to assess alignment with mission needs, priorities, strategic direction, major process reengineering, etc. This review may include a trend analysis to show how patterns of investment and spending have changed, as well as an analysis to estimate how the spending pattern may change with the proposed IT portfolio. No matter how rigorous or structured the organization's decision- making process is, decisions about which projects to select for funding are ultimately managerial decisions. If senior managers 34 Select Control Evaluate Process Data Decisions select projects that score low when compared to other projects (e.g., high-risk, high-return projects) the justification for these decisions should be documented and the project's progress should be closely monitored during the Control phase. Making such exceptions should be kept as minimal as possible, however, to preserve the integrity of the decision-making process. The process of reviewing and selecting IT projects should be explicitly linked with other business processes (e.g., planning, budgeting, acquisition). Most investment decisions should mirror a planning decision or business objective and should be reflected in related budgeting documents and decisions. In addition, new investment proposals should be highlighted in the agency's capital plan that is submitted to OMB. An agency's capital plan should include a statement of the agency's strategic plans, analysis of the portfolio of assets already owned or in procurement, a description of the gap between goals and objectives and existing capacity, justification for new acquisitions proposed for funding, and other related information. The investment review group's responsibilities will usually not end once it has decided upon the mix of projects that will be proposed to comprise the current year's investment portfolio. Instead, the group should meet on a regular basis (often quarterly) to discuss the status of projects and to make further project decisions. The group may also be responsible for reviewing investment portfolio decisions that were made by lower-level organizational units. Key questions to ask related to selecting a portfolio of projects The purpose of these questions is to evaluate the presence (or absence) of defined processes and procedures for selecting a final set of IT projects for funding. An assessment of how well the processes are being followed, as well as the quality of the information, will be addressed in later sections. Does the organization have a formal systematic process for determining priorities and making funding decisions? Does the process clearly establish who in the organization has the responsibility and authority for making final IT-related funding decisions? 35 Select Control Evaluate Process Data Decisions • Is the process clear in establishing this responsibility and authority for various organizational levels (department/corporate, bureau, office, etc.)? Has the organization established an IT investment review group (or some other review function)? • Who is on this review group? • Does the process stipulate how membership on this group is determined and can be changed? • Does the process cover the roles, responsibilities, and authority of this group? Is it clear on the role the group will play in (1) selecting, controlling, and evaluating IT investments and (2) suggesting changes to organizational policies, procedures, and practices? • Does the group have authority to (1) approve, cancel, or delay projects, (2) approve plans for mitigating risks, (3) validate expected returns, (4) place constraints on investment proposals regarding project size and duration? • Does the process stipulate the operating rules and procedures for this group, (i.e., explaining when it will meet, how it will function, how decisions will be made, what steps will be taken to resolve conflicts, etc.)? • Is it clear what projects the investment review group (or similar management group) will review (e.g., all IT spending proposals or IT spending proposals that meet or exceed defined decision thresholds--based on cost; level of risk; cross-functional, bureau, or office impact; or involving common infrastructure needs such as telecommunications, data centers, or networks)? Are IT decisions made as part of an overall capital planning process or are IT projects separated out? Does the process explain how decisions made on IT spending will be incorporated into the organization's overall budgeting or capital programming decision- making process? Does the process require that data on obligations, outlays, and actual expenditures be maintained for all IT spending? Are categories of IT-spending defined within the organization (e.g., hardware, infrastructure, telecommunications, operations and maintenance, applications development, data processing services, personnel, etc.)? 36 Select Control Evaluate Process Data Decisions Has the organization conducted a review (in-house or via outside consultant/expert) of its current IT spending portfolio to assess alignment with mission needs, priorities, strategic direction, or major process reengineering? Does the organization have a process for documenting and disseminating results of this review? Has the rate and type of IT spending been aligned with management expectations? • Has trend analysis been done to show how patterns of investment and spending are changing? • Has an analysis been conducted to show how spending patterns could be affected by the proposed IT portfolio? Does the process define how unit or office-level IT decisions will be reviewed? 1.4 Establishing After making funding decisions, each project that was selected Project Review should have a review schedule established for it, or should have its Schedules current review schedule assessed and updated as needed. The time frames for these reviews will depend on various project-specific factors (amount of risk, investment size, mission importance, Applicable criteria: capability of the project team, etc.). CCA 5122(a) It is important that these reviews be conducted on a regular, CCA 5122(b)(1) scheduled basis. These reviews do not necessarily have to CCA 5122(b)(6) coincide with major project milestones. Moreover, "review FASA 41 USC 263 triggers" should be established that automatically require a FASA 10 USC 2220 OMB Memo M-97-02 management review meeting. For example, a cost, schedule, or performance deviation of 10% or greater might require an immediate project review. Key questions to ask related to establishing project review schedules These questions deal with the processes that the organization has in place, not an evaluation of how well the processes are actually working. Decisions and data important to this will be covered later. Does the process stipulate how approved projects are to be monitored by senior management in regular investment control meetings? • Are there procedures for informing project managers of decisions about monitoring schedules made by the investment review group? 37 Select Control Evaluate Process Data Decisions Is the review process clear on criteria for deciding the kinds of projects that will receive regular management monitoring and oversight by an investment review group versus those that will be monitored exclusively by management sponsors? Does the process allow the investment review group to call special project reviews outside of regular meetings if the group deems it necessary? Does the process require any additional certification or reviews before high-risk projects are allowed to proceed (e.g., risk mitigation plans, additional cost certifications, etc.)? 38 Select Control Evaluate Process Data Decisions Select--Data Key data elements to be reviewed: 2.1 Evidence that each project has met project submission requirements 2.2 Analyses of each project's costs, benefits, and risks 2.3 Data on the existing portfolio 2.4 Scoring and prioritization outcomes 2.5 Project review schedules Explanation of Data Requirements for the Selection Phase Good decisions require good data. Ensuring that each project meets established screening and ranking requirements and that the project's information is accurate and up-to-date is essential for ensuring that the most critical needs of the organization are being met by the projects and systems that are selected. In addition, the ex post information that is generated during this phase, such as project review schedules or risk mitigation plans, based on the selection decisions that are made, is critical for controlling and evaluating projects during the next two phases. 2.1 Evidence That The efficiency of the investment management process depends Each Project Has initially upon how well the organization is ensuring that all projects Met Project meet initial project acceptance requirements and that necessary Submission project proposal and justification steps have been performed. There Requirements should be evidence that each project that is submitted has been screened, analyzed, and evaluated according to processes and Applicable criteria: criteria established by the organization. CCA 5122(a) The information that is analyzed may include verification that all CCA 5122(b)(3) requisite selection data were submitted, that answers were received CCA 5122(b)(5) for all relevant questions, that projects met business/program goals CCA 5123(3) and conformed to the agency's information technology architecture, CCA 5126 and that projects that did not meet these requirements were not OMB A-127, Para. 6,7 allowed to move on for further review and consideration. There 39 Select Control Evaluate Process Data Decisions OMB A-123, Part II should also be evidence demonstrating that all business units OMB Memo M-97-02 adhered to organizational policies and procedures regarding the screening and acceptance of projects. Much of the evidence that will be reviewed will consist of cursory completeness and quality checks. For instance, if the organization has requirements that all projects over a certain cost threshold must (1) submit complete cost/benefit and risk analyses, (2) identify the business objectives that the project is meeting, and (3) provide assurance that the project conforms to the organization's technical architecture, then a review of projects that went on for further review should not identify any projects that did not meet these initial requirements. Evidence should also be available demonstrating that each project adhered to the documented process. There should also be evidence that information that was submitted was validated by a quality assurance/control function. Such validation can be performed by in-house quality control/quality assurance staff, internal audit staff (e.g., inspector general), etc. The project information should also be verified to ensure that it is accurate and reflects the most up-to-date information. All project information should be up-to-date, cost numbers should be accurate, benefits should be quantified to the extent possible, risks should be spelled out, alternatives should be identified, and sensitivity analyses should have been conducted. Key questions to ask related to evidence that each project has met initial project requirements These questions should be used to help determine whether the requisite information on costs, benefits, and risks has been submitted for funding proposals, "screened" for completeness and adherence to prescribed agency practices, and validated. For IT proposals that were submitted for funding consideration, was all of the required data/information prepared and submitted in accordance with the prescribed process? 40 Select Control Evaluate Process Data Decisions Is there evidence that the data/information (cost, schedule, performance, risk) for submitted projects has been validated--either independently or using a self-assessment process? Did the information/data presented in the proposals come from agency-recognized information systems (automated or otherwise)? Is the information/data easily accessible for further review and consideration? 2.2 Analyses of Each Each project that is submitted should have a business case prepared Project's Costs, that provides justification for the project. Included in the business Benefits, and Risks case should be identification of the project's functional requirements and estimates of the project's life-cycle costs, benefits, and risks (to the extent possible), as well as the corresponding analyses that were conducted to develop the estimates. Applicable criteria: Making accurate cost savings estimates and benefit determinations requires having at least a rudimentary CCA 5122(a) understanding of the baseline costs and benefits from existing IT CCA 5122(b)(3) capabilities. CCA 5122(b)(5) CCA 5126 A key analysis that should almost always be submitted with project OMB A-94 OMB A-130, parts 8b(1), proposals is a cost/benefit analysis. A complete cost/benefit 8b(2) analysis should OMB Memo M-97-02 • identify and quantify benefits and costs, • identify assumptions and constraints that were used when developing these figures, • evaluate alternatives using net present value, and • perform risk and sensitivity analyses. The amount of rigor and types of analyses that are conducted will depend, in part, on the size of the investment and the amount of risk. It may not be economical to conduct an in-depth cost-benefit analysis for a low-cost, low-risk project that only affects a specific division or office or a limited number of users. The organization should have a process that outlines what project data are 41 Select Control Evaluate Process Data Decisions required given each project's type, cost, and risks; variation in the quality or type of data should not be ad hoc. Listed below are some of the cost, risk, and benefit elements that an organization should keep in mind as it develops project estimates. Costs (recurring and non-recurring) • Up-front costs, such as hardware and software purchases, costs to design and develop the project, transition costs, etc. • On-going costs, such as salaries, software upgrades, training, supplies, operations and maintenance, disposal, etc. • Indirect costs, such as initial productivity losses, computer support (network management, data administration, hotlines), etc. Risks • Project risks, such as the size of the investment, project size and longevity (is the project designed in modules, does the project rely on the implementation of other systems, do other systems rely on this project), has the project group managed other projects of similar risk and complexity. • Organizational risks, such as mission risks if the project does not proceed, program management's commitment to the project, political expectations for the project, whether the project is legislatively mandated, etc. • Technical risks, such as skills required, hardware and software dependencies, application software, etc. Benefits (will usually consist of both tangible and intangible benefits) • Tangible benefits include benefits that can be explicitly quantified. Such benefits may include reducing costs, increasing productivity (e.g., reducing errors, eliminating duplication or needless work steps, etc.), decreasing cycle 42 Select Control Evaluate Process Data Decisions time, or improving service quality (e.g., timeliness, convenience, access, reliability, etc.). • Intangible benefits include benefits that may be easy to identify, but that are difficult to quantify. These benefits may include faster, more efficient decision-making; greater data accuracy; improved data security; reduced customer burden; or increased organizational knowledge. In identifying and measuring IT benefits, it is important to always remember the business function or process that is being supported by the technology. For instance, the benefits that are gained from implementing EDI technology are derived from the increased capability and efficiency that the technology provides to the organization and its customers. All of the information in the business case should be as up-to-date and accurate as possible. If the analyses are to yield meaningful results, it is essential that the project team carefully formulate assumptions, identify feasible alternatives, and provide realistic cost and benefit estimates. Most agencies have criteria or methodologies detailing how cost/benefit analyses are to be conducted and what should be included. In addition, OMB Circular A-94 provides guidance and discount rates for conducting cost/benefit analyses. Key questions to ask related to the analyses of each project's costs, benefits, and risks These questions are concerned with the review of the data and corresponding analyses submitted for project justifications. In many cases, determining answers to these questions will require that a sample of business cases be taken. Are project cost data fully identified--direct, indirect, ongoing? Do the data include full life- cycle costs? Did these cost data come from a recognized agency financial system? 43 Select Control Evaluate Process Data Decisions Have benefits of the investment been identified using quantitative and/or qualitative data/information that relate directly to mission support and performance improvement? Are expected cost savings, productivity gains, and improvements in quality identified and timeframes specified as to when these should occur? Have all foreseeable risks been identified? These risks may include technical, managerial, capability, procurement, organizational impact, stakeholder, etc. Have all concerns and potential problem issues been resolved or accounted for in a risk mitigation strategy for the project? Have business owners been involved in constructing and certifying the accuracy of the data that are submitted? Were baseline cost and performance data used as a basis for analysis? Is this information reliable and accurate? If baseline data were not available, were the estimates generated using a prescribed approach or method? For projects that are requesting continued or additional funding (for a new phase, new module, or as a result of increased costs), is there evidence that the newly submitted data reflect any changes that may have occurred to the cost, benefit, or risk information? Has project schedule information been reviewed in light of competing priorities; skills, capabilities, and availability of agency staff; contractor expertise and experience; etc. Was the cost and return information that was submitted constructed using accepted methods and techniques (prescribed by the agency, OMB guidance, legislative provisions, and/or accepted industry practice)? 2.3 Data on the Because all projects (ongoing, under-development, etc.) go through Existing IT Portfolio the Selection process (usually on an annual basis), portfolio data from previous years should be available to assess and compare previously selected projects. The spending, cost, and obligation Applicable criteria: data in this portfolio should be up-to-date and categorized in ways that are most meaningful to agency management. CCA 5122(b)(1) CCA 5122(b)(3) CCA 5122(b)(5) 44 Select Control Evaluate Process Data Decisions CCA 5122(b)(6) An agency's cost accounting system should be able to distinguish CCA 5126 between what has been obligated to date and what is still OMB A-130, part 8b(2) available, as well as identify what the incurred costs to date were for. In addition, the system may be able to split spending into more specific categories, such as development, operations, maintenance, etc. (Activity-based cost tracking, for example, should provide this detail.) Key questions to ask related to data on the existing portfolio The purpose of these questions is to determine whether information/data on the organization's existing IT portfolio of spending and investment is used as part of the management review process. This helps to ensure that funding decisions are being made in the context of overall spending directions. Does the organization maintain data on its current IT spending portfolio (e.g., are major categories of spending and investment defined and tracked--such as operations and maintenance, applications and systems development, hardware acquisitions, telecommunications, personnel, contracted services, data administration, research and development, etc.)? Are the costs and returns of the IT spending portfolio presented on an aggregate basis (past, current, future)? On a project basis? Was the portfolio information (IT spending categories, cost/benefit estimates, average development costs, etc.) derived from recognized agency management information systems? Are standard definitions and reporting elements used to maintain consistency and uniformity? 2.4 Scoring and There are several pieces of information that should arise out of the Prioritization Selection phase, based on the actual decisions that are made. This Outcomes information includes Applicable criteria: • the initial project scores and ranked list of projects, • the investment review group's scores based on any additional CCA 5122(b)(3) decision-support tools, CCA 5122(b)(5) • the investment group's final list of projects that it is proposing to make up the investment portfolio, 45 Select Control Evaluate Process Data Decisions • documented justification for selecting projects that scored below accepted thresholds (e.g., high-risk, high-return projects), and • funding information, as well as acquisition and development schedules, for all projects that were selected. The organization should also be maintaining net cost and benefit information on the complete portfolio of IT investments. Finally, all of the projects that were selected for funding should be included in the Agency Capital Plan that is submitted to OMB. Information that is submitted in this plan should include baseline cost, schedule, and performance goals for each project. In addition to the Capital Plan, decisions that are made on the mix of existing and new projects should be clearly identified in the agency's annual performance plans. Actions described in the Capital Plan to implement the funding, procurement, and management of the IT projects should also be articulated in these performance plans. Key questions to ask related to scoring and prioritization outcomes These questions help determine whether senior managers consult, review, further assess information/data used to make decisions (final or recommended to the agency head) about annual IT spending. Are summary data presented on each project's costs, benefits, risks for senior management (investment review group, etc.) to consider? Does the management review group conduct scoring exercises to vote on the relative strengths and weaknesses of proposals? Are these scoring exercises recorded and documented? Are the criteria that are used as the basis for the scoring instruments defined and used consistently? Can the costs of the approved list of projects for funding be tracked to available funds and/or reflected in the budget requests? 46 Select Control Evaluate Process Data Decisions When management approves funding for projects that fall outside accepted thresholds (high risks, high project costs, noncompliance with the architecture, etc.), is an explanation/rationale provided for this decision? Are additional management and project reporting requirements stipulated? 2.5 Project Review All projects that are selected for funding should have project review Schedules schedules, risk management plans, and project-specific performance measures established. All of this information will be particularly Applicable criteria: critical for assessing performance, identifying risks, and making decisions during the Control and Evaluation phases. CCA 5122(b)(6) The timing of reviews, as well as the number of reviews that will be conducted, will depend on the investment size of the project, the amount of risk, the capability of the project team, etc. In addition, the investment review group may identify additional project management or investment review reporting requirements (data, information, analysis), beyond what is specified by existing processes, for projects that it determines are particularly high-risk. These additional requirements should be clearly documented and communicated to the responsible project team. The project team should also be given explanation detailing how this information--and its assessment by senior management--may influence project continuation, delay, or cancellation. At some point the project team should develop an outline or strategy describing how any necessary acquisitions will be handled. Key tenants of a sound acquisition strategy are that it appropriately allocates risk between the agency and contractor, effectively uses competition, ties contract payment to accomplishments, and takes maximum advantage of commercial technology. 47 Select Control Evaluate Process Data Decisions Key questions to ask related to project review schedules The purpose of these questions is to determine whether any additional data requirements (beyond that expected by the normal process used for investment control meetings) were decided upon by senior management and whether these requirements were communicated to the project teams or sponsors. Once projects are approved for funding by an investment review group and/or agency head, are any additional project management or investment review reporting requirements (data, information, analysis) established for high-risk, high-cost projects (beyond what may be specified by existing processes)? • If so, have these requirements been clearly documented and communicated to the responsible project team? • Is it clear why this data/information is being requested and what it will be used for? • Has an explanation been given to the project team explaining how this information-- and its assessment by senior management--may influence project continuation, delay, or cancellation? Did each project that was approved have an outline or strategy developed establishing how any necessary acquisitions would be handled? Is this strategy appropriate given the project type? 48 Select Control Evaluate Process Data Decisions Select--Decisions Key decision elements to be reviewed: 3.1 Determining whether projects met process-stipulated requirements 3.2 Deciding upon the mixture of projects in the overall IT investment portfolio Explanation of Decisions Made During the Selection Phase The purpose of the Selection phase is to put the organization in the best possible position to make decisions about which IT proposals or projects to fund. Getting to this final decision requires that initial decisions be made about whether proposed projects should be moved on for further consideration. It then requires decisions to be made about the relative merits of each individual project. This is followed by the most important decisions, in which trade-offs are made between the various projects and systems in order to develop the IT investment portfolio that will be funded for the upcoming year. 3.1 Determining All new projects should have a decision made about whether the Whether Projects project meets all minimal project requirements, at what Met Process- organizational level the project should be reviewed, and the level of stipulated analytical rigor necessary for decisions. While these screening Requirements decisions should be relatively straight-forward, driven primarily by project-level data sufficiency, they should not be thought of as simply a cursory exercise. The overall efficiency and effectiveness Applicable criteria of the entire Selection phase depends to a large extent upon these initial screening decisions. CCA 5122(a) CCA 5122(b)(1) The organization should also have a process for determining where CCA 5122(b)(3) in the organization a funding decision should be made. The CCA 5122(b)(5) efficiency of the investment management process is significantly OMB A-130, parts 8b(2), 8b(4), 8b(5) affected by how well the organization identifies which projects OMB A-11, part 3 should be reviewed where. Senior decisionmakers should not spend 49 Select Control Evaluate Process Data Decisions their time in lengthy, in-depth reviews of projects that could have been easily assessed and decided upon at lower organizational levels. Key questions to ask related to determining whether projects met process-stipulated requirements The purpose of these questions is to determine whether actual decisions are being made that adhere to project screening processes and data prescribed by the organization's IT investment review process. Does the organization make decisions using the process and data that it has put in place? Are decisions being made about project readiness for review using the data/information requirements established by a project screening process? • Are project submissions being reviewed consistently (using the process and information required)? • Are screening decisions being recorded? • Is there evidence of projects being rejected? Were explanations for submission rejections documented and communicated to the business sponsor? If exceptions are being made to screening criteria, is the explanation documented and forwarded with the project proposal? 3.2 Deciding Upon Decisions made at this stage are the most important of all. The the Mixture of projects that are proposed to make up the investment portfolio for Projects in the the year should represent the best match with organizational needs Overall IT and business objectives or, in instances where exceptions were Investment Portfolio made, an explanation should be provided detailing reasons for the exception. Applicable criteria In making the selection decisions, senior managers should be taking into account tradeoffs between the various projects and systems CCA 5122(a) that are going to be funded. Making these tough choices requires CCA 5122(b)(1) the organization to develop and maintain an understanding that not CCA 5122(b)(3) every project or system can be funded. Spending more for CCA 5122(b)(5) operational systems may mean that there is less money for research OMB A-130, parts 8b(2), and development. The relative merits of each project should be 8b(4), 8b(5) OMB A-11, part 3 rigorously assessed and analyzed in order to prioritize and select 50 Select Control Evaluate Process Data Decisions OMB Memo M-97-02 those projects that best match the most critical business needs of the organization. In addition, projects selected for the portfolio should have decisions made about how often they will be reviewed and how associated risks are going to be managed. Key questions to ask related to selecting the composition of the current portfolio The purpose of these questions is to determine whether senior managers are making decisions following an established process and based on reliable and accurate data. Is the organization making decisions using the process and data that have been put in place, and do these decisions reflect an understanding of the necessary tradeoffs that should be made among projects. Do the systems that were selected for the current portfolio represent the best match with mission needs? Do all of the selected projects support objectives or goals established in the organization's strategic plan or annual performance plans? Are all of the selected projects justified on the basis of their relative costs, benefits, and risks? Are there any other factors that appear to have influenced the executives' decisions? For ongoing projects, have projected versus actual data on costs and interim results been used to make decisions about project continuation? Were project decisions made at the appropriate organizational level? Determine the ratio of funding between the various project types (new, proposed, under development, operational, etc.) that were selected. Does this ratio appear to effectively support the organization's business plan and objectives? 51 Select Control Evaluate Process Data Decisions Control--Process Key process elements to be reviewed: 4.1 Consistently monitoring projects 4.2 Involving the right people 4.3 Documenting all actions and decisions 4.4 Feeding lessons learned back in to the Selection phase Explanation and Definitions of Process Requirements for the Control Phase Achieving maximum benefits from a project, while minimizing risks, requires that the project be consistently monitored and managed for successful results. During the Control process, agency executives should be actively engaged in monitoring all of the projects in the investment portfolio, making decisions and taking actions to change the course of a project when necessary, and incorporating their experiences back in to the Selection phase to further refine and improve the process. 4.1 Consistently Each project should be reviewed at key milestones in its life cycle Monitoring Projects (a project review schedule should have been approved when the initial funding decision was made). The focus of these reviews Applicable criteria should expand as projects move from initial design and pilot through full implementation and as the dollar amounts that are CCA 5122(b)(1) expended increase (see figure 8). CCA 5122(b)(6) CCA 5125(c)(2) A low-cost, small-scale research and development project being CCA 5127 conducted to determine the applicability of a systems technology PRA 44 USC 3506 (h)(5) to a business process requirement might receive limited review FASA 10 USC 2220 FASA 41 USC 263 other than assessing whether the general approach is sound and EO 13011 Sec. 2(b)(3) feasible. However, projects that are preparing for limited field or OMB A-11, part 3 full-scale implementation should be reviewed in-depth--including OMB A-130, parts 8b(2), cost and performance to date--to ensure that the project delivers 8b(3) 52 Select Control Evaluate Process Data Decisions promised benefits within cost and risk limitations and to correct any problems before significant dollars are expended. Figure 8: Matching Investment Control With Project Phases Control Control Select $ Control Evaluate Select Control Evaluate Select Evaluate Select Evaluate Prototype Pilot Limited Production Implementation In addition, as the reviews are conducted, the context of the program that the system or project supports should be factored in. For instance, a project may exceed performance expectations, but if it is contributing to a program that is failing or is no longer needed, then little is gained for the organization. The project reviews should assess several aspects of the project and its development. Below are examples of assessment categories that should be considered as part of the project reviews: • Deliverables--results achieved to date versus expected results • Methodology--problems that have arisen concerning the systems development process (including contractor management issues) 53 Select Control Evaluate Process Data Decisions • Technical--technical issues or problems concerning such factors as hardware, software development, or telecommunications • Schedule--estimated time frames versus actuals, including schedule slippages and/or compressions • Costs--estimated costs versus actual costs spent or obligated to date, any changes in funding, and the impact of these changes • Business/program alignment--evaluation of the project's mission improvement effectiveness and relationship to business objectives • Risk--risks that were previously identified are being appropriately mitigated, new risks have been evaluated and steps taken to address them The organization should have some standard policies that help ensure that these different categories are assessed uniformly across the organization; however, the measures that are used to evaluate each project will be specific to that project. For instance, the organization may have a requirement that all projects have their schedules reviewed, but the schedules that are reviewed will be different for each project. The problem with many progress reviews is that they focus almost exclusively on cost and schedule concerns. While these factors are important, the prime focus of progress reviews should be on ensuring that benefits are being accomplished, that risks are being managed, and that the project is still meeting strategic needs. As noted earlier, "review triggers," such as updated benefit/cost ratios or ROI thresholds, done in conjunction with schedule and spending checks, can help the organization determine when actions need to be taken. The organization should have a documented process detailing how reviews will be conducted, what data and project information is required, and how decisions will be made based on the results of the project reviews. This process should include identifying roles and responsibilities for making decisions, as well as rules for how the project decisions will be made. 54 Select Control Evaluate Process Data Decisions Some organizations use a traffic-light method to help make project decisions. Projects are given red, yellow, or green lights depending on how the project rated against expected performance measures. Yellow lights indicate that management action is necessary to avoid potential problems affecting project outcomes. Red lights indicate that major problems have already occurred. (As with all reporting and scoring mechanisms, it is critical that the organization define the conditions associated with element.) The organization should also have an independent audit team, quality assurance group, or independent validation and verification (IV&V) contractor who is responsible for ensuring that project information is valid and verifying that corrective actions have been taken. In addition, the organization should have procedures in place to ensure that information from this quality assurance function is integrated in to the project review process. Finally, the organization should have mechanisms in place to ensure that project teams are complying with the control process. This may include incentives for raising problems to senior managers and disincentives for noncompliance. Project reviews, while helping to ensure accountability, should not be totally viewed as a "gotcha" opportunity, in which project managers are punished when problems are identified. Rather, the reviews should be considered opportunities for raising problems early, when they may be easier to address, rather than allowing the problems to be buried, creating a risk that they will arise later when costs are higher and the potential impact is greater. Key questions to ask related to consistently monitoring projects The purpose of these questions is to determine whether fundamental elements of a management control process are defined and documented. Evaluation of the data and actual implementation of the process are covered in later sections. The process outlined here is for a management level, not project level, review; however, many of the same decision elements are applicable to both. 55 Select Control Evaluate Process Data Decisions Does the organization have a defined, documented, and repeatable process for monitoring and reviewing IT projects? Does this process define what the focus of the investment reviews will be? Some key elements that may be included in the review include the following: • project status, including where the project stands in relation to the development of other projects • business sponsor evaluation of the project • estimated vs. actual costs to date • estimated schedule vs. actual schedule • actual outcomes from modular testing, pilots, prototypes, or limited site testing vs. estimates • technical performance as well as estimated impact on program or business performance • updates on risk mitigation efforts, including identification of any new risks and steps to address • technical issues or problems that have arisen • contractor performance and delivery • review of the methodology or systems development process • new unexpected issues affecting project progress or outcomes Does the process stipulate what project data and information must be submitted for management evaluation? Does the process indicate how data and information being presented for management review are to be verified and validated? Are roles and responsibilities for conducting this verification and validation spelled out? Does the process define a specific group (or groups) of managers that are responsible for conducting IT investment control meetings? • Are procedural rules for how the investment review group will operate and make decisions clearly defined and documented? • Are the roles, responsibilities, and authority of this group(s) clearly defined? • Is the purpose of the investment review group clearly stated? • Are the types of decisions that will be made at the IT investment control meeting defined (e.g., project continuation, delay, cancellation, termination, acceleration, etc.)? 56 Select Control Evaluate Process Data Decisions If investment review processes are used across different units of an organization (e.g., department, bureau, or office levels), do the different units have consistent policies, practices, and procedures for monitoring projects and reaching decisions? Does the process make accommodations for flexible reviews (frequency, required report submissions, etc.) for different kinds of projects (high risk/low return vs. low risk/low return)? Does the process define who is accountable for acting on review decisions? Is it clear the types of actions that are under different people's responsibilities (e.g., project team, CIO staff, business sponsor, financial staff)? Does the process define how open action items are to be addressed? Does the process define roles and responsibilities for making these decisions, as well as criteria for evaluating actions that are taken and determining whether the open item has been resolved? What mechanisms are in place to help ensure compliance with the review process? Are there disincentives in place for noncompliance? Who is responsible for overseeing the process? How stringently is the process maintained? 4.2 Involving the Senior managers (particularly program managers) should be actively Right People involved in the ongoing project reviews and are responsible for making decisions about whether to continue, accelerate, modify, or Applicable criteria cancel a project. While members of the development team can, and should, be part of the decision-making process, they should not CCA 5122(b)(1) have unilateral responsibility or authority to make all project CCA 5125(c)(2) decisions. In addition, site executives and project managers should PRA 44 USC 3506 (a)(4) take part in devising and approving the solution to any problems that are identified. 57 Select Control Evaluate Process Data Decisions Key questions to ask related to involving the right people The purpose of these questions is to help determine who is involved in making project decisions. Who is involved in ongoing project reviews and decisions? Do the review groups include staff from program, IT, and financial offices? Is there membership by a quality assurance or some other outside assessment group? Are project managers or site executives included in devising and approving actions to address deficiencies that were identified? 4.3 Documenting All All of the information in the business case, including the various Actions and analyses that were conducted to justify the project, should be Decisions updated to reflect the current state as project implementation continues and dollar amounts increase. Applicable criteria Some leading organizations estimate that often they cannot accurately estimate costs or quantify benefits until almost 40 CCA 5122(b)(3) percent of the way into the project. CCA 5122(b)(5) CCA 5122(b)(6) The organization should have a uniform mechanism (e.g., CCA 5123(3) management information system) for collecting, automating, and CCA 5126 processing data on expected versus actual outcomes. Specifically, this mechanism should • provide the cost and performance data needed to monitor and evaluate investments individually and strategically, • provide feedback on the project's adherence to strategic initiatives and plans, and • allow for the review of unexpected costs or benefits that resulted from investment decisions. Data in this system should be easily accessible to both the program team and senior managers. Collecting and maintaining project information is important, not only from a project review standpoint, but also from the standpoint of establishing an organizational memory. Decisions 58 Select Control Evaluate Process Data Decisions in all three phases of the investment cycle (Select, Control, and Evaluate) will depend on this information being accessible and up-to-date. Key questions to ask related to documenting all actions and decisions The purpose of these questions is to help determine whether the organization has processes and procedures for capturing project-related information. An assessment of the information and actual implementation of the processes will be covered in later sections. Does the organization have procedures for ensuring the accuracy and reliability or project information? Does the organization define the various pieces of information that are to be updated and maintained? This information may include the following: • project-related decisions that are made • actions that are to be taken, as well as criteria or measures for evaluating improvement • project outcomes • cost/benefit analyses and other associated project information • business case information Does the process stipulate where these data are to be maintained (e.g., official agency information system with uniform data standards and entry procedures)? 4.4 Feeding Lessons Information learned during the Control phase should be fed back in Learned Back in to to the Selection phase to help make future selection decisions and the Selection Phase to modify and enhance the screening and selection decision criteria. To make this easier, there should be some mechanism in place for aggregating decisions and actions in order to identify patterns of Applicable criteria problems or, conversely, patterns of excellence. CCA 5122(b)(1) Document the warning signs that, with hindsight, preceded the CCA 5122(b)(6) problem, identify what remedial steps were taken, and what the CCA 5123(3) outcome of this approach was. Such documentation will help to make future acquisition decisions and identify recurring problems on existing programs. 59 Select Control Evaluate Process Data Decisions Key questions to ask related to feeding lessons learned back in to the Selection phase The purpose of these questions is to help determine whether the organization has processes and procedures for aggregating project information and revising the investment management process based on lessons that are learned. An assessment of the information and actual implementation of the processes will be covered in later sections. Does the organization have a process for evaluating current decision-making processes and suggesting changes to these processes based on lessons that are learned from investment control reviews? • Does the process account for and distinguish between senior management decision- making changes and project-level management changes? • Does the process specify someone who is accountable for identifying lessons that are learned from investment control reviews? Is there a process for refining or updating the selection criteria (both screening and ranking) based on lessons that are learned? Does the organization have a process for aggregating data/information across all major IT projects (or spending categories) in order to compile an overall organizational track record on costs and benefits attributable to IT? • Is someone (or office) charged with this specific responsibility? • Does the organization have procedures for presenting this information to the IT investment review group? For presenting it to all agency executives? 60 Select Control Evaluate Process Data Decisions Control--Data Key data elements to be reviewed: 5.1 Measures of interim results 5.2 Updated analyses of each project's costs, benefits, schedule, and risks Explanation and Definitions of Data Requirements for the Control Phase Because the Selection phase usually occurs only once a year during the annual budget process, project information for that phase tends to be collected and assessed on a periodic basis. In contrast, information in the Control phase is continuously collected, updated, and fed to agency decisionmakers. The data in the Control phase should consist of such items as comparisons of actual results achieved to date versus estimates and an assessment of benefits achieved as part of project pilots or prototypes. Data collected during this phase will also consist of ex post documentation such as executive decisions and changes made to projects to address risks or better meet business requirements. The type and depth of data that are collected and maintained in this phase should be commensurate with the type and size of the project. 5.1 Measures of As projects move from one phase of the project's life cycle to the Interim Results next, and as the dollars that are expended increase, interim results should be compared against estimates to ensure that the project is progressing as expected and to indicate when actions should be Applicable criteria taken as problems arise or requirements change. CCA 5112(c) The organization should have project-specific measures established CCA 5122(b)(1) to help analyze actuals versus estimates, ensure that the project is CCA 5122(b)(5) meeting business requirements, and identify where improvements CCA 5122(b)(6) may be needed. These measures will consist of items such as cost CCA 5123(3) CCA 5125(c)(2) and schedule information, quantitative and qualitative benefit CCA 5126 measures, status of deliverables, risk elements, etc. The measures CCA 5127 should be updated as actual costs, risks, and benefits are identified. 61 Select Control Evaluate Process Data Decisions GPRA 31 USC 1115 Using these measures, the organization should identify and monitor GPRA 31 USC 1116 interim results that are achieved. The following are examples of the CFO 31 USC 902(a)(3) kinds of data that should be analyzed: OMB A-127, Para. 6,7 OMB A-123, Part II • Accumulation of actual cost data and comparisons to estimated cost levels. • Evidence that results for the phase (or results to date) have been compared against initial estimates for cost, schedule, performance, risk, and return. • Documentation of the change between the current number and scope of requirements and the original requirements baseline established for the project. • Documentation of the comparison between the current business conditions and assumptions and the project's initial assumptions and context. • After the release of each new increment, each project participating in the increment should be analyzed to determine what interim benefits have been achieved in comparison to the previous increment. • Documentation of differences between the actual performance of the software organization or contractor and their claims at the beginning of the project (e.g., schedule, costs, functionality, technical solutions, etc.). • Aggregate data covering costs, benefits, and project performance for all IT projects in the investment portfolio. Key questions to ask related to measures of interim results The focus of these questions is on the actual data being examined by senior management as part of the investment control process. In order to answer these questions, a sample of ongoing IT projects could be selected and used as the unit of analysis. Are specific measures of performance being used to track costs, schedule, benefits, and risks? Are these measures updated throughout each project's life-cycle as costs, benefits, and risks become better known? Are data being used to track actual project performance (interim results) against estimates that were used to justify the project? 62 Select Control Evaluate Process Data Decisions Are gaps or differences between estimates and actuals being analyzed and explanatory factors documented for positive or negative variances? Is there documentation to support that interim cost, schedule, benefit, and risk information has been validated or verified? If risks have changed, is this supported by documented data or analyses? 5.2 Updated The cost, benefit, schedule, and risk information that was included Analyses of Each in the business case, including the various analyses that were Project's Costs, conducted to justify the project, should be updated as project Benefits, Schedule, implementation continues and as dollar amounts increase. For and Risks instance, it may have been difficult to precisely estimate costs and benefits when the project was first proposed, but such quantification may be improved as prototype and pilot project Applicable criteria results become available. CCA 5122(b)(6) Information and analyses in the business case should also be CCA 5123(3) updated to provide justification for adding additional functional CCA 5125(c)(2) requirements to the project. This justification should weigh the CCA 5126 costs of adding the requirement late in the development process OMB A-130 parts 8b(1), versus the anticipated benefits that are expected from the added 8b(2), 8b(3) functionality. Older versions of these analyses should be maintained for later comparisons and to feed lessons learned back in to the Selection phase. Key questions to ask related to updated analyses of each project's costs, benefits, schedule, and risks The following questions address whether the organization is updating project cost, benefit, schedule, and risk information as information changes and the project proceeds through development. To answer these questions, a sample of ongoing projects could be selected as the unit of analysis. In investment control meetings, has the information in project business cases been updated to reflect the current state (including project costs to date, current risks and mitigation plans, interim benefit or performance results achieved, etc.)? 63 Select Control Evaluate Process Data Decisions Are project-level data (current and historical) being maintained and updated using agency approved databases/information systems? Have any business assumptions or environmental factors (political, funding, stakeholder support) changed? If so, have the impacts of these factors on project outcomes been evaluated? If the project is behind schedule, have risk mitigation plans been updated and explanatory factors thoroughly evaluated? If contractor assistance is being utilized, have contractor performance reports been conducted and results made available for management consideration? 64 Select Control Evaluate Process Data Decisions Control--Decisions Key decision elements to be reviewed: 6.1 Deciding whether to cancel, modify, continue, or accelerate a project 6.2 Aggregating data and reviewing collective actions taken to date Explanation of Decisions Made During the Control Phase The primary focus of the Control phase should be on making project management decisions. Actions should be taken quickly to address problems as they are identified and senior managers should be actively involved in making decisions about all of the projects in the investment portfolio. While many of these decisions will be implicit (the project is right on course, no problems have been identified, requirements have remained the same and, thus, the decision will usually be to continue the project as is), it is critical, nonetheless, that a conscious decision be made about the future of each project. 6.1 Deciding As each project is reviewed at various stages in its life-cycle, Whether to Cancel, decisions should be made about the future of the project. These Modify, Continue, or decisions will be unique for each particular project and should be Accelerate a Project based on the particular merits of the project. In addition, some explanation or documentation of the decision should be included. Applicable Criteria Even implicit decisions should have some documentation to show that a conscious decision was made to continue the project. CCA 5122(b)(1) CCA 5122(b)(6) Projects that have deficiencies or problems identified (actuals CCA 5123(3) exceed estimated levels, risks are increasing, requirements have CCA 5125(c)(2) changed, etc.) should have a decision made by senior managers CCA 5126 CCA 5127 about what to do with the project. Decisions will usually involve FASA 10 USC 2220 one of four alternatives: FASA 41 USC 263 EO 13011, Sec. 2(b)(3) • modify the project OMB A-11, part 3 • cancel the project OMB A-130, part 8b(3) • continue the project as is 65 Select Control Evaluate Process Data Decisions • accelerate the project's development Decisions may also be made to suspend funding or make future funding releases conditional on corrective actions being taken. These decisions should be documented, along with an explanation or criteria stating how funding can be reobtained. The decisions should also be reflected in budget information. For instance, if a project's development is halted while the feasibility of an alternative is assessed, budgetary spending information should reflect such a halt in funding. There should also be an explanation or documented criteria stating what must occur before funding is reinstated. In addition, depending on what decisions are made about a project, future "cascading" actions resulting from these decisions should be clearly identified and delineated. For instance, halting the development of a project will impact a number of other areas, including project management, personnel and staffing decisions, budget decisions and spending priorities, etc. These cascading actions should also be reviewed and monitored to ensure that money is not continuing to be spent and that all development activities have ceased. An independent review should be conducted prior to funding being reinstated to ensure that all corrective actions have been taken and to determine whether additional changes or modifications are still needed. Key questions to ask related to deciding whether to cancel, modify, continue, or accelerate a project The purpose of these questions is to help determine whether management decisions and actions are being taken using investment control processes and requisite project data. A sample of projects that have progressed through a management investment control process can be used to help answer these questions. 66 Select Control Evaluate Process Data Decisions Is the organization reviewing projects and using data analysis to make decisions about the future of each project? Were the decisions that were made reasonable given the situation and condition of the project? For those projects whose status remained stable, was a conscious decision made that the project should proceed or was it implicitly assumed that the project would continue? If problems were identified, was a decision made about what actions should be taken? Who made this decision? Is there some explanation or justification given for this decision? Were the actions that were taken appropriate to the problem (i.e., was the problem an IT problem or a business/process problem)? What evidence is there, if any, that actions have been taken based on the results of project reviews? For instance, if the organization determined that new requirements have been introduced, what actions were taken to respond to these additional requirements? Were these actions documented? Did these actions fully address the requirements? If decisions are made that affect a project's funding, such as suspending project funds or cancelling a project, is there evidence in budget documents and spending information that reflects this decision? Are there criteria identifying what must be done for funding to resume? Are future "cascading" actions resulting from project decisions clearly identified and delineated? Is a management group following agency policies, procedures, and practices in making regular decisions about the continuation of major IT projects? Are project data being used to make decisions and take action on IT projects that are subjected to investment reviews? Are decisions being made at the right time (i.e., as prescribed by the agency process or as agreed to by senior management as part of project approval)? Are decisions regarding projects being executed? Is accountability and follow-up clearly defined? 67 Select Control Evaluate Process Data Decisions Is an independent review conducted afterward to ensure that actions were taken and to make further changes as necessary? If projects are allowed to proceed when investment review data and analyses raise serious questions about the project, has documentation been provided detailing how management reached its decision? 6.2 Aggregating A review of past activities and decisions made about a particular Data and Reviewing project can influence both current and future managerial decisions. Collective Actions This is a primary reason why aggregating information is important. Taken to Date Aggregating allows trends to be more easily identified. Looking at projects across an agency or bureau, for example, can help pinpoint Applicable criteria those divisions that have had repeated success at developing and managing IT projects, and those that have had more trouble. This CCA 5122(b)(1) in turn can be used as inputs for decisionmakers when weighing CCA 5122(b)(6) organizational capability risks and determining project review CCA 5123(3) schedules. CCA 5125(c)(2) FASA 10 USC 2220 Aggregating can also help as the organization looks to refine and FASA 41 USC 263 EO 13011, Sec. 2(b)(3) improve the screening and selection criteria and performance OMB A-11, part 3 measures. Data can be aggregated by project, or can be grouped OMB A-130 part 8b(3) along unit, divisional, bureau, or agency lines. Problems that are identified from this analysis may be serve as an indication of specific endemic weaknesses with project management, contractor oversight, or cost-estimation practices that need revision and corrective actions. In addition, positive trends that are identified can provide valuable lessons for highlighting and reinforcing organizational strengths. 68 Select Control Evaluate Process Data Decisions Key questions to ask related to aggregating data and reviewing the collective actions taken to date The following questions are focused on determining management's awareness of trends-positive and negative--that the aggregate set of IT project investment reviews reveal. Has the organization aggregated data in order to assess organizational performance and to identify patterns or trends? At what levels--unit, division, agency, departmental--has information been aggregated? Is this information being fed back in to decisionmakers to help make future decisions? 69 Select Control Evaluate Process Data Decisions Evaluate--Process Key process elements to be reviewed: 7.1 Conducting post-implementation reviews (PIR) using a standard methodology 7.2 Feeding lessons learned back in to the Selection and Control phases Explanation and Definitions of Process Requirements for the Evaluation Phase The Evaluation phase "closes the loop" on the IT investment management process by comparing actuals against estimates in order to assess performance and identify areas where future decision-making can be improved. Lessons that are learned during the Evaluation phase should be geared towards modifying future Selection and Control decisions. Central to this process is the post-implementation review with its evaluation of the historical record of the project. 7.1 Conducting Post- Once a project has reached a final end point (e.g., the project is implementation fully implemented, the project has been cancelled, etc.), a post- Reviews (PIR) Using implementation review (or post-investment review) should be a Standard conducted. This review will usually occur about 3 to 12 months Methodology after a project has reached its final end point and should be conducted by a group other than the project development team to ensure that it is conducted independently and objectively. Applicable criteria Organizations often spend significant time and resources focused CCA 5112(c) on selecting IT projects, but less attention is given to evaluating CCA 5122(b)(1) projects after they are implemented. Yet the information gained CCA 5122(b)(6) from PIRs is critical for improving how the organization selects, CCA 5125(c)(2) manages, and uses its IT resources. CCA 5126 CCA 5127 PRA 44 USC 3506 (h)(5) Each PIR that is conducted should have a dual focus--it should PRA 44 USC 3514 (1) provide an assessment of the implemented project, including an (a)(2)(D) evaluation of the development process, and (2) indicate the extent 70 Select Control Evaluate Process Data Decisions EO 13011 Sec. 2(b)(3) to which the organization's investment decision-making processes GPRA 31 USC 1115 are sustaining or improving the success rate of IT projects. The GPRA 31 USC 1116 following are three essential areas that should be evaluated as part CFO 31 USC 902(a)(3) of a complete PIR: OMB A-130, parts 8b(1), 8b(3) (1) Customers Surveys should be conducted to determine users' satisfaction with the end product. There should also be a focused look at how well the project supports the organization's various business processes. Many of the intangible benefits that were identified at the outset will relate to how customers and end users feel about the final project. (2) Mission/Program Impact A close look should be taken to determine whether the implemented system has achieved its intended impact, and whether this impact is still aligned with mission goals. An assessment should also be made of other project-specific aspects, such as an estimate of cost savings that have been achieved, compliance with the information technology architecture, evaluations of the information product (accuracy, timeliness, adequacy, and appropriateness of information), and identification of additional maintenance or security issues. (3) Technical Capability Finally, an evaluation should be made of the technical aspects of the project, both current and future. Such an evaluation may focus on such factors as the competency of the workforce to use the new system and employee satisfaction or retention, the extent to which advanced technology was used, and the methodological expertise of the development team. To ensure that each project is evaluated consistently, the organization should have a documented methodology for conducting 71 Select Control Evaluate Process Data Decisions PIRs. This methodology, which should be used at all organizational levels, should spell out roles and responsibilities for conducting reviews and for taking actions based on the results. PIRs should be required on a regular basis to ensure that completed projects are reviewed in a timely manner. The organization should also have policies or procedures that document how information from the PIRs is to be relayed back to decisionmakers. Finally, because there is a great deal of knowledge that can be gained from failed projects, evaluations should also be conducted for projects that were cancelled prior to being fully implemented. Although project accountability is important, these evaluations should focus on identifying what went wrong with the project, in order to learn from mistakes and minimize the chances of their being repeated. Key questions to ask related to conducting post-implementation reviews using a standard methodology The purpose of these questions is to determine whether a defined, documented process is used to conduct post- implementation reviews of specific IT projects. These questions can help determine whether an organized process exists to conduct these reviews. Data and actual use of the process for making decisions will be covered in later sections. Does the organization have a defined, documented process for conducting post- implementation reviews (PIR) of IT projects? • Is the purpose(s) of the PIR process clearly explained and communicated? • Is the process clear about when PIRs are to be conducted? Are PIRs required on a regular basis to ensure that completed projects are reviewed in a timely manner? • Does the process delineate roles, responsibilities, and authorities for people and offices involved in conducting the PIRs? Does the process help ensure that these assessments are objective? • Does the process stipulate how conclusions and recommendations resulting from PIRs are to be communicated to and reviewed by senior management? 72 Select Control Evaluate Process Data Decisions Does the organization have a standardized methodology for conducting PIRs? Does this methodology, at a minimum, include assessments of customer satisfaction, mission/programmatic impact, and technical performance/capability? Is the methodology required at all levels of the organization? Does the process define how the outcomes of PIRs are to be addressed? Are roles and responsibilities defined for taking action to address any concerns or problems that are identified? What steps does the organization require to ensure that PIRs are conducted independently and objectively? Are the results of the PIRs validated or verified? Are the causes of project and process problems identified as part of the PIR? 7.2 Feeding Lessons All of the PIR information gained in the Evaluation phase should be Learned Back in to collected and maintained with project information gathered during the Selection and the other two phases. Developing this complete library of project Control Phases information helps to establish an organizational memory in which both successes and failures can be used for learning. Applicable criteria There should be some mechanism or process to ensure that information is being aggregated and fed back in to improve the CCA 5122(b)(6) investment management process. For instance, the cost, risk, and CCA 5123(3) benefit criteria (including the weights given to each) for the CCA 5125(c)(2) Selection phase may be refined to ensure greater implementation OMB A-130, part 8b(3) success of future projects. The organization should also determine whether there may be different or more appropriate cost, benefit, and risk measures that could be established that would help better monitor projects. 73 Select Control Evaluate Process Data Decisions Key questions to ask related to feeding lessons learned back in to the Selection and Control phases The purpose of these questions is to determine whether the organization uses a defined process for deciding how to take action to institutionalize knowledge, experience, and lessons learned from PIRs that are conducted. Does the organization's process or methodology for conducting PIRs include provisions for (1) changing or improving existing management decision-making processes and (2) strengthening project-level management? Does the organization have a mechanism for tracking (over time and for different kinds of projects) and aggregating the results of PIRs that are conducted? • Are the results of PIRs collected and maintained (in a manual or automated database)? Is the information reported in an timely manner? Are the results easily accessible? • Has the organization identified roles and responsibilities for collecting and analyzing PIR report information? Does the organization have procedures for regularly reporting PIR results to senior management? Is this information reliable, accurate, and easily accessible? Does the organization have procedures for regularly assessing the PIR process for completeness, quality, and contribution to project level and executive decision-making? 74 Select Control Evaluate Process Data Decisions Evaluate--Data Key data elements to be reviewed: 8.1 Measurements of actual vs. projected performance 8.2 Documented "track record" (project and process) Explanation and Definitions of Data Requirements for the Evaluation Phase Data collected during the Evaluation phase will be primarily historical in nature focusing on the outcome of a project as compared to executives' expectations for the project. In addition, ex post information that is developed should include modifications made to the Selection and Control phases, as well as the institutionalized lessons-learned information. This information should be used to revise the Selection and Control phases and to help make future investment decisions. 8.1 Measurements of The focus of the PIR should be on evaluating a project's actual Actual vs. Projected results compared to estimates in terms of cost, schedule, Performance performance, and mission improvement outcomes. An attempt should also be made to determine the causes of major differences Applicable criteria between planned and end results. And the PIR should be used to help identify any inappropriate systems development and project CCA 5122(b)(5) management practices. CCA 5122(b)(6) CCA 5123(3) The PIR should provide a wide range of information regarding both CCA 5125(c)(2) the project and the process for developing and implementing the CCA 5126 project. Specific information includes the following: GPRA 31 USC 1115 GPRA 31 USC 1116 CFO 31 USC 902(a)(3) • an assessment of the project's effectiveness in meeting the OMB A-130, part 8b(3) original objectives, 75 Select Control Evaluate Process Data Decisions • an identification of benefits that have been achieved, an assessment of whether they match projected benefits, and a determination of reasons for any discrepancies. • an evaluation of whether original business assumptions used to justify the project were valid, • a comparison of actual costs incurred against projected costs, • a determination of how well the project met time schedules and implementation dates, • management and user perspectives on the project, and • an evaluation of issues that still require attention. Outputs of the PIR should include user evaluations of the effectiveness of the project, actual costs broken out by category, measurements used to calculate benefits, a comparison matrix of actuals to estimates, and business-as-achieved documentation. Key questions to ask related to measurements of actual vs. projected performance The purpose of these questions is to determine the adequacy of data being used in conjunction with post- implementation reviews. To answer these questions, a sample of completed PIRs might be evaluated. Is the organization collecting projected versus actual cost, benefit, and risk data as part of the post-implementation reviews? • Has the cost, benefit, and risk information that was used for initial project justification been preserved? Have updates that have been made to costs, benefits, or risks been noted? Have these updates also been preserved? • Have project benefits that were obtained been quantified? If not, are qualitative measures being used to determine impact? • Have the cost data been verified or validated? Are these data contained in a recognized agency financial management/accounting database? Does the PIR include assessments of customer satisfaction (end-users, business or program unit sponsor, etc.)? Does the PIR include assessments of technical capability (e.g., conformance to recognized systems development methodology, architecture compliance, contractor performance and oversight)? 76 Select Control Evaluate Process Data Decisions 8.2 Documented The organization should be maintaining documentation of all "Track Record" decisions, changes, actions, and results that occurred throughout (Project and the project's life cycle, as well as other relevant project information, Process) such as the business case and updated cost/benefit analyses. The organization should also be tracking recommendations (for both Applicable criteria improving the project and refining the overall investment management process) that arise out of the PIRs. CCA 5122(a) CCA 5122(b)(6) This "track record" will be invaluable for helping the organization CCA 5125(c)(2) refine and improve its processes as more and more information is CCA 5126 collected and aggregated. Key questions to ask related to documented "track record" (project and process) The purpose of these questions is to determine whether aggregate data are being collected, analyzed, and maintained to support improvements in project management and executive-level decision-making. Has the organization conducted trend analyses (descriptive or statistical) using the results of PIRs that have been conducted? • Do these analyses attempt to correlate actual project performance results with factors that may have caused these results (positive and negative)? • Are the results of these analyses presented to management as a regular part of the investment decision-making process? Are special reports issued to executive management? Are recommendations for specific projects and senior management decision-making processes presented in PIRs? • Do these recommendations cover changes to process, data, or decision-making procedures used for both the Selection and Control phases? • Are these recommendations well documented and supported by existing analyses? 77 Select Control Evaluate Process Data Decisions Evaluate--Decisions Key decision elements to be reviewed: 9.1 Assessing the project's impact on mission performance and determining future prospects for the project 9.2 Revising the Selection and Control phases based on lessons learned Explanation of Decisions Made During the Evaluation Phase A number of key decisions will be made during the Evaluation phase, including an assessment of how well the project met its intended objectives, a determination of what changes or modifications to the project are still needed, and an identification of ways to modify or improve the overall investment management process to better maximize results and minimize risks. In addition, the organization may assess the overall performance of its IT investments in improving mission performance. To make these decisions, agency executives must gauge the degree to which past decisions have influenced the outcome of IT projects, understand why these decisions had the effect that they did, and determine how changing the processes for making decisions could create a better outcome for current IT projects and future IT proposals. 9.1 Assessing the The results and recommendations that arise out of the PIRs, Project's Impact on combined with other project information, are a critical input for Mission Performance senior decisionmakers to use to assess the project's impact on and Determining mission performance. In making this assessment, senior managers Future Prospects for will need to ask a number of questions about the project, including the Project the following: • How effective was the project in meeting the original Applicable criteria objectives? Are these objectives still valid? • Were the original business assumptions used to justify the CCA 5122(a)(1) project valid? CCA 5122(b)(3) • What is the current status of the system? CCA 5122(b)(5) 78 Select Control Evaluate Process Data Decisions CCA 5123(3) • Are further changes or modifications necessary? CCA 5123(4) CCA 5123(5) Even after a project has been implemented, decisions should be CCA 5125(c)(2) made on a regular basis about the status of the project. Senior CCA 5126 managers should regularly question whether (1) the current system meets organizational needs, (2) the system should be modified to better meet these needs, (3) a new system is needed to best meet these needs, or (4) the needs could best be met by outsourcing the work. In addition, because operation and maintenance (O&M) costs, for such activities as hardware upgrades, system software changes, and ongoing user training, can consume a significant amount of IT resources (some have estimated that ownership costs, operations and maintenance costs, and disposition costs can consume as much as 80 percent of a project's total life-cycle costs), a plan should be developed for the continued support and operation of each IT project. Key questions to ask related to assessing the project's impact on mission performance and determining future prospects for the project The purpose of these questions is to help assess decisions that the organization has made regarding its evaluation of the impact that projects have had on mission performance as well as the continued direction being given for these projects. What decisions have been made by senior management (or investment review group) regarding whether implemented projects met defined criteria for successful outcomes? Were corrective actions for specific projects included in senior management's decisions? • Were timetables and steps for implementing these changes established as part of the decision? • Were follow-up management reviews established? Has a clear purpose for these reviews been defined? Has a plan been developed detailing how future O&M and disposition costs will be addressed? 79 Select Control Evaluate Process Data Decisions Are decisions that are being made on specific projects cognizant of the potential (or actual) impact the decision may have on other related projects? Have decisions regarding the status of projects been finalized? • Have expected changes been communicated to the project manager? 9.2 Revising the An organization's investment management process will usually not Selection and remain static, but will evolve and change over time as the Control Phases organization learns more and more about what has been successful Based on Lessons and what still needs to be improved. Modifications that may be Learned made to the process include the following: Applicable criteria • changing the mixture of members on the organization's decision-making investment review group CCA 5112(c) • changing the Selection phase decision-making criteria (both CCA 5122(b)(1) screening and ranking criteria) CCA 5122(b)(3) • changing the Control phase criteria used for monitoring the CCA 5122(b)(5) progress of projects CCA 5123(3) CCA 5125(c)(2) • modifying the time frames for reviewing projects during the CCA 5126 Control phase GPRA 31 USC 1116 • modifying the triggers for identifying projects for review OMB A-127, Para. 6,7 • modifying the PIR methodology OMB A-123, Part II The results from one project will often not provide enough information to allow significant modification to be made to the agency's IT decision-making processes. However, a significant, recurring system development problem found across multiple projects over time would be cause for refining or even significantly revising the organization's decision-making processes and criteria. The causes for differences between planned and actual results should be determined and corrective actions to the overall IT management process, decision criteria, etc. should be identified and documented. Once the causes for differences between planned and actual results have been determined, steps should be taken to 80 Select Control Evaluate Process Data Decisions address these causes in order to ensure greater success in the future. All alterations or updates that are made to the Selection and Control phases, based on the results of PIRs, should be documented. Key questions to ask related to revising the Selection and Control Phases based on lessons learned The following questions are focused on assessing whether the organization has made changes to the IT investment management process based on lessons that were learned. What decisions have been made to modify existing organizational IT investment management processes? • Have these decisions been communicated to staff? • Are changes to existing processes, operating procedures, and data requirements aligned with conclusions and recommendations documented in PIRs? • Has the organization clearly established and communicated when these changes to existing management processes will take effect? 81 SECTION 4 RELATED LEGISLATION AND EXECUTIVE BRANCH GUIDANCE LEGISLATION Congress has passed several pieces of legislation that lay the groundwork for agencies to establish an investment approach for managing IT. The following are five key pieces of legislation that put in place various requirements related to the IT investment management process: The Clinger-Cohen Act of 1996 (CCA) The Paperwork Reduction Act of 1995 (PRA) The Federal Acquisition Streamlining Act of 1994 (FASA) The Government Performance and Results Act of 1993 (GPRA) The Chief Financial Officers Act of 1990 (CFO) Notes: Brief summaries of the legislative provisions are cited; readers should consult copies of the statutes themselves for actual language contained in the law. The phases of the IT investment process that are most related to each legislative provision follow each provision. The phase symbols are as follows: Selection phase -- Control phase -- Evaluation phase -- The Clinger-Cohen Act of 1996 (CCA) (formally the Information Technology Management Reform Act, Division E of Public Law 104-106) The Clinger-Cohen Act requires federal agencies to focus more on the results achieved through IT investments while streamlining the federal IT procurement process. Specifically this act introduces much more rigor and structure into how agencies approach the selection and management of IT projects. Among other things, the head of each agency is required to implement a process for maximizing the value and assessing and managing the risks of the agency's IT acquisitions. Specific sections of the Clinger-Cohen Act related to IT investments include: 82 CCA 5112(b) - The OMB Director is to promote and be responsible for improving the acquisition, use, and disposal of IT to improve the productivity, efficiency, and effectiveness of federal programs. CCA 5112(c) - At the same time the President submits the budget to the Congress, the OMB Director is to submit a report to the Congress on the net program performance benefits achieved as a result of agencies' major capital investments in information systems and how the benefits relate to the achievement of agency goals. CCA 5112(c) - The OMB Director is to develop, as part of the budget process, a process for analyzing, tracking, and evaluating the risks and results of all major capital investments for information systems by the federal agencies. This process is to cover the life of each system and include explicit criteria for analyzing the projected and actual costs, benefits, and risks associated with the investments. CCA 5113(b)(1) - The OMB Director shall evaluate the IRM practices of executive agencies with respect to the performance and results of IT investments. (b)(4) - The Director shall implement periodic reviews of selected information resources management activities of the executive agencies through the budget process. CCA 5113(b)(5) - Specific actions that may be taken by the Director of OMB to enforce agency accountability for its information resources management and investments made in information technology include (1) recommending increases or reductions in the agency's IRM budget, (2) using administrative controls to restrict the availability of agency funds, and (3) designating an executive agent to contract with private sources for the agency's management and acquisition of information technology. 83 CCA 5122 - Agency heads are to design and implement a process for maximizing the value and assessing and managing the risks of their IT acquisitions; the process (among other things) is to provide for the selection of investments using minimum criteria on whether to undertake an investment (including quantitatively expressed projected net, risk-adjusted return on investment, and specific quantitative and qualitative criteria for comparing and prioritizing alternative information systems projects) and to provide a means for senior management to obtain timely information regarding progress (at established milestones) in terms of cost, capability of the system to meet requirements, timeliness, and quality. CCA 5122(b)(2) - The IT investment process of executive agencies is to be integrated with the processes for making budget, financial, and program management decisions. CCA 5123(3) - Agency heads shall ensure that performance measurements are prescribed for IT used by or to be acquired for the agency; the performance measurements are to measure how well the IT supports agency programs. CCA 5123(5) - Agency heads are to analyze the missions of the agency and, based on the analysis, revise the agency's mission-related and administrative processes (as appropriate) before making significant investments in IT used to support those missions. CCA 5125(b)(1) - The agency CIO is responsible for providing advice and other assistance to agency heads and other senior managers to ensure that IT is acquired and information resources are managed for the agency in a manner that implements the policies and procedures of this act, consistent with the Paperwork Reduction Act, and the priorities of the agency head. CCA 5125(b)(2) - The agency CIO is responsible for developing, maintaining, and facilitating the implementation of a sound and integrated IT architecture for the agency; the architecture is an integrated framework for evolving or maintaining existing IT and acquiring new IT to achieve the agency's strategic and IRM goals. 84 CCA 5125(c)(2) - The agency CIO is to monitor the performance of IT programs of the agency, evaluate the performance of those programs on the basis of applicable performance measures, and advise the agency head regarding whether to continue, modify, or terminate the program or project. CCA 5126 - The agency head, in consultation with the CIO and Chief Financial Officer (or comparable official) is to establish policies and procedures that (1) ensure that accounting, financial, and asset management systems and other information systems are designed, developed, maintained, and used effectively to provide financial or program performance data for the agency's financial statements; (2) ensure that financial and related program performance data are provided on a reliable, consistent, and timely basis to agency financial management systems; and (3) ensure that the financial statements support the assessment and revision of agency processes and performance measurement. CCA 5127 - The agency head shall identify in the agency's IRM plan (required by PRA) major IT acquisition programs, or phase or increment of such program, that has significantly deviated from the cost, performance, or schedule goals established for the program. CCA 5202 - The head of the agency should, to the maximum extent practicable, use modular contracting for the acquisition of major information technology systems. (This section also describes key characteristics of "modular contracting," including (1) that the system be acquired in successive acquisitions of interoperable increments complying with common or commercially accepted IT standards, (2) that contracts should be awarded within 180 days after the date on which the solicitation is issued, and (3) the information technology should be delivered within 18 months after the date on which the solicitation resulting in award of the contract was issued.) 85 Paperwork Reduction Act of 1995 (PRA) (Public Law 104-13) The Paperwork Reduction Act of 1995 (PRA) is the "umbrella" IT legislation for the federal government, with other statutes elaborating on the goals contained within PRA. PRA requires agencies to use information resources to improve the efficiency and effectiveness of their operations and fulfillment of their missions. Specific sections of PRA related to IT investments include: PRA 44 USC 3502(7) - Definition of IRM: the process of managing information resources to accomplish agency missions and improve agency performance. PRA 44 USC 3506(a)(4) - Agency program officials, in consultation with the Chief Information Officer and Chief Financial Officer (or comparable official), are to define program information needs and develop strategies, systems, and capabilities to meet those needs. PRA 44 USC 3506(b)(2) - Each agency is to develop and maintain a strategic IRM plan on how IRM activities help accomplish agency missions (the plan is to include plans for reducing information burdens imposed on the public, for enhancing public access to and dissemination of government information, and for meeting the information technology needs of the government). PRA 44 USC 3506(b)(3)(A) - each agency is to maintain an ongoing process to ensure that IRM operations and decisions are integrated with organizational planning, budget, financial management, human resources management, and program decisions. PRA 44 USC 3506(b)(3)(C) - agencies are to establish goals for IRM to improve the productivity, efficiency, and effectiveness of agency operations and methods for measuring progress in achieving the goals. 86 PRA 44 USC 3506(h)(5) - Agencies are to maximize the value and assess and manage the risks of major information system initiatives through a process that (a) integrates budget, financial, and program management decisions and (b) is used to select, control, and evaluate the results of the initiatives. PRA 44 USC 3514(a)(2)(D) - the OMB Director is to provide an annual report to the Congress on (among other things) the extent to which agencies have improved program performance and the accomplishment of agency missions through IRM. Government Performance and Results Act of 1993 (GPRA) (Public Law 103-62) GPRA requires agencies to set goals, measure performance, and report on their accomplishments. As such, an agency's IT investments should directly support the accomplishment of these goals The specific sections of GPRA related to IT investments include: GPRA 5 USC 306 - By September 30, 1997, agency heads are to submit to OMB and the Congress a strategic plan for their program activities, including a comprehensive mission statement covering major agency functions and operations. GPRA 31 USC 1115 - Starting with fiscal year 1999, agencies are to prepare annual performance plans covering each program activity set forth in the budget. The plans are to establish performance goals in objective, quantifiable, and measurable form and performance indicators to be used in measuring relevant outputs, service levels, and outcomes of each program activity. 87 GPRA 31 USC 1116 - No later than March 31, 2000, and annually thereafter, agency heads are to prepare and submit to the President and the Congress program performance reports setting forth performance indicators and comparing actual program performance against the performance goals. The Federal Acquisition Streamlining Act of 1994 (FASA) (Public Law 103-355) Title V of FASA requires agencies to define cost, schedule, and performance goals for federal acquisition programs (to include IT projects) and monitor these programs to ensure that they remain within prescribed tolerances. If a program falls out of tolerance, FASA requires the agency head to review, take necessary actions, and, if necessary, terminate the program. Specific sections of FASA related to IT investments include: FASA 10 USC 2220 - The Secretary of Defense shall approve or define the cost, performance, or schedule goals for major defense acquisition programs and for each phase of the acquisition cycle; whenever the Secretary determines that major defense acquisition programs are not achieving, on average, 90 percent of the cost, performance, and schedule goals established, the Secretary shall ensure that there is timely review of the program and identify suitable actions (including termination) to be taken with respect to the program. FASA 41 USC 263 - The head of each executive agency shall approve or define the cost, performance, and schedule goals for major agency acquisition programs (congressional policy is that each agency should achieve, on average, 90 percent of the cost and schedule goals established for major and non-major programs of the agency without reducing the performance or capabilities of the items being acquired); whenever necessary to implement the congressional policy, agency heads are to determine whether there is a continuing need for the programs that are significantly behind schedule, over budget, or not in compliance with the performance or capability requirements and identify suitable actions, including termination, to be taken. 88 Chief Financial Officers Act of 1990 (CFO) (Public Law 101-576) Having accurate financial data is critical to understanding the costs and assessing the returns on IT investments. The CFO Act focuses on the need to significantly improve the financial management and reporting practices of the federal government Specific sections of the CFO Act related to IT investments include: CFO Act 31 USC 501 - Purpose statements: (1) provide for improvement, in each agency of the federal government, of systems of accounting, financial management, and internal controls to ensure the issuance of reliable financial information and to deter fraud, waste, and abuse of government resources; and (2) provide for the production of complete, reliable, timely, and consistent financial information for use by the Executive Branch and the Congress in the financing, management, and evaluation of federal programs. CFO Act 31 USC 902(a)(3) - The agency Chief Financial Officer shall develop and maintain an integrated agency accounting and financial management system, including financial reporting and internal controls, that provides for (1) complete, reliable, consistent, and timely information which is prepared on a uniform basis and that is responsive to the financial information needs of agency management; (2) the development and reporting of cost information; (3) the integration of accounting and budgeting information; and (4) the systematic measurement of performance. EXECUTIVE BRANCH GUIDANCE In addition to these legislative provisions, OMB and the White House have also issued several pieces of guidance related to the acquisition and management of information resources. This executive branch guidance includes OMB Circular A-11 OMB Circular A-94 OMB Circular A-109 OMB Circular A-123 OMB Circular A-127 OMB Circular A-130 Executive Order 13011 Sec. 2(b)(3) OMB Memorandum M-97-02 89 OMB Circular A-11 provides detailed instructions and guidance on the preparation and submission of agency budget requests and related materials, including program performance information. Part 2 of the Circular provides specific instructions on the preparation and submission of agency strategic plans, as required by GPRA. Part 3 provides guidance on the planning, budgeting, and acquisition management of major fixed assets and requires agencies to provide information on all major fixed asset projects included in their budget submissions to OMB. OMB Circular A-94 provides general guidance for conducting cost-benefit and cost-effectiveness analyses. This guidance serves as a checklist for determining whether an agency has considered and included all necessary elements for sound cost-benefit and cost-effectiveness analyses. The circular also provides specific guidance on the discount rates to be used in evaluating federal programs whose benefits and costs are distributed over time. OMB Circular A-109 establishes policies for acquiring major systems. Major systems are defined as those programs that are critical to fulfilling an agency mission, entail the allocation of relatively large resources, and warrant special management attention. Among other requirements, the circular requires that an agency acquiring a major system to (1) ensure that the system fulfills a mission need, (2) make appropriate trade-offs among investment costs, ownership costs, schedules, and performance characteristics, (3) ensure adequate system testing and evaluation, (4) accomplish system acquisition planning, built on an analysis of agency missions, (5) tailor an acquisition strategy for each program, (6) maintain a capability to predict, review, assess, negotiate, and monitor system life-cycle costs, and (7) assess cost, schedule, and performance experience against predictions for consideration at key decision points. OMB Circular A-123 provides guidance on establishing, assessing, correcting, and reporting on management and internal controls. Part II provides a definition of management controls and establishes guidance for designing management structures that help ensure accountability for results as federal agencies develop and execute strategies for implementing or reengineering agency programs and operations. 90 OMB Circular A-127 prescribes policies and standards for developing, operating, evaluating, and reporting on financial management systems. Part 6 of the circular lays out policy guidance for establishing governmentwide financial systems and compatible agency systems. Specifically, these systems are to provide complete, reliable, consistent, timely, and useful financial management information on federal government operations to enable central management agencies, individual operating agencies, divisions, bureaus, and other subunits to carry out their fiduciary responsibilities; deter fraud, waste, and abuse of federal government resources; and facilitate efficient and effective delivery of programs through relating financial consequences to program performance. Part 7 defines the specific requirements that financial management systems should have in place to meet the policy requirements established in part 6. OMB Circular No. A-130 provides uniform governmentwide information resources management policies as required by the Paperwork Reduction Act. Section 7 of the circular describes basic considerations and assumptions, while sections 8(a) and 8(b) describe information management policy and information systems and IT management policy, respectively. Section 8b(1)--Evaluation and Performance Measurement. Agencies are to promote the appropriate application of federal information resources by (1) seeking opportunities to improve the effectiveness and efficiency of government programs through work process redesign and the judicious application of information technology; (2) preparing and updating a cost-benefit analysis for each information system as necessary throughout its life cycle; (3) conducting cost-benefit analyses to support ongoing management oversight processes; and (4) conducting post-implementation reviews of information systems to validate estimated benefits and document effective management practices. 8b(2)--Strategic Information Resources Management (IRM) Planning: Agencies are to establish and maintain (1) strategic information resources management planning that addresses how the management of information resources promotes the fulfillment of an agency's mission; (2) information planning that promotes the use of information throughout its life cycle to maximize the usefulness of the information, minimize the burden on the public, and preserve the appropriate integrity, availability, and confidentiality of information; and (3) operational information technology planning that links information technology to anticipated program and mission needs, reflects budget constraints, and forms the basis for budget requests. An agency's IRM planning is also to coordinate with other agency planning processes including strategic, human resources, and financial resources. 91 8b(3)--Information Systems Management Oversight: Agencies are to establish information system management oversight mechanisms that (1) ensure that each information system meets agency mission requirements; (2) provide for periodic review of information systems; (3) ensure that the official who administers a program supported by an information system is responsible and accountable for the management of that information system throughout its life cycle; (4) provide for the appropriate training for users of federal information resources; (5) ensure that federal information system requirements do not unduly restrict the prerogatives of state, local, and tribal governments; (6) ensure that major information systems proceed in a timely fashion towards agreed-upon milestones in the information system's life cycle, meet user requirements, and deliver intended benefits to the agency and affected publics; and (7) ensure that financial management systems conform to the requirements of OMB Circular A-127. 8b(4)--Use of Information Resources: Agencies are to create and maintain management and technical frameworks for using information resources that document linkages between mission needs, information content, and information technology capabilities. These frameworks should guide both strategic and operational IRM planning. They should also address steps necessary to create an open systems environment. Among other requirements, agencies are to (1) develop information systems in a manner that facilitates necessary interoperability, application portability, and scalability of computerized applications across networks of heterogeneous hardware, software, and communications platforms, (2) ensure that improvements to existing information systems and the development of planned information systems do not unnecessarily duplicate information systems available within the same agency, from other agencies, or from the private sector, and (3) establish a level of security for all information systems that is commensurate with the risk and magnitude of the harm resulting from the loss, misuse, or unauthorized access to or modification of the information contained in these information systems. 8b(5)--Acquisition of Information Technology: Agencies are to acquire information technology in a manner that makes use of full and open competition, that maximizes return on investment, and that considers the need for accommodations of accessibility for individuals with disabilities to the extent that needs for such access exist. In addition, off-the-shelf software from commercial sources is to be acquired, unless the cost- effectiveness of developing custom software to meet mission needs is clear and has been documented. Finally, all information technology is to be acquired in accordance with OMB Circular A-109, where appropriate. 92 Executive Order 13011, "Federal Information Technology," highlights the need for executive agencies to significantly improve the management of their information systems, including the acquisition of information technology, by implementing the relevant provisions of PRA, the Clinger-Cohen Act, and GPRA. Agencies are to refocus their information technology management to directly support their strategic missions, implement an investment review process that drives budget formulation and execution for information systems, and rethink and restructure the way they perform their functions before investing in information technology to support that work. Agency heads are to strengthen the quality and decisions of employing information resources to meet mission needs through integrated analysis, planning, budgeting, and evaluation processes. Section 2(b)2 makes agency heads responsible for establishing mission-based performance measures for information systems investments that are aligned with agency performance plans prepared pursuant to GPRA. Section 2(b)3 makes agency heads responsible for establishing agencywide and project-level management structures and processes that are responsible and accountable for managing, selecting, controlling, and evaluating investments in information systems. Agency heads also have the authority to terminate information systems when appropriate. OMB Memorandum M-97-02 "Funding Information Systems Investments": This memo establishes eight decision criteria that OMB will use, starting with fiscal year 1998 budget proposals, to evaluate major information system investments proposed for submission in the President's budget. The first four decision criteria describe criteria related specifically to capital planning. The fifth criterion establishes the critical link between planning and implementation--the information architecture--which aligns technology with mission goals. The last three criteria establish risk management principles that are intended to help provide assurance that the proposed investment will succeed. 93 APPENDIX I APPENDIX I APPENDIX I PROPOSED STRUCTURE FOR PRESENTING RESULTS The following reporting structure is based on a previously completed evaluation of one organization's IT investment decision-making. STRENGTHS WEAKNESSES OVERALL Implemented partial investment management Meager evidence of business results from process: past and existing IT investments. • Created IRB and executive involvement Critical flaws in implementation still exist: • Established investment decision criteria • Started post-implementation reviews • lack of quality data used in investment decisions Plans established to address remaining • inadequate quantitative metrics (ROI, investment process elements. benefit/cost, and risk analysis) • low proportion of post-implementation Significance: A foundation has been put in reviews to new investments place for a complete investment process. • lack of smaller, more manageable project- level investment controls • inadequate tradeoffs of maintenance of existing systems to new investments Process is not institutionalized. Significance: Until process is complete, congressional funding of IT capital investment faces unnecessary risks. Until institutionalized, investment management will be sporadic and variable in character across a massive IT modernization program. 94 APPENDIX I APPENDIX I STRENGTHS WEAKNESSES SELECTION Good start on criteria, includes right Few current IT investments have the categories for decision criteria (cost, risk, minimal data requirements (ROI, benefit) and requirements for business case benefit/cost, risk analysis). and benefit/cost analyses. Significance: Poor quality information is Significance: Ensuring a more producing poor quality decisions; comprehensive analysis of projects than used acceptance of unjustified projects and in the past. misjudging of project costs, benefits, and risks. Have involved executive management team, including field representatives and Project-level cost and benefit data, in departmental executives. many cases, do not exist. Significance: Selected top-level management Significance: Perception of control does is engaged and learning from using an not match reality (large-size projects create investment-oriented, decision-making risk and instability). approach. Heavy reliance on qualitative, judgmental IRB is meeting frequently and making data and analysis at the expense of more decisions. quantitative measures, especially ROI. Significance: IRB is becoming a visible Significance: Decisions are weighted more decision-making entity with vested authority toward personal judgments without and responsibility for IT investment adequate consideration of critical, decisions. objective, independently verifiable facts about measurable mission impact. All modernization projects placed under a single authority. Inadequate tradeoffs among maintenance and new development projects. Significance: Single focal point for accountability for resource allocations of Significance: Inefficient allocation of capital IT investments. resources. = key deficiencies 95 APPENDIX I APPENDIX I STRENGTHS WEAKNESSES CONTROL Assessments of cost, schedule, and IT projects remain too large in size and performance are being made through program scope. control meetings. Significance: Increases cost and schedule Significance: Mechanism in place to engage risks beyond existing management and program sponsors on status of technology technical capabilities of the organization. modernization projects. More rigor needed in decisions being made Targeting go/no-go decisions at different at critical milestones ("tight gates" with no stages of project life-cycle. exceptions). Significance: Regular evaluations of cost, Significance: Some projects allowed to schedule, and some performance at critical proceed despite limitations in quality of project milestones. analysis. Consideration of internal and external audit Lack of systematic risk management findings. processes. Significance: Attempts being made to be Significance: Organization taking risks responsive to technical and managerial beyond capability to address or manage problems key to long-term improvement. them. Better coordination needed between BPR efforts and IT investment process. Significance: Business line reengineering efforts are not aligned with major modernization components which raises questions about long-term process improvement. 96 APPENDIX I APPENDIX I STRENGTHS WEAKNESSES EVALUATION Post-implementation review approach has Lack of proportional management been revised and is being used on projects. attention and emphasis on completing post-implementation reviews, despite Significance: Provides an evaluation having already made heavy existing and mechanism to compare expected results with new investments. actual outcomes. Significance: Track record does not exist Two post-implementation reviews now to justify new large-scale IT investments. complete, with four more scheduled for near- term completion. Complete evaluations (cost, schedule, AND completion performance) not being used at Significance: Creating factual, verifiable each project milestone. milestone evidence on results of IT investments. Significance: Systems being deployed with incomplete evidence on costs versus Initial efforts being made to learn from post- benefits. implementation reviews. Learning not widespread or quickly Significance: Beginning to show propensity translated into revised decision-making to start learning from past mistakes. processes. Significance: Slow learning rate equals slow improvement rate. = key deficiencies 97 APPENDIX II APPENDIX II APPENDIX II EXAMPLE OF ONE ORGANIZATION'S CRITERIA FOR COMPARING AND RANKING PROJECTS A. RISK (relative weight = 20 points) This category measures the risk resulting from uncertainty. The more risk carried by the project or system, the lower the score (except for the risk of not doing the project). A.1. Schedule Risk (4 of 20 points) Evaluate the probability this project can be completed on schedule. Score from zero to four points based on where the project best fits on a scale from very risky to low risk. Key characteristics of both ends of the scale are as follows: Zero Points: Very risky. Execution of project is likely to slip; acquisition strategy indicates contract may not be awarded in time to meet schedule or obligate budget year dollars. Project staff is limited in size and/or experience and project is complex. An accelerated project schedule was imposed rather than developed from project planning. Four Points: Low risk. Execution of project is not likely to slip; acquisition strategy should result in timely contract award such that funds can be obligated as planned. Adequate project staff is available and has requisite experience to execute the project; project is not complex. Project schedule has not been accelerated to meet artificial deadlines. A.2. Cost Sensitivity. (4 of 20 points) Evaluate the sensitivity or quality of the cost estimates. Score from zero to four points based on the scale described below. Zero Points: Very risky. Project is complex and cost estimates appear to require additional refinement. Software development is required and represents more than 50 percent of the predicated cost. Four Points: Low risk. Cost estimates are well supported. Little software development required or a software cost estimating technique has been used to produce a reasonably reliable cost estimate. 98 APPENDIX II APPENDIX II A.3. Technical Risk. (4 of 20 points) Evaluate the risk to complete the system from a technical point of view. Score from zero to four points based on the scale described below. Zero Points: Very risky. Hardware and/or software solution does not conform to organization's technical architecture and/or there is little or no experience with this technology in the organization. Hardware, software, or support is not now available commercially and requires development specifically for the organization. Four Points: Low risk. Planned hardware and software conform to organization's technical architecture and there is successful experience in using this technology in the organization. Hardware, software, and support are commercially available and do not have to be developed for use in the organization. A.4. Organizational Risk. (4 of 20 points) Assess the risk that the proposed system will fail due to organizational disruption. Score from zero to four points based on the scale described below. Zero Points: Very risky. System implementation requires significant organizational change, process redesign and/or people's jobs to be done differently and the project is not proactively seeking to mitigate this risk. Four Points: Low risk. System has little impact on the organization or the project is mitigating this risk through training and/or investment in a business process redesign effort which builds commitment to the system. A.5. Risk of Not Doing It. (4 of 20 points) Assess the risk to the organization of not proceeding with this project. Score from zero to four points based on the scale described below. Zero Points: Low risk. This in a incremental improvement to an existing system. The impact of this system can be achieved by other means. Four Points: Very risky. This system is important to provide future opportunities for cost savings and/or much improved customer service. If this system is not built or is delayed for a year, the organization will probably fail to meet customer demands in the near future. 99 APPENDIX II APPENDIX II B. ORGANIZATIONAL IMPACT (relative weight = 10 points) Measures the impact on organizational personnel of the system. The more favorable the impact on the organization the higher its score. B.1. Personnel and Training. (3 of 10 points) Assess the impact of the system on the knowledge, skill, and training of organizational personnel if the system in implemented. Score from zero to three based on the scale below. Zero Points: System is likely to require significant new skills to operate and support and the project does not appear to mitigate this impact through appropriate training, changes in rating qualifications, etc. Three Points: System is an improvement to an existing system and will require relatively little new skill and/or knowledge to operate or support. If it is a new system, it will introduce valuable new skills and knowledge to the organization and the project will mitigate any adverse impact through appropriate training, planning for rating qualification changes, etc. B.2. Scope of Beneficiaries/ Cross-Functional. (4 of 10 points) Assess a higher score (zero to four) the broader the scope of beneficiaries. Zero Points: Limited number of beneficiaries. This system will be used by only one office in headquarters, a single area, or district. Not a cross-functional system. Four Points: System is cross-functional and serves a number of offices, areas, and/or districts. System will be used by a large number of organizational units. System will be used by the public. B.3. Quality of Work Life. (3 of 10 points) Measures the improvement in quality of work life expected for the systems. Score higher (zero to three points) the more work life improvement is expected. Zero Points: Little if any positive impact on the quality of work life. System may increase the work required (e.g., additional data entry). Three Points: Positive contribution to the quality of work life will clearly result. For example, the system will improve medical care for dependents or allow a job to be done much faster such that job satisfaction will increase. 100 APPENDIX II APPENDIX II C. MISSION EFFECTIVENESS (relative weight = 20 points) Measures the impact of the system on both external and internal customers. It is a measure of the system's ability to improve the performance of support or operational programs. This improvement should be measured in quantitative terms, but not in dollars. The economic (dollar) impact is captured in the benefit/cost ratio. However, the same benefits might be measured here in a different manner. For example, improvements might be expressed in terms of accomplishing a task sooner (hours or minutes), delivering a service with fewer mistakes, increasing the availability of a computer system for customer use (hours per month saved in time for system backups), or a number of similar terms. The more the project or system improves mission effectiveness the higher the score. C.1. Improve Internal Program Services. (10 of 20 points). Assess the expected improvement in service to internal customers. For example the system might improve the timeliness of financial reporting throughout the organization. Score (zero to ten) higher, the more that service will be improved in response to a problem expressed by users of the service. Zero Points: System does not appear to meet a problem defined by an internal customer. Little improvement in important customer service criteria, such as timeliness, quality, or availability is expected. An improvement is described but not quantified. Ten Points: A significant improvement expected in areas such as timeliness, quality, or availability, and the improvement is quantified. The improvement also addresses an important problem or area of service improvement defined by the customer. C.2. Improved Service to the Public. (10 of 20 points). Assess the expected improvement in service to the public. Score (zero to ten) higher, the more improvement is anticipated in response to a requirement defined by the public. Zero Points: System appears to provide little or no direct improvement in service to the public. Systems may make a small improvement in timeliness, quality, or availability, but there is no documented need for such an improvement. The improvement is not quantified. Ten Points: System significantly improves service to the public in a mission where need is demonstrated or provides a new type of service to meet changing customer demands. The improvement is quantified. 101 APPENDIX II APPENDIX II D. STRATEGIC ALIGNMENT. (relative weight = 25 points) Measures to what extent the proposed investment supports strategic organizational objectives. Scoring is based primarily on explicit documentation of the need for the IRM system in planning documents. The more the project or system is aligned with program/strategic goals, the higher the score. D.1. Business Model. (7 of 25 points). Assess the degree of alignment with the organization's business model. Zero Points: Proposed project or system does not support organizational products/services or processes identified in the business model. One to Four Points: Proposed system is specifically mentioned in the 5-year IRM plan and supports organizational products/services or processes identified in the business model. (Score one to two points if the system supports products/services or processes in the business model, but is not listed in the 5-year IRM plan.) Five to Seven Points: Proposed system is specifically mentioned in the 5-year IRM plan and supports products/services or processes identified in the business model, and the project has been coordinated with all offices identified by the model for the respective processes the system supports. D.2. Level of Interest. (12 of 25 points). Assess the level of interest by senior managers (at agency and departmental level) and/or the Congress. Score (zero to twelve) higher the greater the level of interest. Zero Points: No expressed support for this system by senior managers or the Congress. Twelve Points: System has strong support from the Congress, departmental senior managers, and/or the head of the agency. System is specifically mentioned in determinations. D.3. Business Process Redesign. (6 of 25 points). Assess the degree this system enables the organization to do business in a better way. Score (zero to six) higher the greater the expected improvement. Zero Points: This system automates an existing business process with little improvement of the process (i.e., helps to do the same thing faster). Six Points: System enables a significant improvement in the way business is conducted. 102 APPENDIX II APPENDIX II E. BENEFIT-COST IMPACT(S) (relative weight = 25 points). Measures the value of the system in dollar terms. The system benefit/cost ration is the key indicator. This ration is developed using the standard benefit-cost guidance and spreadsheet promulgated by the organization. The standard guidance ensures all system studies include a common set of costs and approach benefits definition in a similar manner. The standard spreadsheet assists system sponsors in the benefit/cost calculation. The higher the benefit/cost ratio, the better the score. Zero Points: Any benefit/cost ratio less than one (i.e., costs exceed the benefits). One Point: Benefit/cost ratio of one. Five Points: Benefit/cost ratio of 1.5 to 1.75. Ten Points: Benefit/cost ratio of 1.76 to 1.99. Fifteen Points: Benefit/cost ratio of 2.0 to 2.99. Twenty Points: Benefit/cost ratio of 3.0 to 3.99. Twenty-five Points: Benefit/cost ratio of 4.0 or greater. 103 Glossary Activity: a named process, function, or task that occurs over time and has recognizable results. Activities use up resources to produce products and services. Activities combine to form business processes. Activity-Based Costing: a set of accounting methods used to identify and describe costs and required resources for activities within processes. Agency Capital Plan: document that identifies existing and proposed capital assets and that provides justification for new capital funding. Included in the capital plan should be a statement of the agency's strategic plan, a description of assets already owned by the agency or in procurement, an analysis detailing the performance gap between existing capabilities and the goals and objectives highlighted in the strategic plan, justification for new capital acquisitions proposed for funding, and other related information. Alignment: the degree of agreement, conformance, and consistency among organizational purpose, vision and values; structures, systems, and processes; and individual skills and behaviors. Annual Performance Plan: document, covering each program activity identified in an agency's budget, that describes the actions and goals that the organization will undertake during the year to work towards the long-term goals established in the organization's strategic plan. Specifically, the annual performance plan establishes the agency's performance goals for the year, describes strategies the agency will use to meet these goals, and identifies performance measures to measure or assess the relevant service levels, outcomes, or outputs that are to be achieved and to compare actual program results with the established performance goals. Annual Program Performance Report: report submitted with an agency's budget submission that compares actual agency performance to the annual goals established in the agency's annual performance plan. Baselining: obtaining data on the current process that provide the metrics against which to compare improvements and to use in benchmarking. Benchmark: a measurement or standard that serves as a point of reference by which process performance is measured. Benchmarking: a structured approach for identifying the best practices from industry and government, and comparing and adapting them to the organization's operations. Such an approach is aimed at identifying more efficient and effective processes for achieving intended results, and suggesting ambitious goals for program output, product/service quality, and process improvement. 104 Benefit: term used to indicate an advantage, profit, or gain attained by an individual or organization. Best Practices: the processes, practices, or systems identified in public and private organizations that performed exceptionally well and are widely recognized as improving a organization's performance and efficiency in specific areas. Successfully identifying and applying best practices can reduce business expenses and improve organizational efficiency. Business Case: a structured proposal for business improvement that functions as a decision package for organizational decision-makers. A business case includes an analysis of business process performance and associated needs or problems, proposed alternative solutions, assumptions, constraints, and a risk-adjusted cost-benefit analysis. Business Process: a collection of related, structured activities--a chain of events--that produce a specific service or product for a particular customer or customers. Business Process Reengineering: in government, a systematic disciplined improvement approach that critically examines, rethinks, and redesigns mission-delivery processes and subprocesses within a process management approach. In a political environment, the approach achieves radical mission performance gains in meeting customer and stakeholder needs and expectations. Business Vision: description of what senior management wants to achieve with the organization in the future. Business vision usually refers to the medium to long term and is often expressed in terms of a series of objectives. Capital Asset: Tangible property, including durable goods, equipment, buildings, installations, and land. Cost/benefit Analysis: a technique used to compare the various costs associated with an investment with the benefits that it proposes to return. Both tangible and intangible factors should be addressed and accounted for. Customer: groups or individuals who have a business relationship with the organization-- those who receive and use or are directly affected by the products and services of the organization. Customers include direct recipients of products and services, internal customers who produce services and products for final recipients, and other organizations and entities that interact with an organization to produce products and services. Cycle Time: the time that elapses from the beginning to the end of a process or subprocess. 105 Decision Criteria: documented set of factors that are used to examine and compare the costs, risks, and benefits of various IT projects and systems. These decision criteria consist of (1) screening criteria, which are used to identify whether new projects meet initial acceptance requirements and ensure that the project is reviewed at the most appropriate organizational level, and (2) criteria for assessing and ranking all projects. These ranking criteria weigh and compare the relative costs, risks, and benefits of each project against all other projects. Discount Rate: The interest rate used in calculating the present value of expected yearly benefits and costs. Discount Factor: The factor that translates expected benefits or costs in any given future year into present value terms. The discount factor is equal to 1/(1 + i)t where i is the interest rate and t is the number of years from the date of initiation for the program or policy until the given future year. Financial System: an information system, comprised of one or more applications, that is used for any of the following: collecting, processing, maintaining, transmitting, and reporting data about financial events; supporting financial planning or budgeting activities; accumulating and reporting cost information; or supporting the preparation of financial statements. Information Engineering: an approach to planning, analyzing, designing, and developing an information system with an enterprisewide perspective and an emphasis on data and architectures. Information Management: the planning, budgeting, manipulating, and controlling of information throughout its life cycle. Information Resources Management (IRM): the process of managing information resources to accomplish agency missions. This term encompasses information itself, as well as related resources, such as personnel, equipment, funds, and information technology. Information System: the organized collection, processing, transmission, and dissemination of information in accordance with defined procedures, whether automated or manual. Information systems include non-financial, financial, and mixed systems. Information Technology (IT): the hardware and software operated by an organization to accomplish a federal function, regardless of the technology involved (e.g., computers, telecommunications, etc.). 106 Information Technology Architecture: an integrated framework for evolving or maintaining existing IT and acquiring new IT to achieve the agency's strategic and IRM goals. A complete IT architecture should consist of both logical and technical components. The logical architecture provides the high-level description of the agency's mission, functional requirements, information requirements, system components, and information flows among the components. The technical architecture defines the specific IT standards and rules that will be used to implement the logical architecture. Intangible Benefit: benefits produced by an investment that are not immediately obvious and/or measurable. IT Investment Management Approach: an analytical framework for linking IT investment decisions to an organization's strategic objectives and business plans. The investment management approach consists of three phases--select, control and evaluate. Among other things, this management approach requires discipline, executive management involvement, accountability, and a focus on risks and returns using quantifiable measures. Investment Review Board (IRB): decision-making body, made up of senior program, financial, and information managers, that is responsible for making decisions about IT projects and systems, based on comparisons and trade-offs between competing projects and an emphasis on meeting mission needs and improving organizational performance. Life-cycle Cost: The overall estimated cost for a particular program alternative over the time period corresponding to the life of the program, including direct and indirect initial costs plus any periodic or continuing costs for operation and maintenance. Mixed System: an information system that supports both financial and non-financial functions. Model: a representation of a set of components of a process, system, or subject area. A model is generally developed for understanding, analysis, improvement, and/or replacement of the process. Modular Design: information system project design that breaks the development of a project into various pieces (modules) that each solve a specific part of the overall mission problem. These modules should be as narrow in scope and brief in duration as practicable. Such design minimizes the risk to an organization by delivering a net benefit that is separate from the development of other pieces. 107 Net Present Value (NPV): the future stream of benefits and costs converted into equivalent values today. This is done by assigning monetary values to benefits and costs, discounting future benefits and costs using an appropriate discount rate, and subtracting the sum total of discounted costs from the sum total of discounted benefits. Outcome: the ultimate, long-term, resulting effect--both expected and unexpected--of the customer's use or application of the organization's outputs. Performance Gap: the gap between what customers and stakeholders expect and what each process and related subprocesses produces in terms of quality, quantity, time, and cost of services and products. Performance Measurement: the process of developing measurable indicators that can be systematically tracked to assess progress made in achieving predetermined goals and using such indicators to assess progress in achieving these goals. Post-implementation Review (PIR): an evaluation tool that compares the conditions prior to the implementation of a project (as identified in the business case) with the actual results achieved by the project. Return on Investment (ROI): a figure of merit used to help make capital investment decisions. ROI is calculated by considering the annual benefit divided by the investment amount. Risk Analysis: a technique to identify and assess factors that may jeopardize the success of a project or achievement of a goal. This technique also helps define preventive measures to reduce the probability of these factors from occurring and identify countermeasures to successfully deal with these constraints when they develop. Sensitivity Analysis: analysis of how sensitive outcomes are to changes in the assumptions. The assumptions that deserve the most attention should depend largely on the dominant benefit and cost elements and the areas of greatest uncertainty of the program or process being analyzed. Stakeholder: an individual or group with an interest in the success of an organization in delivering intended results and maintaining the viability of the organization's products and services. Stakeholders influence programs, products, and services. Examples include congressional members and staff of relevant appropriations, authorizing, and oversight committees; representatives of central management and oversight entities such as OMB and GAO; and representatives of key interest groups, including those groups that represent the organization's customers and interested members of the public. 108 Strategic Plan: document used by an organization to align its organization and budget structure with organizational priorities, missions, and objectives. According to requirements of GPRA, a strategic plan should include a mission statement, a description of the agency's long-term goals and objectives, and strategies or means the agency plans to use to achieve these general goals and objectives. The strategic plan may also identify external factors that could affect achievement of long-term goals. Strategic Planning: a systematic method used by an organization to anticipate and adapt to expected changes. The IRM portion of strategic planning sets broad direction and goals for managing information and supporting delivery of services to customers and the public and identifies the major IRM activities to be undertaken to accomplish the desired agency mission and goals. Sunk Cost: a cost incurred in the past that will not be affected by any present or future decision. Sunk costs should be ignored in determining whether a new investment is worthwhile. Tangible Benefit: a benefit produced by an investment that is immediately obvious and measurable. Value-Added: those activities or steps that add to or change a product or service as it goes through a process; these are the activities or steps that customers view as important and necessary. 109
Assessing Risks and Returns: A Guide for Evaluating Federal Agencies' IT Investment Decision-making
Published by the Government Accountability Office on 1997-02-03.
Below is a raw (and likely hideous) rendition of the original report. (PDF)