United States General Accounting Office GAO Accounting and Information Management Division ________________________________________________ September 1997 Executive Guide Measuring Performance and Demonstrating Results of Information Technology Investments EXPOSURE DRAFT GAO/AIMD-97-163 Preface The Government Performance and Results Act of 1993 requires government executives to focus on defining missions, setting goals, measuring performance, and reporting accomplishments. In addition, with the passage of the Federal Acquisition Streamlining Act of 1994 (FASA) and the Clinger-Cohen Act of 1996, performance-based and results- oriented decision-making is now required for all major investments in information technology (IT). Clearly, this intense focus on results is one of the most important management issues now confronting federal agencies. To assist federal agencies in understanding and devising effective IT measurement implementation approaches, we examined certain public and private organizations well- known for their IT performance leadership and management expertise. Similar to our past efforts examining comprehensive information management practices of other leading organizations,1 we have taken the lessons learned from these organizations and developed a suggested framework for agencies to consider when designing and implementing their IT performance management approaches. We have briefed numerous Chief Information Officers, agency executives, and agency IT managers on our work over the last 6 months as part of our effort to advance a pragmatic understanding of what is required to effectively measure the contribution of IT to mission performance and program outcomes. Using comprehensive performance information for information management and technology decisions can advance more informed decision-making about IT investments at a time when resources are limited and public demands for better government service are high. Ultimately, the success of results-oriented reform legislation will demand concerted management effort and long-term commitment. The key practices and steps outlined in this guide can help agencies achieve success. This exposure draft was prepared under the direction of Dave McClure, Senior Assistant Director for Information Resource Management Policies and Issues. If you have questions or comments about the report, he can be reached at (202) 512-6257. Other major contributors are listed in appendix IV. Gene L. Dodaro Assistant Comptroller General Accounting and Information Management Division 1 Executive Guide: Improving Mission Performance Through Strategic Information Management and Technology--Learning From Leading Organizations (GAO/AIMD-94-115), May 1994; Assessing Risks and Returns: A Guide for Evaluating Federal Agencies' IT Investment Decision-making (GAO/AIMD-10.1.13), February 1997. GAO/AIMD-97-163 IT Performance Measurement Guide Page 1 GAO/AIMD-97-163 IT Performance Measurement Guide Page 2 Contents Page Preface 1 The Demand for Performance Management 4 Fundamental Practices: The Foundation of IT Performance Management 11 Practice Area 1: Follow an Information Technology "Results Chain" 18 Practice Area 2: Follow a "Balanced Scorecard" Approach 31 Practice Area 3: Target Measures, Results, and Accountability at Decision-making Tiers 45 Practice Area 4: Build a Comprehensive Measurement, Data Collection, and Analysis Capability 53 Practice Area 5: Strengthen IT Processes to Improve Mission Performance 64 Key Lessons Learned for Effective Implementation 70 Appendix I: Selected Bibliography 76 Appendix II: Objectives, Scope and Methodology 79 Appendix III: Case Study Organizations and Participants 81 Appendix IV: Major Contributors to This Report 82 Page 2 GAO/AIMD-97-163 IT Performance Measurement Guide Figures Page Figure 1: IT Performance Management Approach 12 Figure 2: Performance Measurement - A Strategic Information Management Best Practice 15 Figure 3: Implementing the Results Act - Key Steps and Critical Practices 16 Figure 4: An IT Results Chain 19 Figure 5: A Hypothetical Chain of Events/Evidence for Achieving Environmental Quality 27 Figure 6: An IT Balanced Scorecard Approach 33 Figure 7: Balanced Scorecard - IT Strategic Measures 37 Figure 8: Balanced Scorecard - IT Customer Measures 39 Figure 9: Balanced Scorecard - IT Internal Business Measures 41 Figure 10: Balanced Scorecard - IT Innovation and Learning Measures 43 Figure 11: Performance Measurement Tiers 46 Figure 12: A Tiered Performance Scorecard Example 50 Figure 13: Information Systems Process Architecture Version 2.0 Process Framework 65 Figure 14: IT Measurement Implementation Roadmap 72 Figure 15: Stages in IT Performance Management 74 GAO/AIMD-97-163 IT Performance Measurement Guide Page 3 The Demand for Performance Management _________________________________________________________________________________ Increasingly, federal policy-makers are insisting that government executives provide hard facts on mission and program results. Program authorizations, resource decisions, and oversight requirements increasingly hinge on how well agencies perform against expectations and improve performance over time. As such, a new standard for management expertise is evolving: setting performance targets, designing efficiency and effectiveness measures, systematically and accurately measuring outcomes, and then using the results for informed decision- making. Information technology (IT) products, services, and delivery processes are important resources for results-driven government programs and operations. For purposes of this guide, IT also includes the organizational unit or units and contractors primarily responsible for delivering IT. Line managers--the operational customers2 relying on IT products and services--and IT managers themselves, want to know "How are information technology products and services, including the information infrastructure, supporting the delivery and effectiveness of the enterprise's (agency) programs?" As we pointed out in an earlier report, successful organizations rely heavily on performance measures to operationalize mission goals and objectives, quantify problems, evaluate alternatives, allocate resources, track progress, and learn from mistakes.3 Operational customers and IT managers in these organizations form partnerships to design, manage, and evaluate IT systems that are critical to achieving improved mission success. Performance Management: What Are the Benefits? In an effective performance management approach, measures are not used for assigning blame or to unknowingly comply with reporting requirements. Quite simply, they are used to create and facilitate action to improve performance. Measures and performance information must link to strategic management processes. An effective performance 2 For purposes of this guide, we define an "operational customer" as a program or other function to whom the information technology organization or units delivers IT products and services. The operational customer can include organizations and entities outside traditional agency boundaries affected by the use of IT products and services. 3 GAO/AIMD-94-115. Page 4 GAO/AIMD-97-163 IT Performance Measurement Guide management system produces information that delivers the following benefits.4 • Provides an early warning indicator to correct problems, or to examine if corrective action is having any effect. • Provides input to resource allocation and planning. It can help organizations prepare for future conditions that likely will impact program and support function operations and the demands for products and services, such as decreasing personnel, financial resources or changes in workload. Use of measures can give organizations a long lead time for adjustment if these conditions are known in advance. • Provides periodic feedback to employees, customers, stakeholders, and the general public about the quality, quantity, cost, and timeliness of products and services. Across all of these benefits is an overarching one -- measures build a common results language among all decision-makers. What measures the organization picks basically say what the organization is accountable for, and what it should benchmark and compare against. Results-Oriented Legislation Provides Impetus for Performance Based Management For the past several years, the Congress has emphasized federal performance improvement, accountability for achieving results, and cost reduction. Legislative requirements in the Chief Financial Officers Act (CFO) of 1990, the Government Performance and Results Act (Results Act) of 1993, the Federal Acquisition Streamlining Act (FASA) of 1994, the Paperwork Reduction Act of 1995, and the Clinger-Cohen Act of 1996 expect improvements in IT performance management. These laws reinforce financial accountability, emphasize results-oriented management, define cost performance and schedule goals, and improve the acquisition of IT to streamline federal programs. Under the CFO Act, CFOs are responsible for developing and maintaining integrated accounting and financial management systems that include systematic measurement information on agency performance. OMB's 4 This information is based on material in Jack A. Brizius and Michael D. Campbell, Getting Results (Washington, D.C., Council of Governors' Policy Advisors, 1991) and performance management guidance materials prepared by the National Academy of Public Administration and the Innovation Group. GAO/AIMD-97-163 IT Performance Measurement Guide Page 5 Management Circular A-11 (agency guidance for preparing and submitting budget estimates) encourages agencies to review program performance information contained in the most recent financial statements prepared under the CFO Act when developing their program performance indicators. The Results Act directs federal agencies to improve their program management by implementing outcome-oriented performance measurement systems. Agencies are to prepare annual performance plans with objective, quantifiable, and measurable performance indicators to measure relevant outputs, service levels, and outcomes of each program activity. Meeting these requirements is critical in developing the mission goals and performance expectations which IT products and services will support. Our Executive Guide: Effectively Implementing the Government Performance and Results Act contains additional guidance on results-oriented management under the Results Act.5 The Federal Acquisition Streamlining Act requires federal agencies to assess major acquisition cost, performance, and scheduling. Agency heads must determine if there is a continuing need for programs that are significantly behind schedule, over budget, or not in compliance with performance or capability requirements. Congressional policy is that each executive agency should achieve, on average, 90 percent of the cost and schedule goals established for agency programs. The Paperwork Reduction Act requires agencies to establish information resources management goals improving the productivity, efficiency, and effectiveness of agency operations and methods for measuring progress in achieving the goals. OMB's Circular A-130 (guidance resulting from the Paperwork Reduction Act) highlights the importance of evaluation and performance measurement. It recommends that agencies seek opportunities to improve the effectiveness and efficiency of government programs through work process redesign and the judicious application of IT. Agencies must perform various benefit-cost analyses to support ongoing management oversight processes and conduct post- implementation reviews of information systems to validate estimated benefits and document effective management. The most recent legislation affecting IT performance management, the Clinger-Cohen Act, requires agencies to establish efficiency and effective program improvement goals using IT. Performance measurements must assess how well IT supports agency programs. Agency heads must benchmark agency process performance against comparable processes in terms of cost, speed, productivity, and quality of outputs and outcomes. Agency heads must analyze agency missions and make appropriate 5 See Executive Guide: Effectively Implementing the Government Performance and Results Act (GAO/GGD-96-118, June 1996). Page 6 GAO/AIMD-97-163 IT Performance Measurement Guide changes in mission and administrative processes before making significant IT investments to support missions. Annual performance reports cover how well each agency improves agency operations through IT. OMB has issued specific guidance to assist agencies in the implementation of the Clinger-Cohen Act that requests specific information on IT cost, benefit, and risk. Most notable is OMB Management Circular A-11, Part 3, that provides instructions on agency budget submissions and the OMB Director's policy memorandum M-97-02 (also known as "Raine's Rules") that specifies investment criteria that IT projects are to meet to qualify for inclusion in the President's budget submission to the Congress. These various legislative and executive branch requirements create pressure for top management's increased attention on IT performance. However, for federal agency managers, the challenge is not complying with these legislative and regulatory requirements, but "managing and measuring to results" using a well-crafted IT performance management system. At a high level, federal managers can start framing a system by asking: • What are enterprise (agencywide) and key operational customer IT performance expectations? • What are the vital few IT objectives given these expectations? • What measures are appropriate for these IT objectives? • What is IT's current baseline performance and what should be the target performance? • How will IT management and customers work together to use these measures to leverage and improve IT performance in ways that will improve mission delivery? Answering these questions signals a significant change in determining how IT contributes to achieving improved program outcomes. Traditional measures such as response time and systems availability are by themselves insufficient to answer IT performance questions. "Measures such as machine hours are easy," said one manager we interviewed, "what is difficult is how to measure the business value of the applications." GAO/AIMD-97-163 IT Performance Measurement Guide Page 7 Identifying Best Practices: Learning from Leading Organizations Federal managers are seeking guidance on developing and implementing agency IT performance management systems.6 To assist agencies, we examined how certain leading organizations approach IT performance management, studying practices of both public and private sector organizations recognized by peers and independent researchers for their IT performance efforts. (A more detailed description of our case study selection methodology is found in appendix 2.) The private sector companies used as case studies were • Xerox Corporation, • Eastman Kodak Company, • Texas Instruments, • Motorola Semiconductor Products Sector, and • American Express Travel Related Services Company. In the public sector, we studied • Oregon Department of Transportation, • Sunnyvale, California, and • Phoenix, Arizona. We also selectively included the U.S. Immigration and Naturalization Service, the U.S. General Services Administration Information Technology Service, and the U.S. Department of Agriculture to assess 6 Recent efforts include The Connection: Linking IRM and Mission Performance, a resource paper sponsored by the Association for Federal Information Resources Management, September 1995; Practitioner's Guide to I.S. Performance Measurement, a guide issued by the Society for Information Management's Advanced Practices Council, March 1995; Information Management Performance Measures, a report for the U.S. Department of Defense issued by the National Academy of Public Administration, January 1996, Performance-Based Management: Eight Steps to Develop and Use Information Technology Performance Measures Effectively, a guide prepared by the General Services Administration, December 1996; and Guide for Managing Information Technology (IT) as an Investment and Measuring Performance, Department of Defense guidance issued by Assistant Secretary of Defense for Command, Control, Communications, and Intelligence, March 3, 1997. Page 8 GAO/AIMD-97-163 IT Performance Measurement Guide early federal practices.7 We gathered additional information from general and IT performance management literature and reports. We preceded the organizational research by an extensive review of the generic performance management and IT performance management and measurement literature, guides, and reports. We also consulted with experts involved in IT performance management and measurement efforts. We collected organizational data through interviews and documentary analysis, not direct observation. Case study organizations reviewed our results for accuracy and completeness. We also gave briefings to federal officials to discuss our results. Appendix II provides a more detailed description of our scope and methodology. Our guide contains key practices we found from our organizational research and key concepts and practices extracted from available literature, guides, and reports. They supplement recent GAO reports and testimonies.8 Much still remains to be learned. This guide is an initial effort in an area very much in its infancy. Without exception, those in the organizations we studied noted that IT performance management and measurement practices are not completely defined. However, their experiences did translate to the key practices we describe in this guide. Understanding the Context of IT Performance Management We have found from our research that there is not one "best" approach to IT performance management. How IT performance management is designed, implemented, and sustained in each organization depends on a multitude of contextual factors, such as 7 Other organizations we consulted were the Florida Legislature Joint Committee on Information Technology Resources, the Oregon Department of Administrative Services, the Oregon Secretary of State Audits Division, the Office of the Texas State Auditor, the Minnesota Office of the Legislative Auditor, the Minnesota Department of Transportation, the Federal Emergency Management Agency, the city of Portland, the Treasury Board of Canada, the United Kingdom's Government Centre for Information Systems and Royal Mail, the Society for Information Management, and the MITRE Corporation. 8 These include Government Reform: Goal-Setting and Performance (GAO/AIMD/GGD-95- 130R, March 27, 1995); Managing for Results: Experiences Abroad Suggest Insights for Federal Management Reform (GAO/GGD-95-120, May 2, 1995); Managing for Results: Critical Actions for Measuring Performance (GAO/T-GGD/AIMD-95-187, June 20, 1995); Managing for Results: Achieving GPRA's Objectives Requires Strong Congressional Role (GAO/T-GGD-96-79, March 6, 1996); The Government Performance and Results Act: 1997 Governmentwide Implementation Will Be Uneven (GAO/GGD-97-106, June 2, 1997); Managing For Results: The Statutory Framework for Improving Federal Management and Effectiveness (GAO/T-GGD/AIMD-97-144, June 24, 1997). GAO/AIMD-97-163 IT Performance Measurement Guide Page 9 Page 10 GAO/AIMD-97-163 IT Performance Measurement Guide • whether the organization's culture--leadership, decision-making, appraisal and reward systems--supports IT performance management; • how important IT is for program (mission) delivery; • how widespread IT is used in the organization; • what IT activities are centralized, dispersed, or decentralized; and • the availability of resources such as skills and tools to support performance management. Factors such as these, taken together, provide a unique environment impacting IT performance. However, as with most things important to organizational health, another significant contextual factor is top management ownership, involvement in, and use of IT performance information. What the top leaders pay attention to, what messages they send about IT performance management, and what action they take based on the measures tells the true story of IT performance management acceptance in any organization. In the organizations we studied, there is strong management attention on IT measures and their rigorous use in decision-making at all management levels to improve IT performance. A second significant factor for effective IT performance is the partnership forged between IT and the enterprise and operational customers. In a large sense, the enterprise and operational customers "co-produce" IT results because they are the consumers and users of IT products and services. The starting point for IT objectives is organizational goals and objectives. What IT does and what is measured must directly align with those organizational goals and objectives. While IT management and staff serve as business consultants on how current and emerging IT can aid mission objectives, they must rely on the ongoing engagement of organizational customers in defining how IT can facilitate mission accomplishment. GAO/AIMD-97-163 IT Performance Measurement Guide Page 11 Fundamental Practices: The Foundation of IT Performance Management ______________________________________________________________________________ Our case study research clearly indicated that knowing what performance management is and is not is the starting point for developing an IT performance management system. In simple terms, any performance management system assesses how well an organization delivers expected products and services directly that are tied to its goals and objectives. It also incorporates the products and services from enterprise and program support functions such as IT, financial management, or human resources management. Within this context, IT performance management and measures are considered subsets of overall performance management systems. Our IT performance management approach includes several important distinguishing characteristics that are discussed in greater depth in different parts of this guide. These characteristics include: • differentiating between IT''s impact on intermediate versus final program outcomes, • using a good balance of different kinds of IT measures, • understanding that measures may differ by management tier within an organization, and • evaluating both the overall performance of the IT function within an organization and the outcomes for individual IT investments. Our approach suggests three distinct practice areas that involve aligning IT systems with agency missions, goals, and programs; constructing measures that determine how well IT is supporting strategic, customer, and internal business needs; and implementing performance measurement mechanisms at various decision-making levels within an organization. Two supporting practice areas are important to keep the overall IT measurement process working. Data collection and analysis capabilities must effectively support the performance management system being used in such a way that performance data is accessible, reliable, and collected in the least burdensome manner. The benefit of effective automated data and management information systems is that performance information can be effectively and efficiently used to make strategic, managerial, and day-to-day operational decisions. In addition, a constant focus on strengthening the processes and practices being used to deliver IT products and services is essential for building and maintaining effective IT organizations. Page 12 GAO/AIMD-97-163 IT Performance Measurement Guide Figure 1 shows the generic model produced by our case study research on IT performance measurement. Figure 1: IT Performance Management Approach ALIGN Practice Area 1 Use an IT Results Chain IMPLEMENT CONSTRUCT Practice Area 4 Practice Area 3 Build data Practice Area 2 Target Measures Collection/analysis Follow a at Capability Balanced Decision-making Scorecard Levels Practice Area 5 Approach Improve IT Processes REINFORCE Practice Area 1: Follow an IT "results chain" Leading organizations build and enforce a disciplined flow from goals to objectives to measures and individual accountability. They define specific goals, objectives, and measures, use a diversity of measure types, and develop a picture as to how IT outputs and outcomes directly impact operational customer and enterprise (agency) program delivery requirements. The IT performance management system does not optimize individual customer results at the expense of an enterprise (agency) perspective. Operational customer goals and measures meet IT department or unit objectives that are matched to enterprise strategic directions or goals. GAO/AIMD-97-163 IT Performance Measurement Guide Page 13 Practice Area 2: Follow a balanced scorecard approach Leading organizations use an IT goal, objective, and measure approach that translates organizational strategy and IT performance expectations into a comprehensive view of both operational and strategic measures. Four generic goal areas include meeting the strategic needs of the enterprise, meeting the needs of individual operational customers, addressing internal IT business performance, and addressing ongoing IT innovation and learning. Practice Area 3: Target measures, results, and accountability at different decision-making tiers For the balanced scorecard areas, leading organizations match measures and performance results to various decision-making tiers or levels. These tiers cover enterprise executives, senior to mid-level managers responsible for program or support units, and lower-level management running specific operations or projects. The organizations we studied place IT goals and measures in widely distributed IT performance improvement plans. Individual appraisals tie IT performance to incentives. Practice Area 4: Build a comprehensive measure, data collection, and analysis capability Leading organizations give considerable attention to baselining, benchmarking, and the collection and analysis of IT performance information. They use a variety of data collection and analysis tools and methods which not only keep them on top of IT performance management, but reduce the burden of collection and analysis. They also periodically review the appropriateness of their current measures. Practice Area 5: Improve performance of IT business processes to better support mission goals In the leading organizations, IT performance improvement begins and ends with IT business processes. The organizations map their IT business processes and select those processes which must be improved to support an enterprise and operational customers' business processes. Page 14 GAO/AIMD-97-163 IT Performance Measurement Guide Measurement Maturity: Start With the Basics and Increase Sophistication Over Time Developing performance measures that demonstrate the impact of information technology on mission performance requires management commitment, experience in constructing and evaluating measures, and a constant learning environment. Many of the organizations we talked to indicated that had attempted to develop strategic or mission impact measures without first realizing that they had to demonstrate strong capability and sound performance in the basics of IT management. In short, if an IT unit was not being very successful in providing quality products and services to the rest of the organization, it had little credibility in measuring strategic contributions to mission or business results. As such, several IT managers emphasized the need to start with the basics by assessing the quality and effectiveness of existing internal IT operations. This evaluation and early measurement construction exercise can focus on such things as (1) delivery of reliable, cost-effective, high quality IT products and services, (2) adherence to industry standards for systems design, cost estimation, development, and implementation, (3) internal customer satisfaction, (4) staff productivity, and (5) technical skills and capability. All of these factors are important dimensions in providing effective IT support to business operations and improving overall organizational performance. Starting with measures of internal IT operations offers some advantages, even though they should be viewed as a substitutes for other measurements of IT contribution to specific program and mission area results. First, it preempts the problem of IT organizations waiting for development and consensus of mission or business specific performance measures. Second, it provides the IT organization with valuable experience in performance measurement construction and evaluation which is easily transferable to mission-related measures. Third, constructing performance measures for IT operations conforms with a balanced scorecard approach which emphasizes the need for a diversity of measures in examining IT performance. And fourth, introducing performance measurement maturity over time is a critical factor affecting the overall success of implementing performance management in an organization--an issue we discuss in greater detail in the final section of this report. GAO/AIMD-97-163 IT Performance Measurement Guide Page 15 IT Performance Is Essential for Strategic Information Management Performance measurement is not an end, but rather the means to achieving better management results. In our May 1994 Executive Guide on strategic information management,9 we noted that leading organizations use performance measures to objectively evaluate mission, business, and project outcomes. These organizations (1) focused performance measures on gauging service to key management processes, (2) embedded performance measures in key management processes, (3) used internal and external benchmarks to assess relative performance, and (4) tailored performance measures to gauge whether information technology made a difference in improving performance. As shown in figure 2 performance measurement is a cornerstone practice that GAO advocates as part of an integrated strategic information management approach. Figure 2: Performance Measurement--A Strategic Information Management Best Practice Decide to Communicate the urgency to change IT practices Change Get line management involved and create ownership Take action and maintain momentum Define customer needs and mission goals Direct Measure the performance of mission processes Change Focus on process improvement Manage IT projects as investments Integrate planning, budgeting, and evaluation Establish customer/supplier relationships Support between managers and IT professionals Change Create a Chief Information Officer Upgrade IT skills and knowledge 9 GAO/AIMD-94-115. Page 16 GAO/AIMD-97-163 IT Performance Measurement Guide In June 1996, GAO issued a companion Executive Guide on a suggested performance measurement implementation approach for the Results Act.10 The approach, depicted in figure 3 identifies certain key steps and associated practices that agencies may find useful for implementation of the Results Act. The approach is based on the actions taken by certain organizations that have successfully undertaken performance improvement initiatives similar to that required by the act. Figure 3: Implementing The Results Act -- Key Steps and Critical Practices STEP 1 Define Mission and Desired Outcomes Practices: 1. Involve stakeholders 2. Assess environment 3. Align activities, core processes, and resources STEP 3 STEP 2 Reinforce GPRA Measure Performance Use Performance Information Implementation Practices: Practices: 4. Produce measures at Practices: 9. Devolve decision-making each organizational with accountability level that 6. Identify performance demonstrate results, gaps 10. Create incentives are limited to a vital 7. Report information 11. Build expertise few, 8. Use information 12. Integrate management respond to multiple reforms priorities, and link to responsible programs 5. Collect data Each organization GAO studied set its agenda for management reform according to its own environment, needs, and capabilities. In striving to become more results oriented, they commonly took three steps in implementing a performance based management approach. First, they defined clear missions and desired outcomes. Second, they measured performance to gauge progress. Third, they used performance information as a basis for decision-making. 10 GAO/GGD-96-118. GAO/AIMD-97-163 IT Performance Measurement Guide Page 17 Page 18 GAO/AIMD-97-163 IT Performance Measurement Guide Along with these steps, certain practices proved especially important to the success of their efforts. Taken together, these steps and practices were useful in making changes necessary for these organizations to become results oriented. These fundamental steps and practices are consistent with the Results Act requirements and provide a useful framework for federal agencies to adopt in implementing key provisions of the law. The IT performance management approach outlined is this guide works in tandem with GAO's Results Act implementation model and demonstrates how IT performance measurement can be implemented within an overall performance management framework. Most importantly, mission goals, objectives, and strategies must be understood in order to evaluate how IT contributes to performance improvements. In the sections that follow, each practice area in our approach is explained in detail, listing specific characteristics and providing case study examples to illustrate how they are implemented. A final section discusses key steps involved in implementing an IT performance management system. GAO/AIMD-97-163 IT Performance Measurement Guide Page 19 Practice Area 1: ALIGN Follow An Practice Area 1 Use an Information IMPLEMENT IT results chain CONSTRUCT Technology Practice Area 3 Practice Area 4 Build data Practice Area 2 "Results Chain" Target measures at decision-making collection/analysis capability Follow a balanced scorecard levels Practice Area 5 approach Improve IT processes REINFORCE Practice Area Characteristics: 1. Directly map IT goals and measures to organizational mission goals, objectives, and measures. 2. Prepare a chain of events and evidence to understand IT's contribution to enterprisewide and operational customer objectives 3. Use a diversity of measures to evaluate IT performance Practice Area Overview To maximize the results of IT investments, leading organizations ensure that IT programs align with and directly support high-level organizational missions, goals, and objectives. This practice provides an approach for linking organizational goals and objectives to the "vital few" IT performance measures needed to manage for effective results. This framework, formal or informal, follows a systematic movement through what can be called an "IT results chain." The results chain approach provides discipline for aligning performance expectations and measures at all levels. Any effort to measure IT performance must begin with clearly defined organizational and programmatic goals and objectives. In other words, an organization cannot properly define its IT goals and objectives (much less measure to determine the degree of success in meeting them) unless it has clearly defined goals and objectives for the programs that IT supports. The resulting IT goals and measures must, in all cases, map back to program or strategic (enterprise-level) goals. To help understand the relationships between operational programs and the IT contribution Page 18 GAO/AIMD-97-163 IT Performance Measurement Guide to their success, many organizations prepare a chain of events and evidence to show how programs work and how success might be measured. Finally, a diversity of qualitative and quantitative measures are used for the inputs, outputs, and outcomes of IT programs. In short, a results chain approach • defines what the organization is attempting to accomplish, • allows an organization to identify success, and • links IT projects directly to business goals and objectives. As shown in figure 4, the chain shows the links from organizational goals and objectives to IT performance measures. In the organizations we studied, measuring IT's contribution begins by defining organizational goals and objectives--for the enterprise and for internal and external customers. These goals and objectives should be based on defined organizational mission statements. Figure 4: An IT Results Chain M ea Organizational su a onv and di re nd erg me v Goals & Objectives e rg m A e as e/ en lig go ur c t D nm als es IT Purpose ev en , ob (Mission/Vision) el t jec op m IT Goals en t tiv C es Vital Few C ctio ol , M mu An om n IT Objectives le ea ni al su ca ys re tio is, Vital Few , IT Measures n, U IT Management and Staff se Performance Expectations GAO/AIMD-97-163 IT Performance Measurement Guide Page 19 Page 20 GAO/AIMD-97-163 IT Performance Measurement Guide As for other programs, IT management and staff can then develop a purpose statement that specifically defines how IT products and services will be used to support the achievement of organizational and customer goals. The purpose statement is then translated into IT goals, objectives, and measures. IT goals and objectives should be consistent with the IT purpose statement and clearly linked to organizational goals and objectives. Leading organizations focus on a "vital few IT objectives" to demonstrate results in selected key areas. Similarly, the number of measures for each IT goal should be limited to the "vital few." These should be limited to the key IT performance dimensions that will enable the IT organization to assess accomplishments, make decisions, realign processes, and assign accountability. Attempts to manage an excess number of IT measures increase risks of confusing excess data with key IT performance issues. Lastly, management and staff performance evaluations are linked to the performance measures as a way of comparing achievements with planned results. As part of a results-oriented management approach, IT performance measurement must be used in the decisionmaking process. Measurement development and alignment involves consideration of all organizational goals and objectives and converging on the vital few IT goals, objectives, and measures. Effective measurement must be supported by sound data collection and analysis methods and communication of results to management and staff. Practice Area Characteristics 1. Directly Map Information Technology and Management Goals and Measures to Strategic Goals Use of an IT results chain is only as good as the clarity and specificity of the overall organizational goals and objectives. Leading organizations build consensus among program managers, IT managers, customers, stakeholders, and staff to establish joint ownership for performance management. They work together to achieve a common understanding of goals, objectives, measures, and anticipated outcomes. As a practical matter, those who will judge the success of programs and the supporting functions should agree on the links in the results chain from IT's purpose to the vital few measures. 11 11 For related guidance, see Agencies' Strategic Plans Under GPRA: Key Questions to Facilitate Congressional Review (GAO/GGD- 10.1.16, May 1997). GAO/AIMD-97-163 IT Performance Measurement Guide Page 21 In the organizations we examined, IT goals and measures flow directly from strategic goals. IT managers and staff do not develop performance management systems that optimize operational customer results without considering an enterprisewide perspective. IT goals and measures in support of individual operational customers must meet IT department or unit objectives. In turn, IT department or unit objectives must map directly to both programmatic and enterprisewide strategic directions or goals. The result is that IT goals and measures track in a seamless fashion back to enterprise strategic directions or goals. If such mapping is not obvious when comparing measures and high-level goals, the IT function is probably not measuring the right things. Case Study 1: Linking Measures to Specific IT Goals and Objectives The city of Phoenix's Information Technology Department (ITD) wanted a set of business-oriented customer-based measurements to measure the effectiveness of its ITD operations. One manager said, "We set forth to come up with a 'net result measuring scheme.' It would (1) focus on customer satisfaction, (2) be employee friendly, (3) not affect program work, (4) improve customer satisfaction, and (5) give the customer the ability to understand the value our organization gives to them, and let them suggest how to help us get better." The department wanted to (1) lay the groundwork for continuous monitoring and improving customer satisfaction with ITD service, (2) better define the value-added of ITD services from a business perspective, (3) identify continuous improvement needs in internal ITD operations, and (4) strengthen ITD relationships with other city departments and manage customer expectations using hard facts and measures. The department identified measures based on input from key selected customers and staff and the ITD objectives. Objectives were developed using the City of Phoenix Strategic Directions Report and the IT architecture vision. Generally, the strategic directions were to (1) create and maintain a well-informed and involved community with a clean and safe environment, efficient transportation systems, a quality education system, and economic opportunity for all citizens, (2) provide cost-effective, high-quality community services through a productive, efficient organization and an empowered work force, and (3) generate and maintain desirable revenue flows and sound financing within a system of rational resource allocation. Goals for the architecture vision were to make all applications integrated and fully compatible, using a city-wide deployed IT architecture supported by commonly shared data repositories. The department then revised the measures to meet all of the ITD objectives and correlated the measures to customer-defined goals of time, cost, and customer satisfaction. The table on the following page shows how Phoenix measures progress towards achieving department objectives. These measures focus on timeliness (service delivery), cost (producing city services at acceptable or reduced cost), or customer satisfaction (evidence that internal and external customers are satisfied and that ITD's contribution can be quantified). Page 22 GAO/AIMD-97-163 IT Performance Measurement Guide Case Study 1: The City of Phoenix: Relating Performance Metrics to IT Objectives IT Department Objectives Net Results Metrics (Examples) Type of IT Objectives for Meeting City Measure Linked To Strategies Measures 1 Build partnerships with Successful delivery of ITD products Time 2, 3, 4, 5, 6, 11 city departments. or services on time v. goal. 2 Provide enabling Reduced delivery time v. goal. Time 3, 4, 5, 6, 11 technology to city departments. 3 Increase customer Problem responsiveness v. goal. Time 1, 3, 4, 5, 6, 8, 12 satisfaction. 4 Reduce cycle times in Reduction attributed to ITD v. goal. Cost 3, 9, 10, 11, 12 delivery of ITD products and services. 5 Reduce service delivery Customer satisfaction and Customer all costs. relationships v. goal. satisfaction 6 Improve service Reliability of products and services Customer 3, 6, 8, 12 delivery processes. v. goal. satisfaction 7 Implement citywide Systems using citywide Customer 1, 3, 4, 7, 8, 9, 10 technology architecture/new emerging satisfaction architecture. technologies v. goal. 8 Improve citizen access ITD staff trained in customer Customer 1, 2, 3, 9, 10, 11 to information and service skills and new technology v. satisfaction services through goal. technology. 9 Increase resource New ideas received and adopted Customer 1, 3, 6, 9, 10, 11 versatility. from ITS staff v. goal. satisfaction 10 Improve leadership Employee satisfaction v. goal. Customer 3, 9, 10, 11 skills. satisfaction 11 Increase employee Effective communications v. goal. Customer 1 ,2 ,3, 4, 11, 12 confidence. satisfaction 12 Achieve world class ITD involvement in departmental Customer 1 ,2, 3, 4, 7, 11, 12 results technology planning v. goal. satisfaction GAO/AIMD-97-163 IT Performance Measurement Guide Page 23 Case Study 2: Connecting Goals, Strategies, Objectives and Measures To better meet customer needs and respond to a changing market environment, in the mid 1990's Xerox developed a Xerox 2000 business strategy. In response to the new strategy, the information management organization--Global Process and Information Management (GP&IM)--developed four strategies and three breakthrough goals to align IM programs to support achievement of the outcomes anticipated by the corporate level business drivers. Senior executives ultimately decided on eight measures to determine whether IM programs were supporting corporate-level strategies. GP&IM concluded that five factors were significantly influencing how it developed a new IT strategy for Xerox. These were (1) the push to reengineer Xerox business processes, (2) knowledge that the current information management environment was not the foundation for the future of Xerox, (3) information systems investment decisions were currently driven by "entitlements," not a strong selection strategy, (4) renewal of the existing infrastructure was too costly, and (5) the cycle time for information management was not meeting business requirements. GP&IM then developed four strategies to respond to these five IT strategy "drivers": (1) reduce and redirect information spending through global outsourcing, reducing legacy system spending, and consolidating and sharing resources, (2) infrastructure management including asset management, (3) leverage worldwide information resources of people, hardware, and software, and (4) develop business process-driven solutions. As shown in the figure on the following page, GP&IM has several outcomes identified for each of the four strategies. GP&IM also developed three proposed high-level "breakthrough" goals covering the four strategies, shown on page 24. The goals are defined as follows: Manage IM spending suggests that IM2000 must be to managed to a flat ceiling for base spending while doubling the amount of investment in new development. Deliver infrastructure renewal determines the penetration of the new infrastructure in conjunction with some renewal of the old infrastructure (known as Globalview). Deliver business process solutions covers reengineering process deployment and drives operational optimization and the development of a baseline for day-to-day work. The baseline is for comparison with Electronic Data Systems (EDS) outsourcing arrangements. Each breakthrough goal area has measure examples. Xerox top management decided that the eight measures cited in figure 8 provided the right information on whether their IT strategies were being achieved. For example, the measures told them how well they were performing in actually redirecting IT spending into new applications and systems development, retiring the old infrastructure and replacing it with more companywide solutions, reducing operations and maintenance costs, and whether new process-driven IT solutions were providing timely, high-quality products near estimated costs. Page 24 GAO/AIMD-97-163 IT Performance Measurement Guide Case Study 2 Xerox IM 2000: Connecting Strategies With Outcomes Strategies Expected Outcomes GAO Orderly retirement of older infrastructure and applications Reduce/Redirect IM Increased IT investment into new applications and infrastructure Spending development and deployment Smaller Xerox IM staff focused on high value IT projects Dollar savings from reduction in acquisition costs Better workstations and productivity tools Infrastructure Movement from central to distributed processing Management Increased interoperability Increased use of company-wide software/applications Better alignment of IT spending with business priorities/needs Leverage Worldwide Increased IM productivity IM Resources Improved IT resource allocation via a new solutions delivery process IT resources focused on business process needs Business Process Increased speed in delivering new solutions Driven Solutions Tighter IT spending focus Old applications replaced and legacy spending eliminated GAO/AIMD-97-163 IT Performance Measurement Guide Page 25 Case Study 2: Xerox IM2000: Breakthrough Goals and Related Measures IM2000 Strategies Breakthrough Goals Measures Used IM spending as % of Reduce/Redirect IM Manage IM revenues reduced Spending Spending % [investment fund/total IM spending] Infrastructure % [new infrastructure Management installed/total eligible Deliver sites] Infrastructure % old infrastructure removed Renewal Leverage Worldwide % [approved reengineered processes deployed/total IM Resources business processes] % compliance to applications retirement plan Deliver Process % reuse code for in-house Business Process Driven Solutions deployed applications Driven Solutions % compliance to approved solution deployment Page 26 GAO/AIMD-97-163 IT Performance Measurement Guide 2. Prepare a Chain of Events and Evidence A common measurement problem is determining what impact support functions such as IT have on operational customer or enterprise outcomes in comparison to other factors. Since final enterprise or program outcome measures are difficult to measure for a support function like IT, it is useful to think in terms of a "chain of events and evidence." If given careful consideration, a chain of events and evidence can help IT managers understand how IT resources are being used to support mission or specific program-related outcomes. In some cases, IT is not directly related to final program results. However, there are intermediate outcomes that must occur before the program or business outcome can be achieved. In contemporary organizations, IT can play a pivotal role in achieving these intermediate outcomes. A chain of events is a theory of how inputs in an action chain move to outputs and finally to end outcomes. A corresponding chain of evidence shows the evidence and measures that match to the chain of events. The chains of events and evidence recognize that agency programs set in motion a sequence of events which are expected to achieve desired goals. The sequence of events bridges inputs to outputs to outcomes. The hypothetical example presented in figure 5 illustrates a simple chain of events, evidence, and measures for an environmental water quality program. At the beginning point of the program results chain, a series of simple events are mapped out to indicate steps essential to achieving the end outcome of a desired level of environmental quality. Certain types of evidence would then be used as indicators of whether the steps are occurring. IT provides essential support for several of the steps in the results chain which can indirectly affect the achievement of the program outcome. Measuring how well IT is supporting these intermediate steps is necessary to demonstrate how well it is performing from a mission perspective. GAO/AIMD-97-163 IT Performance Measurement Guide Page 27 Figure 5: A Hypothetical Chain of Events/Evidence for Achieving Environmental Quality Chain of Events Companies Target industry Industry participate in water Pollutants Quality of the group reached to discharge quality reduced; environment reduce discharges practices changed environmental water quality achieved to meet new monitoring program increased standards Chain of Evidence Number of Number of Percent End outcome: firms firms pollutants in improved participating conforming water quality health/ to standards welfare IT Support Chain INTERMEDIATE OUTCOMES END OUTCOMES Providing computer More efficient Timely, program software Effective information and effective support for mission accurate help desk applications program assistance meeting program accomplishment operations needs IT Measures Improvement in Percent PROGRAM PROGRAM timeliness of customers OUTCOME PERFORMANCE assistance due reporting MEASURE satisfactory use MEASURE to systems support of applications In an iterative process, organizations examine the chain of events and evidence for enterprise and operational customer goals and objectives. The chains explain how an enterprise or operational customer produces its results and just what those results are. The assumption in using the chains of events and evidence is that there is a clear understanding of just how a program, for example, is supposed to work in producing the expected results. In turn, a support function such as IT needs to define its own chains of events and evidence, demonstrating how its products and services affect enterprise and operational customer chains. For IT, the operational customer is interested in having IT applications that help achieve efficient and effective program operations and service delivery. In essence, final IT outcomes are often efficient and effective program operations-- actually an operational customer's goal. The organization in Page 28 GAO/AIMD-97-163 IT Performance Measurement Guide large part gauges IT failure or success by how well IT supports the chain of events the operational customer has in place to achieve its desired effect. Building chains of events and evidence in partnership with enterprise and organizational customers can be a difficult process, but it helps IT managers understand exactly how IT products and services support customers and how IT performance should be measured. Using a chain of events approach also enhances customer understanding of just how IT can contribute to the eventual enterprise or program outcome. 3. Use a Diversity of Measures to Evaluate IT Performance An effective IT performance management system should have a diversity of measures, matched to the right organizational need and level of decision-making, and action taken on the measure. These measures can capture performance at the individual, organizational, program, and process levels. Generically, measures--both qualitative and quantitative-- are often categorized into four main types: Input Measures are assessments of the resources used to carry out a program or activity over a given time period with the purpose of achieving an outcome or output. Input measures can include number of IT managers and employees, labor hours, IT funds, computer and telecommunications equipment or facilities, or supplies. Output Measures are assessments of the actual level of work accomplished or services provided over a given time period. They are often used to control resources. Number of reports issued, number of projects completed, number of answers on hot line services, and function points delivered are examples of output measures. These, too, are often process measures. Outcome Measures assess the actual results, effects, or impacts of a program or support function compared to its intended purpose. Outcomes can be difficult to measure because results may not be immediately evident or several organizational units and external suppliers or customers are involved and it is difficult to assign relative contributions to them. Outcome measures may be the level of customer satisfaction with IT services or cycle time reduction attributable to automated work processes. Combinations of Single Measures combine single output, outcome, and/or input measures into measures designed to demonstrate improvements in efficiency or effectiveness. An efficiency measure is output over input, such as number of PC applications installed per number of IT staff. An effectiveness measure may compare actual GAO/AIMD-97-163 IT Performance Measurement Guide Page 29 results to estimated or expected results or compare existing levels of performance (output) to accepted industry standards or target performance goals. A wide range of measures provide a balance for different decisionmaker needs. Input and output measures assess workload for an enterprise or specific program and how much in demand are its products and services. Combination measures assess efficiency and effectiveness. Outcome measures assess results compared to expectations. The key point is that the right measure is used at the right time and for the right reason. Input and output measures are absolutely vital for measuring how well a process to deliver IT products and services is performing. Use Contextual Information to Augment Performance Measures The organizations we studied track contextual or explanatory information to use with their IT performance measures. The contextual or explanatory information describes the broader environment that surrounds IT activities which can influence IT inputs, outputs, and outcomes. For example, measuring changes in mainframe to client server use, policy changes that impact performance, or changes in IT organizational structure could be important explanatory information. Capturing contextual and explanatory information can help managers understand the measures, assess performance, and evaluate the significance of underlying factors that may affect reported performance. Managers can also capture actions that have been taken or are being taken to in response to reported information, particularly for unexpectedly high or low performance. Often, IT performance data displays show the contextual or explanatory information either as a footnote to the performance data or in an explanatory page attached to the performance information. Page 30 GAO/AIMD-97-163 IT Performance Measurement Guide How to Get Started To proceed with the development and use of an IT results chain approach, organizations should: • clarify--with top management participation and stakeholder involvement--major organizational goals and objectives; • establish a simple chain of events and evidence for key mission areas of the organization; • create supporting IT outcome and process measures for each organizational mission goal area; • use one of the organizational goals and objectives to develop IT goals, specific objectives, and related performance measures for that IT goal; • examine existing IT measures, categorize them as input, output, or outcome measures, and decide on the combination of measures that best provides performance results; and • test the performance measurement system and make revisions based on initial lessons learned. GAO/AIMD-97-163 IT Performance Measurement Guide Page 31 Practice Area 2: ALIGN Practice Area 1 Follow A "Balanced Use an IT results chain Scorecard" Approach IMPLEMENT CONSTRUCT Practice Area 4 Practice Area 3 Build data Practice Area 2 Target measures collection/analysis Follow a at capability balanced decision-making scorecard levels Practice Area 5 approach Improve IT processes REINFORCE Practice Area Characteristics 1. Develop IT goals, objectives, and measures in operational and strategic areas 2. Focus on the most important "vital few" objectives and measures in four IT goal areas: • Achieving the strategic needs of the enterprise • Satisfying the needs of individual customers • Fulfilling IT internal business performance • Accomplishing IT innovation and learning Practice Area Overview A second best practice is to use a balanced scorecard approach to IT performance measurement. The approach attempts to create a measurement balance across the overall performance management framework. A balanced approach to measuring the contribution of IT to mission outcomes and performance improvement recognizes the broad impact of IT's supporting role. By measuring IT performance across four goal areas that are critical to overall IT success, the scorecard forces managers to consider measurement within the context of the whole organization. This limits the possibility of overemphasizing one area of measurement at the expense of others. In addition, measuring IT performance from different perspectives helps strengthen the analysis of intangible and tangible benefits attributable to technology. GAO/AIMD-97-163 IT Performance Measurement Guide Page 31 In the four IT goal areas discussed in this section, we present three or four key objectives that were common among the organizations we examined. Corresponding to each objective we provide some sample measures that come from our case study research and from supporting literature. Our purpose is to illustrate possible types of measures, not to prescribe a definite list of measures that all organizations should be using. Some of the measures are very basic, but they are clearly related to the objectives. Also, many of the measures are percentages or ratios. This is important because successful organizations begin with good baseline data on performance and, therefore, can accurately measure progress against the baseline as they move forward. In several of the organizations we studied, management is developing measures across key areas covering both long- and short-term strategies and activities. This approach is best captured in Robert Kaplan and David Norton's balanced scorecard approach, which many of the organizations either used directly or incorporated into the development their own approaches.11 The Kaplan and Norton scorecard evaluates performance in four areas: financial (how does the organization look to shareholders?), customer (how do customers see performance?), internal business (at what must the organization excel?), and innovative and learning (can the organization continue to improve and create value?). In order to summarize the IT performance methods being used by the organizations we studied, we have adopted a balanced scorecard approach similar to the Kaplan and Norton framework. However, a balanced scorecard is just one approach available for agencies to adopt in conducting IT performance management and measurement. Other approaches such as the value management framework, critical success factor analysis, and information economics also offer useful IT performance measurement methodologies.12 Like other methodologies, a balanced scorecard approach translates organizational strategy into specific measurable objectives, operating from several key concepts: • no single measure provides clear performance targets or places attention on critical mission areas, • goal, objective, and measure areas should give a comprehensive view of all levels of activities, from the project level to the strategic level, • limiting the number of measures used minimizes information overload, and 11 Kaplan and Norton published a series of articles in the Harvard Business Review which explain the concept and its application. The articles are listed in the bibliography. 12 A complete description of IT performance measurement approaches can be found in the Society for Information Management's Practitioner's Guide to I.S. Performance Measurement, referenced in appendix I. Page 32 GAO/AIMD-97-163 IT Performance Measurement Guide GAO/AIMD-97-163 IT Performance Measurement Guide Page 33 • a scorecard guards against optimizing one goal area at the expense of others. Practice Area Characteristics 1. Develop IT Goals, Objectives, and Measures in Operational and Strategic Areas For IT, measures cover a great diversity of value-added activities, including those for projects, a portfolio of applications, and infrastructure development. Organizations should know about success in all of them. As shown in figure 6, an IT results chain can be translated into a scorecard framework that looks at goals, objectives, measures (tiered for various decision-making levels), and accountability in key goal areas. The key starting point in developing a balanced scorecard is the question of purpose from the IT results chain--"What is the current and future purpose of IT? Then to meet that purpose, the IT organization must answer the goal question--"If we succeed, how will we differ?" in terms of specific goals. Figure 6: An IT Balanced Scorecard Approach Enterprise and Operational Customer Goals and Objectives IT Purpose In meeting the In meeting the In internal IT In the ability to strategic needs needs of business innovate and of the individual performance? learn? enterprise? customers? Enterprise level Program level Project The Balanced Scorecard level Page 34 GAO/AIMD-97-163 IT Performance Measurement Guide The IT goals and objectives of the organizations we studied most often focused on the following: • customer commitments and satisfaction, • cycle and delivery time, • quality, • cost, • financial management, • IT infrastructure availability, • internal IT operations, • IT skill availability, and • customer business process support. For example, Motorola's Semiconductor Products Sector (SPS) focuses its goals and objectives on four areas: (1) delivering reliable products (quality environment, best-in-class staff, worldwide resource optimization, communication), (2) providing integrated IT solutions (integrated data and systems architecture, technology roadmap and migration planning, distributed computing effort), (3) building client partnerships (client involvement, lead the information technology community), and (4) achieving competitive advantage (prioritize the projects that benefit SPS, deploy resources for maximum impact, speed of execution). We developed four balanced scorecard goal areas, objectives, and measures for IT that were among the most common across the organizations we studied. As such, the four goal areas illustrate an approximate consolidation of the performance management efforts of the organizations involved in our research. The four balanced scorecard goal areas are designed to measure how well IT is • achieving the strategic needs of the enterprise as a whole, in contrast to specific individual customers within the enterprise, • satisfying the needs of individual customers with IT products and services, • fulfilling internal IT business performance that delivers IT products and services for individual customers and the enterprise, and • accomplishing ongoing IT innovation and learning as IT grows and develops its skills and IT applications. The first two goals address whether IT is providing the right products and services for the enterprise and individual customers. The latter two goal areas address how well IT is performing in its own capability to deliver those products and services. The strategic and customer perspectives are key for linking to mission planning requirements in the Results Act, the Chief Financial Officers Act, the Paperwork Reduction Act, and the Clinger- Cohen Act. GAO/AIMD-97-163 IT Performance Measurement Guide Page 35 Managers in our case study organizations emphasized that "balance" does not necessarily mean "equality." Use of a balanced approach only means the consideration of several goal areas and the development of objectives and measures in each. For example, Kodak managers liked the balanced scorecard approach because it was a multivariable approach. Before using it, the IT organization was very cost conscious and tended to judge investments in new applications or skills largely from a cost perspective. The balanced scorecard examines short-term cost goals and potential business value in the context of various other nonfinancial operating parameters. 2. Focus on the "Vital Few" Objectives and Measures Each leading organization customizes a set of measures appropriate for its organizational goals and, for IT, how IT fits into the enterprise's strategic direction and mission delivery plans. The organizations concentrate their IT performance management efforts on a vital few objectives and measures within the goal areas. The organizations did not severely limit the number of measures developed at the beginning. But, over time, and with experience, the organizations became more focused in the measures they used. However, use of a balanced scorecard approach gets rid of "safety net" measures which organizations often collect but do not use for decision- making, resource allocation, or oversight reporting purposes. As is explained in the sections that follow, the measure examples illustrate the need for diversity. Within some of our case study organizations, similar measures are being used, but the measures remain under development, requiring more refinement and documentation. The measures presented here do not represent the full universe of what an organization might use. Also, in practice, the goal and objective areas may be more specific than those presented on the following pages. For example, one of our goal areas centers on the customer perspective. One objective of this goal area is customer satisfaction. In practice, an actual customer objective statement might be stated as "This fiscal year, at least 98 percent of customers will be satisfied with IT products, services, and processes." In short, the following sections discuss a general categorization of IT goals, objectives, and sample measures. Balanced Scorecard Goal Area 1: Achieving the Strategic Needs of the Enterprise IT strategic measures are designed to evaluate the aggregate impact of IT investments on the organization. In short, these measures provide insights into impacts made by the organization's entire portfolio of IT investments. Page 36 GAO/AIMD-97-163 IT Performance Measurement Guide This goal area focuses on ways to measure how IT supports the accomplishment of organizational strategies. The strategic perspective recognizes that in successful organizations, all components, including IT, must align with enterprise goals and directions. When evaluating the impact of IT in terms of strategic needs, the following questions should be considered: • How well integrated are our IT strategies with business needs? • How well is the overall portfolio of IT investments being managed? • Is IT spending in line with expectations? • Are we consistently producing cost-effective results? • Are we maximizing the business value and cost effectiveness of IT? IT managers and staff often attempt to satisfy individual operational customers without a check against enterprise interests. Having this goal area prevents targeting IT efforts for individual operational customers which may be very counter-productive to enterprise IT needs and expectations. Doing so is difficult, as one manager said, "It has been a cultural plan to look at [the company] as a whole versus maximizing for individual business partners. The reward and incentive systems are set up to emphasize that all senior managers must succeed together." The four IT strategic enterprise objectives presented in figure 7 reflect several key objective areas of the organizations we studied. These objectives cover enterprise strategic planning and goal accomplishment, enterprise management of the portfolio of IT applications, IT financial and investment performance, and use of IT resources across the enterprise. The first objective in this goal area addresses how well IT plans and efforts reflect enterprise mission goals. This objective area assumes the enterprise has defined its mission goals and can make the clear link to how IT supports those goals. The sample measures capture the contribution of IT solutions and services, compare what was planned for IT benefits and IT strategies against what actually happened, and compare IT strategies and planning and enterprise strategies and planning. The overall measurement thrust is to make sure that enterprise mission goals direct IT activities. The second objective, portfolio analysis and management, is a growing concern among the organizations we studied. Leading organizations want to make sure they have the right portfolio of IT applications either planned or in place that will enhance business or mission performance. GAO/AIMD-97-163 IT Performance Measurement Guide Page 37 Figure 7: Balanced Scorecard-IT Strategic Measures Objectives Sample Measures Enterprise percent mission improvements (cost, time, quality, quantity) attributable to IT solutions and services mission goals percent planned IT benefits projected v. realized Portfolio analysis percent IT portfolio reviewed and disposed percent old applications retired and management percent applications retirement plan achieved percent reusability of core application modules percent new IT investment v. total IT spending Financial and percent and cost of services provided in-house v. industry standard investment performance IT budget as percent of operational budget and compared to industry average net present value, internal rate of return, return on investment, return on net assets IT resource usage percentconsolidated/shared resources across units percentcross-unit shared databases and applications percenthardware/software with interoperability capabilities Kodak defines an application portfolio as a comprehensive inventory of computer applications that were developed or purchased to manage an organization's processes and information. The inventory contains detailed data relative to each application's size and characteristics, effectiveness in meeting business needs, potential for growth, and development and maintenance costs. The application portfolio forms the foundation for an overall IT investment strategy. Xerox has defined its IT inventory in a similar manner and made IT portfolio management a key objective area as part of its overall strategic enterprise strategies. As described in an earlier case study describing its IM2000 strategy, Xerox wanted to improve information management spending, deliver IT infrastructure renewal, and deliver process-driven IT solutions to customers. A key part of the overall IT strategy was to evaluate existing IT applications and determine how they supported, if at all, the IM2000 strategy. Xerox ran each of its existing applications through a rigorous analysis process, categorizing each into one of nine "disposition" categories ranging from stopping those of low usage and value to keeping others as corporatewide applications. Page 38 GAO/AIMD-97-163 IT Performance Measurement Guide For Xerox, the IT portfolio strategy helps accomplish several performance goals. The strategy reduces unnecessary operational costs and increases support productivity; identifies and consolidates similar applications and eliminates low value applications; identifies and retires legacy applications, data, and infrastructure as new solutions are deployed; and identifies consolidation and sharing opportunities. The principle is to view application systems as corporate assets.13 The third objective in this goal area examines financial and investment performance. While the two objectives above cover mission goals and portfolio management, this objective addresses management of IT costs and returns. The sample measures capture costs in major IT financial categories such as hardware and software. They also provide information on the balance of spending between legacy and new development applications and between in-house and outsourced operations. Another sample measure compares the IT budget to the enterprise operational budget, and how that compares to industry standards. Sample measures also look at the return on the IT investments, offering several different methodologies such as rate of return and net present value. Much of this information is traditionally benchmarked with other IT organizations of similar size and IT penetration. These measures are tied to customer and enterprise strategic perspectives to assess if scarce resources are being invested wisely. This is an especially important area in the federal government with the emphasis on cost reduction and the best possible use of existing resources. Lastly, IT resource usage as an objective targets how well the organization can leverage and share its IT resources across the enterprise. The measures evaluate factors such as what resources can be shared, what has been consolidated, and employee access to computing services. From a strategic perspective, this objective recognizes the need for shared, enterprisewide applications and the use of an IT infrastructure and architecture for the entire organization. 13 The Office of Management and Budget and GAO issued a guide, Evaluating Information Technology Investments: A Practical Guide, in November 1995 to assist federal agency and oversight staff in evaluating a portfolio of information technology investments in a similar manner. The approach is also an underlying feature of GAO's guide Assessing Risks and Returns: A Guide for Evaluating Federal Agencies' IT Investment Decision-making (GAO/AIMD-10.1.13, February 1997). GAO/AIMD-97-163 IT Performance Measurement Guide Page 39 Balanced Scorecard Goal Area 2: Satisfying the Needs of Individual Customers IT customer measures are designed to measure the quality and cost effectiveness of IT products and services. When evaluating the impact of IT on customer satisfaction, the following questions should be considered: • How well are business unit and IT staff integrated into IT systems development and acquisition projects? • Are customers satisfied with the IT products and services being delivered? • Are IT resources being used to support major process improvement efforts requiring information management strategies? If so, are the IT projects delivering the expected share of process improvement? The purpose of the second goal area is to meet the needs of individual operational customers. The three objectives shown in figure 8 capture the key objective areas we found in our research. Figure 8: Balanced Scorecard-IT Customer Measures Objectives Sample Measures Customer percent projects using integrated project teams partnership and percent joint IT customer/supplier service level agreements involvement Customer percent customers satisfied with IT product delivery satisfaction percent customers satisfied with IT problem resolution percent customers satisfied with IT maintenance and support percent customers satisfied with IT training percent products launched on time percent service-level agreements met Business process percent IT solutions supporting process improvement projects support percent users covered by training to use new IT solutions percent new users able to use applications unaided after initial training Page 40 GAO/AIMD-97-163 IT Performance Measurement Guide Two of the objective areas, customer satisfaction and business process support, address direct IT support. Customers were especially interested in time, cost, quality, overall customer satisfaction, and business process support. One official we talked to said, "[Our IT organization] looks at the business process characteristics of our customers. IT personnel ask are there better ways to support product innovation and development? How does IT support that? The question is the effectiveness of IT in supporting business processes--not cranking out function points."14 The first objective area, customer partnership and involvement, stresses a mutual partnership between the IT organization and customers in developing the best possible IT products and services. The sample measures examine a variety of areas, ranging from cooperation and joint development to involvement in project management. Customer satisfaction measures assess how well customers are satisfied with many IT activities. Sample measures also cover the accomplishment of system design requirements, complaints, problem resolution, error and defect rates, timeliness, and service-level agreement accomplishments. The business process support objective area emphasizes the importance of business process improvement as organizations streamline and reengineer. Business process improvement is a central objective area for many of the organizations we studied. The sample measures capture how well IT supports business process improvement plans and process analysis. They also examine the adaptability of IT solutions, training for new IT solutions and the effectiveness of the training, and costs in moving applications to new hardware. Balanced Scorecard Goal Area 3: Addressing IT Internal Business Performance Internal IT business measures are designed to evaluate the operational effectiveness and efficiency of the IT organization itself. The ability of the IT shop to deliver quality products and services could have a direct impact on decisions to outsource IT functions. When evaluating internal IT business functions, the following questions should be considered: • Are quality products delivered within general industry standards? 14 A function point measures an IT application in terms of the amount of functionality it provides users. Function points count the information components of an application, such as external inputs and outputs and external interfaces. GAO/AIMD-97-163 IT Performance Measurement Guide Page 41 • Are quality products being delivered using accepted methods and tools? • Is our infrastructure providing reliable support for business needs? • Is the enterprise architecture being maintained and sustained? One manager we interviewed said, "There are two dimensions of [IT] performance. One is the dominant or visible component--the use of IT in the context of [customer] business processes. The other is transparent--the functional excellence of IT." The first two goal areas stress the use of IT as it supports enterprise and operational customers. On a day-to-day basis, it is the functional excellence of IT internal business processes which delivers that support. Figure 9 shows four objective areas and sample measures we synthesized from our case study organizations and the general IT literature. Figure 9: Balanced Scorecard-IT Internal Business Measures Objectives Sample Measures Applications number of function points delivered per labor hour number of defects per 100 function points at user acceptance development and number of critical defects per 100 function points in production maintenance percent decrease in application software failures, problems mean time to resolve critical defects cycle time for development Project percent projects on time, on budget percent projects meeting functionality requirements performance percent projects using standard methodology for systems analysis and design Infrastructure percent computer availability percent communications availability availability percent applications availability on-line system availability Enterprise number of variations from standards detected by review and audit per year architecture percent increase in systems using architecture standards percent staff trained in relevant standards compliance IT managers and staff, along with enterprise senior management, decide which of the many IT processes truly must excel for meeting customer and enterprise goals in the short and long term. For example, is the IT process for identifying the right technology for customer applications the best it can be? IT managers and staff set specific goals for improvement of internal IT business processes. Page 42 GAO/AIMD-97-163 IT Performance Measurement Guide The first objective covers IT's performance in developing and maintaining applications. The sample measures include dollars expended per function point, average application development cycle time, and cost. The second objective area examines the performance in delivering projects, capturing traditional measurements on project time, budget, functionality, and use of widely accepted methods and tools. Measures also capture backlogs in both development and enhancement or maintenance of applications. The third objective area addresses IT infrastructure availability in a variety of areas, as well as response time and transactions. Many of the organizations we studied stressed the importance of infrastructure availability, an area totally transparent to the customer until something goes wrong. These measures keep managers on top of infrastructure performance where there is little tolerance for down time. The last objective area covers architectural standards. 15 The measures assess how well IT is meeting set standards, most often developed for interconnectivity and interoperability and efficient IT support. Many of the traditional IT measures fall into the internal business performance goal area, often focusing on the efficiency of computing and communications hardware and software. The measures in this goal area frequently are used for individual manager and staff IT accountability, as described in a later practice. Some of the organizations we studied were using the Software Engineering Institute's five-level capability maturity model to guide their IT process improvement efforts. The objective areas and measures are, in contrast to some of the other balanced scorecard areas, highly integrated. For example, project performance relies on effective applications development and maintenance. Balanced Scorecard Goal Area 4: Addressing Innovation and Learning Innovation and learning measures evaluate the IT organization's skill levels and capacity to consistently deliver quality results. This goal area recognizes that without the right people with the right skills using the right methodologies, IT 15 IT architectures explicitly define common standards and rules for both data technology, as well as mapping key processes and information flows. A complete IT architecture should consist of both logical and technical components. The logical architecture provides the high-level description of the organization's mission, functional requirements, information requirements, systems components, and information flows among the components. The technical architecture defines the specific IT standards and rules that will be used to implement the logical architecture. GAO/AIMD-97-163 IT Performance Measurement Guide Page 43 performance will surely suffer. Measures in this goal area should be used to answer the following questions: • Do we have the right skills and qualified staff to ensure quality results? • Are we tracking the development of new technology important to our business/mission needs? • Are we using recognized approaches and methods for building and managing IT projects? • Are we providing our staff the proper tools, training, and incentives to perform their tasks? The four objective areas shown in figure 10 include workforce competency and development, advanced technology use, methodology currency, and employee satisfaction and retention. Figure 10: Balanced Scorecard-IT Innovation and Learning Measures Objectives Sample Measures Workforce percent staff trained in use of new technologies and techniques percent staff professionally certified competency and percent IT management staff trained in management skills development percent IT budget devoted to training and staff development Advanced percent employees skilled in advanced technology applications technology use number of dollars available to support advanced technology skill development Methodology currency of application development methods in use percent employees skilled in advanced application development currency methods percent projects developed using recognized methods and tools Employee percent employee satisfaction with the capability of the existing technical and operating environment to support mission satisfaction and retention percent employee turnover by function This goal area develops the continuous improvement aspect of IT activities. It speaks to capabilities of bringing new technologies to bear on customer problems, practicing the best methodologies, and retaining and developing the best employees. The first objective area stresses the importance of having a capable Page 44 GAO/AIMD-97-163 IT Performance Measurement Guide and competent workforce. In particular, the organizations we studied were very concerned with workforce competence and development. Key measures included training hours and skill development. Most were transitioning from core competencies in operations and maintenance to business process improvement and reengineering, new business solutions, and technical direction of applications development done by others. The second and third objectives, advanced technology use and methodology currency, speak to the ability to recognize and deploy advanced technologies and methodologies in doing IT's work. The last objective, employee satisfaction and retention, measures how well employees themselves are satisfied with the quality of their work environment and general IT strategies and accomplishments. How to Get Started To begin developing a balanced scorecard approach for IT, organizations should: • get agreement among business and IT management on the approach that will be used for developing IT-related performance indicators and measures, • using the agreed upon approach, define and develop the key goal areas and objectives for the IT organization, • develop a full set of measures in one or two priority IT goal areas, then expand out to other goal areas, and • test the performance measurement system and make revisions based upon initial lessons learned. GAO/AIMD-97-163 IT Performance Measurement Guide Page 45 Practice Area 3: ALIGN Target Measures, Practice Area 1 Use an IT results Results, and IMPLEMENT chain CONSTRUCT Accountability at Practice Area 3 Target measures Practice Area 4 Build data collection/analysis Practice Area 2 Follow a Decision-making Tiers at decision-making levels capability Practice Area 5 balanced scorecard approach Improve IT processes REINFORCE Practice Area Characteristics: 1. Track IT measures and appropriate reports to each decision-making level 2. Align measures from the bottom to the top 3. Directly link tiered measures to the balanced scorecard 4. Align individual accountability to IT scorecard goals Practice Area Overview Organizations in our study targeted measures and performance reports at specific decision-making levels or tiers. These tiers cover the enterprise (agency) level, senior to mid-level management (program) level, and specific operations or project level. This approach offers several advantages, including (1) enhanced communication and understanding of performance measurement throughout the organization, (2) systematic links among measures and enterprise, program, project, and individual objectives, and (3) alignment of measures with mission results. IT performance information should drive management actions and decisions that support the attainment of organizational goals. Practice Area Characteristics 1. Track IT Measures and Reports to Decision-making Levels As shown in figure 11, performance measures and reports at each tier have specific purposes. At the enterprise tier, IT performance and measures focus on mission results, or how well IT is meeting its purpose in supporting enterprisewide goals and objectives. Information on final and intermediate outcomes of programs facilitated by IT projects and investments would be shown. A summary report may be prepared for GAO/AIMD-97-163 IT Performance Measurement Guide Page 45 external reporting to stakeholders and the general public. Reports may be prepared on an annual or quarterly basis and highlight IT policy- oriented information showing areas of progress, problems, and contextual or explanatory information to supplement the performance data. Figure 11: Performance Measurement Tiers At the IT program or support unit level, senior to mid-level managers Measurement focus and timing ENTERPRISE Cyclical (e.g., annual or quarterly) information that focuses on mission Executive results used for policy decisions and Information strategies inc Me ases Mission Results re asu fro m to t n fro rem m t PROGRAM Periodic (e.g., quarterly or monthly) op tom lig information that focuses on unit results ent op t bot ures a associated with management and Management gra o bo operational improvements Information nul tto eas Unit Results arit m IT M y PROJECT Immediate (e.g., daily or weekly) information that focuses on activity Activity/Task and task level data used to make Information tactical decisions and execute Workplace Results management decisions want to know how specific units or processes are performing. Measurement most often covers specific IT business processes such as applications development or lines of business (or core business areas). At this tier, more detailed performance information is used for management and the improvement of operations and integrating activities across IT processes or programs. In the third tier, or bottom level, the measurement emphasis is generally at the project level and individual systems performance. Highly detailed tactical and execution information supports immediate and day to day decision-making on funding, contract development and monitoring, Page 46 GAO/AIMD-97-163 IT Performance Measurement Guide project priorities, and possible adjustments in program operating procedures. Here the emphasis is on input and output measures. Tiering in this manner can help assign performance accountability and determine where IT measurement ownership lies. The organizations decide where pieces of IT performance accountability rest, such as user or operational departments, central IT organizations, and/or department or operational IT units. In decentralized organizations, many people are involved in the IT processes that deliver products and services. Case Study 3: Using Performance Measures for Different Management Tiers Xerox information management operational measures are targeted at three distinct organizational tiers. As shown in the following figure, Tier 1 consists of corporatewide metrics, targeted for senior managers and executives. Tier 2 focuses on measures used by specific organizations, divisions, and core business areas. These measures assess what is happening below the enterprise level and actually roll up Tier 3 metrics collected at the project level. Case Study 3: Xerox Tiered Information Management Matrix Tier 1: Global Information Management IT spending as % of revenues reduced (quarterly) Total IT spending as % of IT investment funding (quarterly) Customer satisfaction scores (annually) Consolidate and analyze Identify Tier 2: Organizations, Divisions, Core Areas trends % projects on time (actual versus planned) costs/benefits for IT investments (projects and overall) Communicate overall IT spending profiles (all IT investments) results Benchmark Tier 3: Project Level (contractor performance) $ per function point delivered % projects on time % software delivery re-use productivity (function point function point defects per month) GAO/AIMD-97-163 IT Performance Measurement Guide Page 47 Case Study 4: Decision-making Levels Use Different Types of Measures American Express' Technologies group is also developing a tiered measurement process for decision-making at all management levels. Tier 1 measures consist of executive information that represents the Technologies group's overall effectiveness against such goals as (1) achieving world class time-to-market, (2) developing new business, and (3) enabling business partners. Specific measures include development cycle time, time requirements for ongoing operations, cost of ongoing operations, cost of quality, leading quality indicators, and elimination of root causes of failures. Tier 2 measures consist of management information for senior and mid-level managers and have direct links to tier 1 measures. To illustrate, tier 2 measures supporting the tier 1 measure of development cycle time include "elapsed time per function point" and "effort hours per function point." Tier 3 measures are operational information used for project development, operations, and project leaders. Tier 3 information links directly to tier 2 and forms the basis for management decisions, root cause analysis, and continuous process improvement evaluations. Specific performance measures are used to evaluate individual projects and applications. 2. Align Measures From the Bottom to the Top of the Organization A key performance feature found in the organizations we studied is the notion of aligning--but not necessarily "rolling-up"--measures from the bottom to the top of the organization. IT measures used at the lowest tier--the specific operations or project level--must align upwards with the subsequent tiers. In other words, the IT input and output information collected for tactical and execution management must directly relate to unit results needs and then upwards to the enterprise level. This alignment helps to ensure that performance measures and information at the lowest level directly support policy and mission decisions and strategies. Only rarely will an organization have a single performance measure appropriate for all three levels, or, in other words, those that can "roll-up" to the top of the pyramid. A temptation in performance measures is to layer measures from lower levels on top of each other and pass the information along to higher-level officials. This approach may provide an overload of information not easily understandable or digestible by top executives or even the public. It also creates the potential of hiding embarrassing details in a mountain of data, or can promote the self- selection of favorable data. A few of the bottom tier measures may "roll Page 48 GAO/AIMD-97-163 IT Performance Measurement Guide up" into strategic measures of interest at the other two tiers. However, the type and formatting of IT information and measures and timing of performance reporting appropriate at one tier may not be appropriate for others. Use of all three tiers gives the organization a comprehensive picture of the value of IT and if the individual IT products, services, and processes were worth the investment. 3. Directly Link Tiered Measures to a Balanced Scorecard Once the IT organization has agreement on balanced scorecard goals and objectives, then it would develop limited tiered measures to address specific operational or project measures, program and support unit measures, and enterprise-level measures. The tiering of measures across the balanced scorecard area facilitates the use of measures for decision- making and performance improvement action. Otherwise, if measures are not used, they lose their decision-making impact, which results in less effort to collect and use them and, in turn, leads to less decision-making impact. One tier is not more important than the other two. As one manager noted, an operational customer will never ask for reports containing performance data found in the bottom tiers, such as mainframe availability or application availability. But this operational data provides vital information about how well IT is supporting program operations and is indirectly linked to customer satisfaction measures. The use of performance measurement tiers is not a novel concept, but reflects a change in how measurement information is used by an organization. Traditional IT performance measures are used to examine lines of code generated, number of reports issued, data center downtime, transactions, and the number of projects on time. These can be considered bottom tier measures. More recently, management emphasis has shifted towards performance-oriented customer requirements, productivity and quality improvements, selection of strategic projects, and weighing value of delivered systems. Using some specific objectives under the balanced scorecard goal areas discussed in Practice 2, figure 12 provides some hypothetical examples of IT measures that might be used at different organizational tiers. A combination of input, output, and outcome measures are sprinkled throughout the tiers to accommodate different management information decision-making and reporting needs. GAO/AIMD-97-163 IT Performance Measurement Guide Page 49 Figure 12: A Tiered Performance Scorecard Example Scorecard Tiered Measures Specific Enterprise Program/ Operational Goals Objectives or Project Support Unit Level Measures Measures Measures Strategic Portfolio % compliance to % IT portfolio % reusability of approved IT Enterprise analysis and core modules reviewed and solution disposed Perspective management deployment Customer # complaints to % customers % planned Customer help desk satisfied with IT system design Perspective satisfaction application requirements design realized vs. deliverables Internal Applications # defects per % decrease in application % projects on Business development 100 function time, on budget Perspective and points at user software failures maintenance acceptance Innovation & Workforce % staff % IT budget # staff by skill Learning competency & professionally devoted to area Perspective development certified training and staff development 4. Align Individual Accountability to IT Goals The last question that ties together the IT balanced scorecard and the IT results chain is "Who is accountable for results and how are they held accountable?" The leading organizations have learned that managing performance well depends on making the connection between program and IT purpose, goals, and objectives, and responsible teams and individual staff. Alignment begins when organizational and program goals and objectives are translated into action plans for the improvement of IT products, services, systems, processes, and practices. Just as measure development involves many staff, IT performance plans and targets must be communicated to all levels of the organization. Smaller components develop goals and objectives that support the higher level. In turn, individual accountability is based on the actions that each individual can take to contribute to the organization's goals. For example, the system of accountability must recognize that the final outcomes of IT support Page 50 GAO/AIMD-97-163 IT Performance Measurement Guide activities are usually intermediate outcomes for the higher level program goals and targets. Actual levels of performance should be tracked against targets, and both IT and program managers should be held accountable for the results. Measure and results reviews should report on the actual outcomes and individual performance appraisals should link IT performance to merit pay, bonuses, promotional opportunities, and other incentives. Case Study 5: Aligning Individual Performance with Organizational Goals Texas Instruments Information Systems and Services (IS&S) business excellence improvement process has many events involving measures and results. Operational assessments include quarterly metrics and project reviews. Information sharing events include a presentation of annual results and goals in January and quarterly communication meetings and results reports. The assessments focus on sharing lessons learned, identifying opportunities to transfer best practices, uncovering potential risks, and assigning corrective action. IS&S holds quarterly communication meetings open to all IS&S staff to discuss current topics and answer general questions. IS&S leaders use quarterly department meetings to further disseminate information and plans. Several electronic information sharing sources are available to IS&S staff. IS&S uses a performance evaluation and development process to allow IS&S personnel the opportunity to detail their previous year's accomplishments and identify short and long-term job and career goals. The employee uses the process to align his/her goals to those of the organization. The process is key to promotions, bonuses, and selection for a technical ladder--a designation recognizing technical staff for outstanding technical contributions and quality leadership. IS&S also uses recognition actions to support quality and performance improvement. For example, recognition display boards are located in highly visible areas. These display individual and team pictures of IS&S personnel honored for technical and quality contributions, as well as for involvement in team problem solving activities and community projects. Each year, IS&S leadership sends a memorandum to Texas Instruments' top management on policy deployment and key performance measures. Policy deployment is Texas Instruments' term for aligning the individual to overall corporate business objectives and for continuous or dramatic improvement of key performance measures. The memorandum's performance areas match the Texas Instruments strategic directions described in an earlier chapter--customer satisfaction, continuous improvement, people involvement, and cycle time improvement. Texas Instruments chose cycle time as a measure to drive dramatic improvement in a specific IT process (solutions provisioning). Improving this one process as a starting point addresses IT structural problems that affect cycle time. The IS&S leadership team has a business excellence improvement plan. This plan leads to action plans for each IS&S division supporting the strategic business units and other key organizational units, to team action plans to individual contributions. Although this approach is controversial in the pubic sector, leading private sector organizations have found that failure to translate organizational strategy into program, team, and individual IT goals can GAO/AIMD-97-163 IT Performance Measurement Guide Page 51 result in a focus on short-term and tactical issues rather than the accomplishment of strategic goals. How to Get Started To begin targeting measures, results, and accountability at different decision-making levels, organizations should: • identify enterprise tier IT performance information requirements, • use the enterprise tier information to develop measures across one balanced scorecard area, • begin adjusting management performance review processes at the unit, team, and individual level to reflect the tiered approach, and • test the performance measurement system and make revisions based on initial implementation efforts. Page 52 GAO/AIMD-97-163 IT Performance Measurement Guide Practice Area 4: ALIGN Practice Area 1 Build a Comprehensive Use an IT results chain Measurement, Data IMPLEMENT CONSTRUCT Practice Area 4 Collection, Practice Area 3 Target measures at Build data collection/analysis capability Practice Area 2 Follow a balanced and Analysis Capability decision-making levels Practice Area 5 Improve IT processes scorecard approach REINFORCE Practice Area Characteristics: 1. Use data collection tools 2. Develop and use baseline and benchmark information 3. Assess maturity and develop complete performance definitions 4. Utilize concise, understandable performance reporting 5. Conduct measure reviews and audits Practice Area Overview Building a balanced scorecard and tiered measures is only one step in designing an effective IT performance management system. In the organizations we studied, management paid careful attention to the "back end" of performance management: data collection and analysis. As one manager explained, "You need a lot of analysis to figure out the key drivers for performance, like what will it take to raise customer satisfaction one or more percentage points. And is that goal good for other measures such as cost and time? What is the gain for the effort we must put in and is it worth it?" Most organizations began by benchmarking existing performance against different IT units within the organization, external IT organizations in other businesses, or industry benchmarks published by research organizations. Next, performance definitions were agreed upon and existing data used to "baseline" existing performance and identify information gaps that needed to be addressed with additional data collection and analysis. Performance data are needed at all tier levels. Even so, the collection and reporting of this information should not impose an unnecessary GAO/AIMD-97-163 IT Performance Measurement Guide Page 53 burden on management and staff. Data collection should utilize efficient manual or automated methods. The organizations we studied developed a clear rationale for new and continued data collection and specifications for accuracy, reliability, timeliness, and use before setting out measurement reporting requirements. Most organizations we examined designed efficient and effective ways to present the performance information to management so that it could facilitate better decision- making. Finally, the organizations regularly conducted reviews of their performance measurement systems and revised or updated measures in accordance with management feedback or changing business needs. Practice Area Characteristics 1. Use Data Collection Tools For each data collection requirement--whether qualitative or quantitative-- most of our case study organizations developed manual and automated tools to reduce the burden of collecting IT performance information. These included personal observation, formal performance measure reports, customer satisfaction surveys and interview questions, reviews of records and documents, and automated hardware and software productivity data collection tools. Case Study 6: Collecting IT Performance Information Using Multiple Techniques Across Different Performance Dimensions The Immigration and Naturalization Service's Office of Information Resources Management (OIRM) collects information on both its Information Technology Partnership (ITP) contract, the primary IT support services contract, and for program-level performance regarding mission outcomes. As shown in the following figure, the ITP contract tasks measure within budget, on-time, customer satisfaction, and technical (such as quality of project completeness, use of resources, and completeness and quality of documentation) performance. The program-level performance measures cover cost, efficiencies, and quality in the improvements realized. The program level covers all major operational areas, such as the Border Patrol. INS has specifically designated tools for its data collection under the Information Technology Partnership mentioned earlier. These tools include a performance measures report, a customer satisfaction survey, and a technical performance evaluation form. For mission- critical tasks, quantitative methods, ratios, and performance reports are the tools that are used for data collection and analysis. Page 54 GAO/AIMD-97-163 IT Performance Measurement Guide Case Study 6: INS ITP Performance Measurement Plan All ITP Contract Tasks The Vision Within budget Performance measures report The INS seeks a contractor to perform as Performance Measures On-time a full partner with INS in conceptualizing, T Performance measures planning, developing, implementing, and Customer satisfaction report ITP Contract O maintaining advanced, cost-effective, state- Technical performance ITP customer satisfaction of-the-art technological solutions to the O survey business processes of the INS. The goal L of the partnership is optimal achievement ITP technical performance of the INS mission. In addition, the INS S evaluation form technology partner will provide visionary, ITP self assessment report leading-edge support while participating in the planning to meet the INS information technology needs of the 21st century. High Impact/Mission Critical Tasks Cost (improvements Quantitative methods Performance Measures Methodology realized) T **Return on Investment **Net Present Value Program-Level Identify customer needs Efficiencies (improvements O realized) Process improvement ratios O Document current operating baseline Service-level Agreements Quality (improvements L realized) Set goals for business process Performance measures S report improvements Develop service-level agreement ITP Monitor key indicators "Return on Investment" Most of the organizations we studied feature customer surveys, interviews, and focus groups as important sources of performance information. These data collection exercises are well-designed, tailored to IT performance measurement ownership, and fit into management processes to take action on the results. For example, Motorola uses a companywide survey that asks for ratings ranging from "superior" to "poor" on such issues as • availability of Information Systems (IS) personnel, • responsiveness of IS personnel, • effectiveness of IS in project management communication, • reliability of IS in meeting commitments, GAO/AIMD-97-163 IT Performance Measurement Guide Page 55 • IS cycle time for completing projects, • quality of IS work, • rate of improvement during the past year, • alignment of IS resources with business needs, • overall credibility of information systems support, and • IS performance with respect to total customer satisfaction. In all cases, the organizations started with what they considered a primitive data collection strategy, beginning with data definitions and working with customers and IT staff to gain understanding and agreement. In the early stages, some of the data collection may all be manual, later supplemented or replaced by automated systems. All of the organizations cautioned that IT should never wait for automated data collection systems, but start small and with measures meaningful to enterprise and operational customers, such as IT cycle time and customer satisfaction. Page 56 GAO/AIMD-97-163 IT Performance Measurement Guide Case Study 7: Using Multiple Data Collection Mechanisms and Ensuring System Integrity Texas Instruments' Information Systems & Services (IS&S) gathers, prioritizes, and communicates customer and supplier concerns through a series of teaming activities. IS&S conducts annual strategic intent conferences and objective reviews with top-level customer management. Several times a year senior customer managers on an Internal Information Systems Leadership Team meet with IS&S managers to offer strategic planning and guidance. Other activities include ongoing executive one-on-one interviews and management-level and project-level customer surveys, employee development, reengineering efforts, customer focus workshops, steering teams, quality improvement teams, leadership teams, and quality steering teams. Senior IS&S management uses these opportunities to confer with the customer on business priorities, critical success factors, and operational plans. A service-focused survey is sent to end users of IS&S products and services. Each Texas Instruments organization has a metrics coordinator who handles the manual and automated databases. Much of Texas Instruments' IS&S performance information has been tracked for at least 5 years, with the majority of the information available online. A combination of software error checking, password control, and independent audits ensures the reliability, accessibility, and integrity of the performance databases. IS&S promotes system consistency and maximum data sharing through standardized data formats and system interfaces. IS&S uses an electronic data interchange network to communicate electronically with suppliers and customers. An online complaint system assists customers in registering complaints about IS&S products and services. While there is widespread use of annual customer surveys, most organizations note that they are very limited in the information they provide that can tie to corrective action strategies. One manager said his organization was putting customer surveys on hold until the organization could develop questions which would provide answers for concrete corrective action. Another manager said he planned to do "just in time" surveys on specific projects or IT processes instead of a survey at the end of the year. His rationale was that year end surveys only capture problems or feelings, while he wanted to capture data that the IT organization could take action on, that is linked to organizational activities, and where he can see trends in particular areas and focus on skills and problem areas. His point was that asking general questions is no way to develop actionability or determine significant trends. Managers in the organizations we studies emphasized that it is important to have consistency of some hard data collection from year to year. Some performance information such as computer and communications availability, customer satisfaction percentages, and software capability GAO/AIMD-97-163 IT Performance Measurement Guide Page 57 maturity levels are relatively durable and comparable over longer time periods. These measures track trends in areas where products and services do not change significantly. One manager suggested that measures should not be policy-based as policy can change from year to year. 2. Develop and Use Baseline and Benchmark Information The organizations we studied spent considerable time and effort on baselining and benchmarking, two entirely different activities. They assessed what performance information they had for the measures they had selected (baselining) and how that information might compare to that of other organizations or similar processes within their organization if there were discrete IT units (benchmarking). In baselining, available IT performance information, or information that will have to be collected, becomes part of the performance baseline for each scorecard objective. The current performance becomes the "baseline" against which further performance is measured. Without baselining, there is no standard to measure progress. In fact, one of the initial tasks in IT performance management is determining current performance using the measures designed for the balanced scorecard or a similar approach. The organizations we studied found that performance data within a balanced scorecard cannot be a set of data on top of all the IT performance data that was previously collected. To set actual IT performance targets, organizations often do benchmarking, an activity that is much different from baselining. Benchmarking was done with other IT organizations in the enterprise, with IT organizations outside the enterprise, or with similar processes but in other industries. For example, Kodak benchmarks with companies in markets where it competes (competitive benchmarking), with leading or best of class organizations within any industry (functional benchmarking), and among the operating units within Kodak (internal benchmarking). For example, handling customer support phone lines could be benchmarked against mail order telephone operations. Another organization we studied, Xerox Corporation, was outsourcing many of its IT operations to Electronic Data Systems (EDS). Xerox is benchmarking EDS service content, service delivery, and pricing against the best in many countries. Initially, Xerox established a price and service baseline for comparison of EDS services and prices against the best organizations. Many of the organizations we studied have made benchmarking an integral part of their IT performance management activities. Benchmarking information is often required in decision-making packages Page 58 GAO/AIMD-97-163 IT Performance Measurement Guide sent forward to senior executives. Most have devoted at least one staff member to benchmarking. Texas Instruments, for example, has a corporate office of benchmarking and best practice sharing. Some organizations we studied cautioned that benchmarking requires focus. Benchmarking often requires looking at process improvements versus strategic value. While an organization will know how an organization does a particular IT process or set of activities, that knowledge may do little to improve outcomes. If the benchmarking focus is on key measures, such as reducing cycle time or how well IT business process support customers, then benchmarking realizes a strategic potential. The organizations use baselining and benchmarking information to identify performance gaps between current IT performance and desired achievement levels. Leading organizations recognize that improvement goals must flow from a fact-based analysis of IT performance aligned to organization mission. At least one of the organizations we studied believed that goals, such as six sigma, make benchmarking irrelevant.16 The standard for performance is already set at a "zero defect" goal. One manager noted that "the key to performance is how the [IT] organization supports the business, not how the IT organization compares to other IT organizations." However, others believed that benchmarking provides comparison data on exemplary organizations and sets "stretch" performance standards for the IT organization. 3. Assess Performance Maturity and Develop Complete Performance Definitions A common stumbling block for organizations is the tendency to struggle to develop "perfect" measures instead of thinking in terms of improving measures over time. But as one manager told us, in performance measurement, you simply cannot be good at everything right away which forces you to undertake a phased approach. As an organization moves gains more performance management experience, better and more appropriate goals will be defined so the supporting measures will, in turn, be modified. For many measures, the definitions, data collection techniques, and reporting will need to be refined over time. Measure maturity assessment can be a part of the measure definitions that organizations develop. Generally, these definitions cover what the measure is intended to show and why it is important, how performance data are generated, who is responsible for collecting it, how the measure 16 Six sigma is a measure defining the quality level of a product, service, or process. This organization defined six sigma is 99.9997 percent perfect or 3.4 defects per million opportunities to create a defect. If set as a goal, it basically calls for a virtual "zero defect" standard. GAO/AIMD-97-163 IT Performance Measurement Guide Page 59 is specifically calculated, any limitations placed on the measurement data (for example, factors beyond the organization's control), and whether the data is cumulative or noncumulative. As this process is repeated and refined over time, there is increased confidence that the measure accurately describes performance in the relevant area. Case Study 8: Gaining Experience with Fundamental Measures and then Expanding Out Kodak is one organization that is systematically defining the maturity of each measure it plans to use in its balanced scorecard. Kodak categorizes measure maturity as either fundamental, growing, or maturing. Established indicators are considered as fundamental. Growing measures are evolving from the fundamental, but are not the best they can be. Maturing measures are defined as best-in-class for whatever they are measuring. For example, for internal performance a fundamental measure is to meet all service-level agreements, a growing measure is information delivery excellence, and a maturing measure is defect-free products and services. Kodak believes it is important to build the right fundamental practices first in developing an initial IT performance management system. As part of its development of measures for EDS services, Xerox is starting with what it calls "primitive metrics." These include size, measured in function points; effort, measured in work hours; defects, measured by number of defects; changes, measured by number of changes; and duration, measured by elapsed days. Over time, quarterly performance reviews and examination of measures are expected to result in revisions to these measures. 4. Utilize Concise, Understandable Performance Reporting Leading organizations take great care in designing IT performance reports that are concise, easy to understand, and tailored to various management needs and audiences. Executive managers, in particular, often require data presentations and displays that focus on bottom line performance results. When presented in this manner, they can quickly digest information, focus on problem areas, seek pertinent follow-up data, and be more efficient in making or recommending project or program decisions. Performance reports should be tailored for the audience for which the information is intended. Operational managers may need more details and supporting, contextual information while external stakeholders may require far less. Page 60 GAO/AIMD-97-163 IT Performance Measurement Guide Case Study 9: Presenting Performance Data in Usable Formats The Immigration and Naturalization Service's Office of Information Resources Management (OIRM) collects information on both its Information Technology Partnership contract and on program-level performance regarding mission outcomes. The contract is run by INS in partnership with Electronic Data Systems (EDS). The following figure shows how overall contract-level performance is reported on a quarterly basis for each of the four key critical success factors: (1) customer satisfaction, (2) technical performance, (3)on time, and (4) within budget. These quarterly measures can then be plotted to show a performance trend over the life of the contract. Gathering data such as this will also allow INS to provide documented and objective feedback on a contractor’s past performance. Case Study 9: INS/ITP Quarterly Performance Reporting Percent 1st QTR. Score Weight Weighted Performance Score 120 Measures Customer 75 .30 22.5 100 89 Satisfaction 85 75 80 70 Within Budget 70 .20 14.0 60 On-Time 89 .20 17.8 40 Technical 85 .30 25.5 20 Performance 0 1st Qtr. Cust.Sat Budget ITP Performance Index 79.8 On-Time Tech. Perf. GAO/AIMD-97-163 IT Performance Measurement Guide Page 61 Case Study 10: Effective Presentation of Performance Data and Contextual Information The Minnesota Department of Administration provides business management and administrative services to Minnesota agencies. Most of the Department's operations are fee- based operations in areas such as data processing, printing, vehicle rental, and the sale of office supplies. The Department's InterTechnologies Group (InterTech) provides services in managing and operating information technology resources. Figure 21 illustrates one objective for customer service performance. The objective is written with a specific performance rating standard and the use of a specific survey for data collection. The performance data are reported in this format, with the objective, definition, rationale, and data source. The data also include a discussion of past performance and the plan to achieve targets. These data and reports on other measures are given to the Minnesota Legislature in a formal performance report. Case Study 10: Minnesota Department of Administration's Customer Service Performance Data Template OBJECTIVE 1: InterTech will receive above-average customer service rating (i.e. a Actual performance Fiscal Fiscal rating of 3.0 or better on a scale of 1.0-4.0) for Year 1 Year 2 key services, as rated by customers on an annual customer service survey. Telecom. Services Perform. Targets 3.0 3.3 DEFINITION, RATIONALE, DATA SOURCE: Actual. Perform. 3.1 Definition: Key services are defined as those for Data Processing which customers' ratings averaged 3.0 or greater, Services indicating important or very important. In the FY 1 Perform. Target 3.0 3.4 survey, 27 services were measured with Actual Perform. 3.2 responses from 259 customers. Customers rated 15 of 21 data processing services as important and 8 of 11 telecommunications services as important. The reported satisfaction ratings are DISCUSSION OF PAST PERFORMANCE: based on these services. Prior to FY 1, InterTech periodically surveyed Rationale: Above-average satisfaction is defined customer agencies to determine their level of as customer ratings averaging 3.0 or greater, satisfaction with key services. This effort was indicating satisfied or very satisfied. The future expanded; surveys are now conducted annually. year goal is to improve any services rated below 3.0 by at least 10% each year and to improve any services rated 3.0 or above by at least 5% each PLAN TO ACHIEVE TARGETS: year, until a perfect score of 4.0 is achieved. Targets in future years will be based on the Data sources: Annual customer satisfaction previous year's percent improvement. survey. Page 62 GAO/AIMD-97-163 IT Performance Measurement Guide 5. Conduct Measure Reviews and Audits Most organizations we studied regularly assess their measures to see if they are still appropriate for measuring results and assigning accountability. Most of these organizations do regular external audits of IT measure appropriateness and data collection efficiency. One manager said, "We believe any successful organization needs a good basic independent look at measures, challenging what are the right questions." The reviews and audits serve as an oversight of measures and the data, challenging the necessity of the measures (i.e., are they being used?) and their linkage to enterprise strategic plans and individual performance expectations (i.e., are they having an impact?). Reviewers consider the accuracy, completeness, timing, match to actual conditions, and technical features such as the method of results calculations. For example, as new technology is introduced, measures can become obsolete or less important. Or customer measures supplant or replace traditional measures. In one organization, IT used thousands of lines of code per calendar month as a measure of software productivity. From the customer point of view, the measure was ineffective, and the measure was changed to cycle time. Over time the organization will develop more reliable measures. The reviews also analyze key performance drivers in each goal area. They question what a stress in one area will do to the results in other areas. They also can question what performance gain might be achieved for a certain level of performance effort, and if the performance results are worth that effort. How to Get Started To build a comprehensive measure, data collection, and analysis capability, organizations should • designate specific IT staff to gain skills in measurement, data collection, and analysis, • review existing data collection and reporting and determine what is still appropriate and what should be changed or deleted to match possible measurement scorecard areas, and • determine what are preliminary interdependencies among scorecard goal areas, objectives, and measures. GAO/AIMD-97-163 IT Performance Measurement Guide Page 63 Page 64 GAO/AIMD-97-163 IT Performance Measurement Guide Practice Area 5: ALIGN Strengthen Practice Area 1 Use an IT Processes to IT results chain Improve Mission IMPLEMENT Practice Area 3 Practice Area 4 CONSTRUCT Build data Practice Area 2 Performance Target measures at decision-making collection/analysis capability Follow a balanced scorecard levels Practice Area 5 approach Improve IT processes REINFORCE Practice Area Characteristics: 1. Define the IT business processes that produce IT products and services meeting mission goals 2. Using IT performance information, prioritize IT business processes essential for improving mission performance. Practice Area Overview In many of the organizations we studied, business process improvement is a high priority enterprise strategy. Often, enterprise and operational customer business process improvement can only be accomplished using IT products and services. That means that IT must ensure that its own business processes are the best they can be. Simply put, IT internal business processes deliver IT products and services which the enterprise and operational customers depend on to support their missions. If IT does not have the capability to deliver high quality products and services, then organizational goals can suffer. This is an important reason why the balanced scorecard approach includes internal IT business processes. Practice Area Characteristics 1. Define IT Business Processes That Produce Products and Services Critical for Meeting Mission Goals All leading organizations define their key IT business processes. This helps the IT organization focus on primary activities, identify IT competencies, eliminate processes which do not add value, and facilitate IT process innovation. Several of the organizations we studied noted Page 64 GAO/AIMD-97-163 IT Performance Measurement Guide that process measures are tightly linked with tasks or activities. Should organizational structures change and tasks and activities move within the IT organization, measures can more easily move. In other words, IT processes and subprocesses, once defined, are "portable" from one part of the organizational structure to another. Some of the organizations developed their IT process orientation based on work done by the Ernst and Young Center for Information Technology and Strategy and later published by a workgroup of the Society for Information Management (SIM).17 SIM's Information Systems Process Architecture (ISPA) process framework is a model of how a typical organization obtains and applies information systems and technology. In describing its initial framework issued in September 1993, SIM indicated that its process model (1) defines strategically important IT processes in an overall framework, (2) communicates with IT stakeholders on the value, activities, and organization of IT, and (3) provides a basis for allocating IT resources that is congruent with activity-based costing. As shown in figure 13, SIM's ISPA version 2.0 process framework, issued in March 1996, includes eight IT processes which overlap. This framework, like the earlier version, provides an example of a way to organize IT, determine core competencies, and identify process owners. Figure 13: ISPA Version 2.0 Process Framework Perform customer relations Align Manage Develop & Deliver & Market business enterprise deploy support IT and architecture products/ products/ IT services services Plan IS organization Manage IS organization 17 See the Society for Information Management Working Group on ISPA process architecture documents listed in the bibliography. GAO/AIMD-97-163 IT Performance Measurement Guide Page 65 As explained below, each process in this suggested framework has a specific purpose and suggested metrics (measures). • Perform Customer Relations This process is used to develop and maintain working relationships with the customers of IT products and services. Performance metrics focus on the business impact of IT, customer satisfaction, and product and service measurements, such as the accuracy and response time for solving problems or number of viable IT projects identified. • Market IT This process is used to ensure that customers want, need, and buy the products and services offered. Sample metrics focus on the increased business value of IT, IT return on investment, customer satisfaction, and product and service effectiveness measures. • Align Business and IT This process is used to incorporate IT into strategic business change activities in a way that captures opportunities from current and emerging technologies, and to promote process innovation leadership using IT as the catalyst and using proven visioning and change management techniques. Sample metrics focus on the business value of IT, net present value of projects approved by the strategic business unit, and improvements to business processes. • Manage Enterprise Architecture This process is used to provide a framework for delivering consistent products and services. Sample metrics focus on IT architecture design and implementation, the number of viable IT projects identified, how service chargebacks align with customer views of services, and the degree of technology standardization. • Develop and Deploy Products and Services This process is used to acquire, develop, deliver, and implement new information services for the organization. Sample metrics might include the percent of projects on-time and with the desired functionality, projects completed within budget, customer satisfaction, and degree of technology usage in conducting core business processes. • Deliver and Support the Products and Services This process is used to ensure that the products and services are deployed in the most effective and efficient manner. Metrics can focus on customer satisfaction survey ratings, increased demand for IT information, adequacy of equipment and facilities, availability and accessibility of data, number of problems received and requests satisfied, and mean time to data recovery. Page 66 GAO/AIMD-97-163 IT Performance Measurement Guide GAO/AIMD-97-163 IT Performance Measurement Guide Page 67 • Plan the IS Organization This process is used to shape and support business unit strategies, establish strategy and vision for long-term IS use, develop tactical plans and development and infrastructure resources over a 12 to 18 month horizon, and design key processes within the IS organization. Sample metrics could include employee satisfaction with the IS vision and knowledge about technology plans. • Manage IS Organization Business This process is used to manage the processes within IS that deal with the health and state of the IS organization, its employees, and vendors. Sample metrics include an employee satisfaction and commitment index, team efficiency and effectiveness measures, product and service measurements, and an index of employee skills. Case Study 11: Texas Instruments IT Process Segmentation Effectively Supports Business Needs At Texas Instruments, many IT customer business process reengineering activities established a clear need for rapid provisioning of IT solutions, flexibility with respect to business rules and work flows, and cost effective use of advanced technology. In response, the Information Systems and Services Group (IS&S) developed its IT process map with five processes, as shown in the following figure. The process map defines activities that "map" into both the operational customer and IT organization sphere. For example, strategy management as a process is the joint responsibility of both the business customers and IS&S (the enterprise IT group). Strategy management sets the vision and timing of all the elements of IT needed to support Texas Instruments. Solution provisioning is the mechanism for assembling the hardware and software pieces together to form solutions for IT customers. The service and product support area deploys and maintains IT products and services. The research and architecture definition area does research, experiments with and sets standards and methodologies for current and future architectures. The component provisioning area provides reusable applications, building blocks, assembly tools, and integration expertise. Page 68 GAO/AIMD-97-163 IT Performance Measurement Guide Case Study 11: Texas Instruments' Information Technology Process Map Concept Provision Business IT Use Strategy IT Needs Management Solution Solutions Service and Provisioning Product Support Compon ent s Enterprise Principles, Architectures, and Methods Enterprise IT Research and IT Market Trends Architecture Component Definition Provisioning Enablers Capability Compo nents IT Commercial IT Marketplace 2. Using IT Performance Information, Prioritize IT Business Processes Essential for Improving Mission Performance Given the many IT business processes that an IT organization manages, which ones are the most important for improvement? In the organizations we studied, customer business process improvement strategies and performance requirements frequently identify and drive major improvements in IT business process. They key is to determine the most efficient and effective manner to organize roles, responsibilities, and delivery of IT products and services within the organization. This requires a clear understanding of what IT functions are the domain of business units, unit-organized IT groups, and the corporate or enterprisewide IT organization. Knowing how effective is this arrangement is in providing effective IT support, which IT functions are performing well, and where service improvements are needed is critical to targeting IT management attention. GAO/AIMD-97-163 IT Performance Measurement Guide Page 69 Case Study 12: Using Business Process Focus to Shape IT Service Delivery Kodak is deciding what IT processes it should pursue to support the enterprise and operational customers' business processes. The focus is to look at each business process and its use of IT and determine if the information systems organization identified the right technology for the business needs. For Kodak, enterprise goals drive a process focus for strategic business units and support functions such as IT. The company's enterprise goals are three fold: (1) a 10 time defect reduction every 3 years, (2) reaching 6 sigma by the year 2000, and (3) a 10 time cycle time reduction in 3 years. Kodak defines a defect as a variation in a product or service which, if not caught, prevents the company from meeting the needs of its customers. Cycle time is the total time to move a unit of work from the beginning to the end of a process. Kodak believes that maximizing business processes is the only way to achieve these dramatic improvement goals. How to Get Started To begin focusing resources on IT process improvement, organizations should • use a model such as that suggested by the Society for Information Management to define IT business processes, • determine which IT business processes are the most critical to meeting enterprise and operational customer performance goals and requirements, and • choose one or more critical IT business processes to baseline existing performance and benchmark against industry standards or leading public/private organizations. Page 70 GAO/AIMD-97-163 IT Performance Measurement Guide Key Lessons Learned for Effective Implementation ______________________________________________________________________________ Agency managers just starting to develop IT performance management systems or those who want to enhance existing ones have a formidable task. They often are faced with resource constraints, demands for immediate IT support and solutions as program areas reduce staff and reengineer their business processes, and skepticism about the value of performance management. From the organizations we studied and the experiences of other organizations described in the literature, three key activities are essential in putting the practices in place. These are assessing organizational readiness for a successful IT performance management system and staging the system development, following a simple measure selection process, and recognizing system maturity will change over time. Assess Organizational Readiness The leading organizations find that assessing organizational readiness for a successful IT performance management system is essential. Here, the organizations look for the involvement, commitment, and day-to-day support of enterprise senior managers. They also determine if they have adequate resources, including staff allocation, skills, time, tools, and use of consultants or technical assistance if needed. A manager characterized a good performance management system as one that "has complete buy-in from top management, involves front line employees in system design, and results in front line employees understanding what they do and why they are doing it." Organizational readiness also means making sure that existing planning and decision making structures can accept performance results so they can be used. As the introductory chapter to this guide explained, performance measures are a central piece of alignment around mission planning, budgeting, and evaluation. These are essentially separate processes linked by performance measures. The organization needs the capability to specify clear goals and objectives to set the focus and direction of IT performance, creating an IT performance improvement plan and revisiting it every one or two years. The IT organization has to understand the business of operational customers and make sure IT measures are consistent with business measures. That means the capability to develop a "theory" of how IT supports enterprise and Page 70 GAO/AIMD-97-163 IT Performance Measurement Guide operational customers so the organization can build the chain of IT events and evidence described in practice 1. The organizations also determine if they have the support of other stakeholders and funding sources, such as legislative staff. As mentioned under practice 1, stakeholders are one of the parties which have to reach a common understanding of IT goals, objectives, appropriate measures, and anticipated outcomes. Lastly, organizational readiness means paying attention to organizational culture -- is it receptive to data collection, measurement, and analysis and accountability for performance and decisions as part of an overall performance improvement system? The organization should have a philosophy that is positive towards performance management and measurement and views it as a way to focus on quality and operational customer satisfaction. That means the organization should be willing to assess organizational values, principles, and how they are working. That is the key to success, one manager explained, "We have the culture to support this -- people get to where they track and report metrics, they think about quality goals. [IT performance management] is a hard sell where there is not a total quality culture, where the focus is on technology, not satisfying the customer." Follow a Simple Measure Selection Process Selecting and implementing IT performance measures is extremely complex. Each enterprise and its operational customers have a mission and goals that often differ significantly from other agencies. A set of IT performance measures that work for one organization likely will not completely work for another organization. The performance measures differ on what is valued most in terms of IT performance. To illustrate, IT goals and objectives which stress cost savings would have more measures related to efficiency versus effectiveness. IT goals and objectives stressing customer satisfaction and other service-oriented goals might have fewer efficiency measures. However, well-managed IT activities should take a balanced approach to goals, objectives, and related measures. In the organizations we studied and in the literature we reviewed, one element of success is sifting through the many possible goals, objectives, and measures before finalizing them in a balanced scorecard or similar approach. The sifting process identifies potential objectives and measures for all parts of IT activities. It assesses which measures will be valuable for which purposes and to whom and eliminates measures which are not relevant to customer and stakeholder needs. And it GAO/AIMD-97-163 IT Performance Measurement Guide Page 71 eliminates measures for which good quality data cannot be practically obtained. One manager said, "We want to put the best measures in place. We want them simple, collectable, repeatable, and concrete." A good example of the selection process is the measurement "roadmap" followed by the Minnesota Department of Transportation. The roadmap examines and reduces the total potential to a vital few number of goals, objectives, and related measures. The roadmap, shown in figure 14 and enhanced with additional information from the United Kingdom's Royal Mail, is a general performance measure selection framework that all program and support functions follow. In the roadmap, the word "organization" can refer to an agency, a program, or a support function. Figure 14: IT Measurement Implementation Roadmap Stage I Clarify Goals and Objectives INPUTS DIVERGENCE Full listing of CONVERGENCE Vision/Mission everything the Determine "vital Strategic/Annual Plans few" goals and Legislation/Policies enterprise and programs are trying specific objectives Performance to achieve Expectations Stage II Create Supporting Outcome and Process Measures DIVERGENCE Finalize measurement Review existing CONVERGENCE system: Select a vital Ask 1) What process and Select "vital few" goal and has to happen? -diversity of measures outcome measures for objectives 2) How will we measures: -clarify linkages from each objective the enterprise to the know? keep? discard? task level Repeat for all Goals and Objectives Stage III Put Measures to Work Reinforce Continually Communicate Establish Tackle progress: improve -measures baselines and improvement opportunities -data display measurement -purpose/benefit do -meaningful system -owner benchmarking -outcome feedback -drive -actionability -gap analysis -process -reinforcement improvement Source: Minnesota Dept. of Transportation and United Kingdom's Royal Mail In stage 1, the organization clarifies its overall goals and objectives, first by fulling listing all possibilities, and then converging on the vital few goals and objectives. In stage II, the organization would develop its measures and make sure its measurement system has a diversity of measures and follows a tiering approach. In the last stage, the organization uses the measures, establishing baselines and targets, Page 72 GAO/AIMD-97-163 IT Performance Measurement Guide comparing against benchmarks, and monitoring progress for continual improvement. For support functions such as IT, two roadmaps are in play. One roadmap develops the objectives and measures for an agency or program area; another roadmap takes that agency or program information and develops objectives and measures for IT. This is the intent of the balanced scorecard approach discussed in practice 2. Develop IT Performance Management System Maturity Over Time In discussing the development of an IT performance management system, we attempted to determine if there was a maturity aspect to system development. Several of the managers we talked to believe there is a staging in performance management emphasis and expectations although staging is not precise or formal. Figure 15 shows possible stages in performance management and their linkage to the balanced scorecard areas, drawing on work done for the city of Phoenix and discussed in the literature. While shown as discrete stages for purposes of illustration, the distinction between them often is not clear-cut as they overlap over time. GAO/AIMD-97-163 IT Performance Measurement Guide Page 73 Figure 15: Stages in IT Performance Management deliver reliable, cost-effective operations 1. Traditional focus on internal IT measurement Internal Internal IT determine adherance to standards Business focus on technical factors such as Emphasis Operations quality, productivity, and impact on Innovation & customer service Learning build on competencies to consistently Emphasis 2. Linking to the deliver new and improved systems, MIssion with New solutions on time and within budget and Improved eliminate root causes of defects; improve processes Services link key technical indicators to mission performance Customer apply IT expertise to the mission Needs create an enabling and evolving IT Emphasis 3. Mission/ architecture Results Oriented have measures of IT outcomes in Strategic Needs mission terms, based on customer Emphasis needs and in "customer" terminology Source: Howard Rubin, Ernst & Young, City of Phoenix The premise is that IT organizations have to be good at the basics in stage one -- traditional internal IT operations -- and then move to stages two (linking enterprise and customer mission with new and improved IT services) and three (be mission-results oriented). In other words, an IT organization that is viewed as a failure in day-to-day operations will not have the credibility nor support of the rest of the organization to play a strategic role directly tied to mission results. In stage one, the IT organization is developing and implementing a performance management system that examines internal operations against standards and acceptable levels of performance. Traditional, activity-based measures such as number of reports issued or mainframe availability are used. In stage two, the IT organization is eliminating root causes of defects, building its competencies to consistently deliver new and improved systems and solutions on time and within budget, and linking operational measures to mission performance. The goal is to improve IT processes and prevent defects. For both stage one and two, most of the measures fall in the internal business goal and innovation Page 74 GAO/AIMD-97-163 IT Performance Measurement Guide and learning goal of the balanced scorecard approach discussed in practice 2. In stage three, the IT organization has the capability of applying IT expertise to enterprise and operational customer mission requirements and putting its IT outcomes in those mission terms. Measures are based on customer needs and benefits, expressing measures in terms customers understand, such as business outcomes. Stage three IT organizations also specify who is responsible for corrective action. In this stage, the measures found in the strategic and operational customer goals of the balanced scorecard approach are prominent. The staging or maturity perspective suggests that organization should consider using the balanced scorecard approach in a building block approach. The organization might initially develop major goals, objectives, and measures in each of the four areas. However, it is possible that it likely must perform well in stages one and two, reflecting the balanced scorecard areas of internal business and innovation and learning, before it can perform well in strategic and operational customer areas. A Final Note The organizations we studied cautioned that the practice of IT performance management and measurement is in its infancy. Most of the organizations we studied have worked on their IT performance management systems for several years and most of those efforts are part of strong performance goals at the enterprise level. The cities of Phoenix and Sunnyvale, for example, have long-standing reputations for being well-managed and their general and specific IT measurement approaches have evolved over many years. In both these organizations, as with others we studied, there is a strong performance management culture. The organizations share many similar performance management values and management objectives that stress IT results and accountability. For them, IT measures make a valued and positive difference in mission and business performance. GAO/AIMD-97-163 IT Performance Measurement Guide Page 75 APPENDIX I APPENDIX I Selected Bibliography Allen, Dan. "Performance Anxiety," Computerworld, February 15, 1993. Allen, Dan. "Take a Good Look at Yourself," Computerworld, February 15, 1993. Association for Federal Information Resources Management. The Connection: Linking IRM and Mission Performance. Washington, DC: Association for Federal Information Resources Management, September 1995. Baatz, E. B. "Altered Stats," CIO, October 15, 1994. Bjorn-Andersen, Niels and Davis, Gordon B. (eds.). Information Systems Assessment: Issues and Challenges. Amsterdam: North-Holland, 1986. Brizius, Jack A. and Campbell, Michael D. Getting Results. Washington, DC: Council of Governors' Policy Advisors, 1991. Carlson. W. and McNurlin, B. Measuring the Value of Information Systems. Rockville, MD: I/S Analyzer, 1989. Davis, Dwight B. "Does Your IS Shop Measure Up?" Datamation, September 1, 1992. Earl, Michael J. and Feeny, David F. "Does the CIO Add Value?" Information Week, May 30, 1994. Financial Management Service, Department of Treasury. Performance Management Guide. Washington, DC: Financial Management Service, November 1993. Gaskin, Barbara and Sharman, P. "IT Strategies for Performance Measurement," CMA Magazine, April 1994. Glover, Mark. A Practical Guide for Measuring Program Efficiency and Effectiveness in Local Government. Tampa, FL: The Innovation Groups, 1994. Gold, Charles L. IS Measures -- A Balancing Act. Boston: Ernst & Young Center for Information Technology and Strategy, 1992. Government Centre for Information Systems. Benchmarking IS/IT. London: HMSO, 1995. Page 76 GAO/AIMD-97-163 IT Performance Measurement Guide APPENDIX I APPENDIX I Governor's Office of Budget and Planning, Legislative Budget Board, and State Auditor's Office. Guide to Performance Measurement for State Agencies. Austin, TX: State Auditor's Office, 1995. Harrington, H. James. Total Improvement Management. New York: McGraw-Hill, Inc., 1995. Jackson, Peter and Palmer, Bob. First Steps in Measuring Performance in the Public Sector. London: Public Finance Foundation, 1989. Kaplan, Robert S. and Norton, David P. "The Balanced Scorecard -- Measures that Drive Performance," Harvard Business Review, Vol. 70, No. 2 (January-February 1992). Kaplan, Robert S. and Norton, David P. "Putting the Balanced Scorecard to Work," Harvard Business Review, Vol. 71, No. 5 (September-October 1993). Kaplan, Robert S. and Norton, David P. "Using the Balanced Scorecard as a Strategic Management System," Harvard Business Review, Vol. 74, No. 1 (January-February 1996). Klinger, Daniel E. "A Matter of Marketing," CIO, September 15, 1995. LaPlante, Alice. "IT's Got What It Takes," Computerworld, October 3, 1994. LaPlante, Alice and Alter, Allan E. "IT All Adds Up," Computerworld, October 31, 1994. Lingle, John H. and Schieman, William A. "From Balanced Scorecard to Strategic Guages: Is Measurement Worth It?," Management Review, March 1996. Moad, Jeff. "New Rules, New Ratings as IS Reengineers," Datamation, November 1, 1993. National Academy of Public Administration. Information Management Performance Measures. Washington, DC: National Academy of Public Administration, January 1996. Parker, Marilyn M., Benson, Robert J., and Trainor, H.E. Information Economics: Linking Business Performance to Information Technology. Englewood Cliffs, NJ: Prentice Hall, 1988. Pastore, Richard. "Benchmarking Comes of Age," CIO, November 1, 1995. Quinn, James B. and Baily, Martin N. "Information Technology: The Key to Service Performance," The Brookings Review, Summer 1994. Rubin, Howard. "Measurement: Shifting the Focus to Business Value," Capacity Management Review, Vol. 19, No. 1 (January 1991). GAO/AIMD-97-163 IT Performance Measurement Guide Page 77 APPENDIX I APPENDIX I Rubin, Howard. "Measure for Measure," Computerworld, April 15, 1991. Rubin, Howard A. "In Search of the Business Value of Information Technology," Application Development Trends, November 1994. Sankaran, Chandran. and Taylor, Henry D. "Taking New Measures," CIO, October 1, 1993. Saunders, Carol S. and Jones, Jack W. "Measuring the Performance of the Information Systems Function," Journal of Management Information Systems, Vol. 8, No. 4 (Spring 1992). Society for Information Management Working Group on ISPA. Information Systems Process Architecture 1.0. Chicago: Society for Information Management, September 1993. Society for Information Management Working Group on ISPA. Information Systems Process Architecture 2.0. Chicago: Society for Information Management, March 1996. Society for Information Management Advanced Practices Council. Practitioner's Guide to I. S. Performance Measurement. Chicago: Society for Information Management, March 1995. Strassmann, Paul A. The Business Value of Computers. New Canaan, CT: The Information Economics Press, 1990. Strassmann, Paul A. and others. Measuring Business Value of Information Technologies. Washington, DC: International Center for Information Technologies, 1988. Page 78 GAO/AIMD-97-163 IT Performance Measurement Guide APPENDIX II APPENDIX II Objectives, Scope, and Methodology The objectives of our research were to (1) identify information technology performance management practices used by leading private and public sector organizations with demonstrated success in developing and using information technology solutions and infrastructure to improve mission or business performance and outcomes, and (2) share our results with federal agencies to help improve overall mission performance. Scope Our research focused on information technology performance management practices used by management teams and staff in five private sector companies, three state and local governments, and three federal agencies. The organizations were chosen purposefully, not at random or to ensure representation of a larger group. We selected the private, state, and local organizations based on (1) recognition by other organizations and independent researchers for their progress in successfully developing and using information technology performance management systems, (2) recognition by professional publications for their performance management systems, and (3) willingness to participate as a case study organization. The federal agencies were selected based on their recognition by federal information resources management officials for initiating comprehensive work on information technology performance management. Because our work often involved data that these organizations regarded as proprietary or sensitive, we agreed not to disclose any data they wished to protect. To supplement our findings from these private and public sector organizations, we gathered additional information from other federal, state, and local organizations. This information included both generic and information technology performance management information, ranging from guides to specific practices. Methodology Our research was conducted with an illustrative case study approach using open-ended and focused interviews and documentary analysis, not direct observations. In conducting the case studies, we interviewed senior executives, line managers, and information technology professionals to learn how the organization measured and managed the contribution of information technology towards organizational goals. Interview information was supplemented by documentary analysis of each organization's information technology performance management approach. For quality assurance, we conducted a meeting of case study participants to obtain group comments on an initial draft of this guide, followed by individual case study participant reviews of a subsequent draft. We also distributed the draft to other experts on information technology performance management, the Office of Management and Budget, and the General Services Administration. We also made numerous presentations as the guide was developed to test our GAO/AIMD-97-163 IT Performance Measurement Guide Page 79 APPENDIX II APPENDIX II preliminary findings and the applicability to the federal government with Federal executives, managers, and staff, representing both line and information technology functions. We have incorporated changes where appropriate. Caveats Information technology performance management and measurement is very much in its infancy in both public and private sectors. As an initial step, this guide presents a framework that begins to document the state of the practice drawn from our analysis of a relatively small number of case studies. Much more research and analysis remains to be done. The practices we have presented can serve as starting point for any organization, tailored to the strategic directions and performance requirements unique to each organization. Page 80 GAO/AIMD-97-163 IT Performance Measurement Guide APPENDIX III APPENDIX III Case Study Organizations and Participants We would like to acknowledge the invaluable assistance of the following individuals who serviced as key contacts for the case study organizations. American Express Travel Related Services Company, Inc. Cliff Shoung (Project Leader, City of Phoenix Benchmarking) Joe Motto (Information Technology Paula Bouthillier (Project Leader, Strategic Director) Advantage) Bob Wingenroth (Deputy City Auditor) U.S. General Services Administration City of Sunnyvale Jon Desenberg (Management Analyst) Shawn Hernandez (Director, Information Technology) Immigration and Naturalization Service Marilyn Crane (Information Technology Janet Keys (Director, Systems Policy and Administrative Services Manager) Planning) J. T. Lazo (Senior Systems Manager, EDS, Texas Instruments INS ITP Contract) Gary Pollard (Total Quality Director) Vicki Flewelling (Total Quality Eastman Kodak Company Management) Jeff Duell (Technology Director, IS Measurement and Benchmarking) U.S. Department of Agriculture Joseph Ware (Director, IT Planning and Motorola Semiconductor Products Program Management) Sector Dorothy Hines (Controller, Sector Quality Xerox Corporation and Support Operations) M. Catherine Lewis (Manager, IM2000 Bret Wingert (Manager, SEI Projects) Integration Planning) Margie Tomczak (Manager, Global Oregon Department of Transportation Applications) Craig Holt (formerly Chief Information Officer) GAO/AIMD-97-163 IT Performance Measurement Guide Page 81 APPENDIX IV APPENDIX IV Major Contributors to This Report Accounting and Information Management Division IRM Policy and Issues Group Christopher W. Hoenig, Director Dr. David L. McClure, Senior Assistant Director Bernard R. Anderson, Senior Analyst Los Angeles Office Dr. Sharon Caudle, Project Director Barbara House, Senior Analyst Page 82 GAO/AIMD-97-163 IT Performance Measurement Guide
Executive Guide: Measuring Performance and Demonstrating Results of Information Technology Investments (Exposure Draft) (Superseded by AIMD-98-99)
Published by the Government Accountability Office on 1997-09-01.
Below is a raw (and likely hideous) rendition of the original report. (PDF)