oversight

Executive Guide: Measuring Performance and Demonstrating Results of Information Technology Investments (Exposure Draft) (Superseded by AIMD-98-99)

Published by the Government Accountability Office on 1997-09-01.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                  United States General Accounting Office

GAO           Accounting and Information
              Management Division
________________________________________________
September 1997    Executive Guide

                  Measuring Performance and
                  Demonstrating Results of
                  Information Technology
                  Investments




                  EXPOSURE DRAFT


GAO/AIMD-97-163
Preface
The Government Performance and Results Act of 1993 requires government executives to
focus on defining missions, setting goals, measuring performance, and reporting
accomplishments. In addition, with the passage of the Federal Acquisition Streamlining
Act of 1994 (FASA) and the Clinger-Cohen Act of 1996, performance-based and results-
oriented decision-making is now required for all major investments in information
technology (IT). Clearly, this intense focus on results is one of the most important
management issues now confronting federal agencies.

To assist federal agencies in understanding and devising effective IT measurement
implementation approaches, we examined certain public and private organizations well-
known for their IT performance leadership and management expertise. Similar to our
past efforts examining comprehensive information management practices of other leading
organizations,1 we have taken the lessons learned from these organizations and developed
a suggested framework for agencies to consider when designing and implementing their
IT performance management approaches. We have briefed numerous Chief Information
Officers, agency executives, and agency IT managers on our work over the last 6 months
as part of our effort to advance a pragmatic understanding of what is required to
effectively measure the contribution of IT to mission performance and program outcomes.

Using comprehensive performance information for information management and
technology decisions can advance more informed decision-making about IT investments at
a time when resources are limited and public demands for better government service are
high. Ultimately, the success of results-oriented reform legislation will demand concerted
management effort and long-term commitment. The key practices and steps outlined in
this guide can help agencies achieve success.

This exposure draft was prepared under the direction of Dave McClure, Senior Assistant
Director for Information Resource Management Policies and Issues. If you have questions
or comments about the report, he can be reached at (202) 512-6257. Other major
contributors are listed in appendix IV.



Gene L. Dodaro
Assistant Comptroller General
Accounting and Information Management Division




1
Executive Guide: Improving Mission Performance Through Strategic Information Management and
Technology--Learning From Leading Organizations (GAO/AIMD-94-115), May 1994; Assessing Risks and
Returns: A Guide for Evaluating Federal Agencies' IT Investment Decision-making (GAO/AIMD-10.1.13),
February 1997.

GAO/AIMD-97-163 IT Performance Measurement Guide                                                Page 1
GAO/AIMD-97-163 IT Performance Measurement Guide   Page 2
Contents
                                                                                  Page

Preface                                                                              1

The Demand for Performance Management                                                4

Fundamental Practices: The Foundation of IT Performance Management                  11

Practice Area 1:
Follow an Information Technology "Results Chain"                                    18

Practice Area 2:
Follow a "Balanced Scorecard" Approach                                              31

Practice Area 3:
Target Measures, Results, and Accountability at Decision-making Tiers               45

Practice Area 4:
Build a Comprehensive Measurement, Data Collection, and Analysis Capability         53

Practice Area 5:
Strengthen IT Processes to Improve Mission Performance                              64

Key Lessons Learned for Effective Implementation                                    70

Appendix   I:     Selected Bibliography                                             76
Appendix   II:    Objectives, Scope and Methodology                                 79
Appendix   III:   Case Study Organizations and Participants                         81
Appendix   IV:    Major Contributors to This Report                                 82




Page 2                                 GAO/AIMD-97-163 IT Performance Measurement Guide
Figures
                                                                               Page

Figure 1: IT Performance Management Approach                                     12

Figure 2: Performance Measurement - A Strategic Information Management
          Best Practice                                                          15

Figure 3: Implementing the Results Act - Key Steps and Critical Practices        16

Figure 4: An IT Results Chain                                                    19

Figure 5: A Hypothetical Chain of Events/Evidence for Achieving
           Environmental Quality                                                 27

Figure 6: An IT Balanced Scorecard Approach                                      33

Figure 7: Balanced Scorecard - IT Strategic Measures                             37

Figure 8: Balanced Scorecard - IT Customer Measures                              39

Figure 9: Balanced Scorecard - IT Internal Business Measures                     41

Figure 10: Balanced Scorecard - IT Innovation and Learning Measures              43

Figure 11: Performance Measurement Tiers                                         46

Figure 12: A Tiered Performance Scorecard Example                                50

Figure 13: Information Systems Process Architecture Version 2.0 Process Framework 65

Figure 14: IT Measurement Implementation Roadmap                                 72

Figure 15: Stages in IT Performance Management                                   74




GAO/AIMD-97-163 IT Performance Measurement Guide                              Page 3
The Demand for Performance Management
_________________________________________________________________________________

            Increasingly, federal policy-makers are insisting that government
            executives provide hard facts on mission and program results. Program
            authorizations, resource decisions, and oversight requirements
            increasingly hinge on how well agencies perform against expectations
            and improve performance over time. As such, a new standard for
            management expertise is evolving: setting performance targets, designing
            efficiency and effectiveness measures, systematically and accurately
            measuring outcomes, and then using the results for informed decision-
            making.

            Information technology (IT) products, services, and delivery processes
            are important resources for results-driven government programs and
            operations. For purposes of this guide, IT also includes the
            organizational unit or units and contractors primarily responsible for
            delivering IT. Line managers--the operational customers2 relying on IT
            products and services--and IT managers themselves, want to know "How
            are information technology products and services, including the
            information infrastructure, supporting the delivery and effectiveness of
            the enterprise's (agency) programs?" As we pointed out in an earlier
            report, successful organizations rely heavily on performance measures to
            operationalize mission goals and objectives, quantify problems, evaluate
            alternatives, allocate resources, track progress, and learn from mistakes.3
            Operational customers and IT managers in these organizations form
            partnerships to design, manage, and evaluate IT systems that are critical
            to achieving improved mission success.

            Performance Management:
            What Are the Benefits?
            In an effective performance management approach, measures are not
            used for assigning blame or to unknowingly comply with reporting
            requirements. Quite simply, they are used to create and facilitate action
            to improve performance. Measures and performance information must
            link to strategic management processes. An effective performance



            2
             For purposes of this guide, we define an "operational customer" as a program or other
            function to whom the information technology organization or units delivers IT products
            and services. The operational customer can include organizations and entities outside
            traditional agency boundaries affected by the use of IT products and services.

            3
                GAO/AIMD-94-115.

            Page 4                         GAO/AIMD-97-163 IT Performance Measurement Guide
management system produces information that delivers the following
benefits.4

•   Provides an early warning indicator to correct problems, or to
    examine if corrective action is having any effect.

•   Provides input to resource allocation and planning. It can help
    organizations prepare for future conditions that likely will impact
    program and support function operations and the demands for
    products and services, such as decreasing personnel, financial
    resources or changes in workload. Use of measures can give
    organizations a long lead time for adjustment if these conditions are
    known in advance.

•   Provides periodic feedback to employees, customers, stakeholders,
    and the general public about the quality, quantity, cost, and timeliness
    of products and services.

Across all of these benefits is an overarching one -- measures build a
common results language among all decision-makers. What measures the
organization picks basically say what the organization is accountable for,
and what it should benchmark and compare against.


Results-Oriented Legislation Provides
Impetus for Performance Based Management
For the past several years, the Congress has emphasized federal
performance improvement, accountability for achieving results, and cost
reduction. Legislative requirements in the Chief Financial Officers Act
(CFO) of 1990, the Government Performance and Results Act (Results
Act) of 1993, the Federal Acquisition Streamlining Act (FASA) of 1994,
the Paperwork Reduction Act of 1995, and the Clinger-Cohen Act of 1996
expect improvements in IT performance management. These laws
reinforce financial accountability, emphasize results-oriented
management, define cost performance and schedule goals, and improve
the acquisition of IT to streamline federal programs.

Under the CFO Act, CFOs are responsible for developing and maintaining
integrated accounting and financial management systems that include
systematic measurement information on agency performance. OMB's


4
 This information is based on material in Jack A. Brizius and Michael D. Campbell,
Getting Results (Washington, D.C., Council of Governors' Policy Advisors, 1991) and
performance management guidance materials prepared by the National Academy of
Public Administration and the Innovation Group.

GAO/AIMD-97-163 IT Performance Measurement Guide                                      Page 5
Management Circular A-11 (agency guidance for preparing and submitting
budget estimates) encourages agencies to review program performance
information contained in the most recent financial statements prepared
under the CFO Act when developing their program performance
indicators. The Results Act directs federal agencies to improve their
program management by implementing outcome-oriented performance
measurement systems. Agencies are to prepare annual performance
plans with objective, quantifiable, and measurable performance indicators
to measure relevant outputs, service levels, and outcomes of each
program activity. Meeting these requirements is critical in developing the
mission goals and performance expectations which IT products and
services will support. Our Executive Guide: Effectively Implementing the
Government Performance and Results Act contains additional guidance
on results-oriented management under the Results Act.5

The Federal Acquisition Streamlining Act requires federal agencies to
assess major acquisition cost, performance, and scheduling. Agency
heads must determine if there is a continuing need for programs that are
significantly behind schedule, over budget, or not in compliance with
performance or capability requirements. Congressional policy is that
each executive agency should achieve, on average, 90 percent of the cost
and schedule goals established for agency programs.

The Paperwork Reduction Act requires agencies to establish information
resources management goals improving the productivity, efficiency, and
effectiveness of agency operations and methods for measuring progress
in achieving the goals. OMB's Circular A-130 (guidance resulting from
the Paperwork Reduction Act) highlights the importance of evaluation
and performance measurement. It recommends that agencies seek
opportunities to improve the effectiveness and efficiency of government
programs through work process redesign and the judicious application of
IT. Agencies must perform various benefit-cost analyses to support
ongoing management oversight processes and conduct post-
implementation reviews of information systems to validate estimated
benefits and document effective management.

The most recent legislation affecting IT performance management, the
Clinger-Cohen Act, requires agencies to establish efficiency and effective
program improvement goals using IT. Performance measurements must
assess how well IT supports agency programs. Agency heads must
benchmark agency process performance against comparable processes in
terms of cost, speed, productivity, and quality of outputs and outcomes.
Agency heads must analyze agency missions and make appropriate


5
See Executive Guide: Effectively Implementing the Government Performance and
Results Act (GAO/GGD-96-118, June 1996).

Page 6                       GAO/AIMD-97-163 IT Performance Measurement Guide
changes in mission and administrative processes before making
significant IT investments to support missions. Annual performance
reports cover how well each agency improves agency operations through
IT. OMB has issued specific guidance to assist agencies in the
implementation of the Clinger-Cohen Act that requests specific
information on IT cost, benefit, and risk. Most notable is OMB
Management Circular A-11, Part 3, that provides instructions on agency
budget submissions and the OMB Director's policy memorandum M-97-02
(also known as "Raine's Rules") that specifies investment criteria that IT
projects are to meet to qualify for inclusion in the President's budget
submission to the Congress.

These various legislative and executive branch requirements create
pressure for top management's increased attention on IT performance.
However, for federal agency managers, the challenge is not complying
with these legislative and regulatory requirements, but "managing and
measuring to results" using a well-crafted IT performance management
system. At a high level, federal managers can start framing a system by
asking:

•   What are enterprise (agencywide) and key operational customer IT
    performance expectations?
•   What are the vital few IT objectives given these expectations?
•   What measures are appropriate for these IT objectives?
•   What is IT's current baseline performance and what should be the
    target performance?
•   How will IT management and customers work together to use these
    measures to leverage and improve IT performance in ways that will
    improve mission delivery?

Answering these questions signals a significant change in determining
how IT contributes to achieving improved program outcomes.
Traditional measures such as response time and systems availability are
by themselves insufficient to answer IT performance questions.
"Measures such as machine hours are easy," said one manager we
interviewed, "what is difficult is how to measure the business value of
the applications."




GAO/AIMD-97-163 IT Performance Measurement Guide                       Page 7
Identifying Best Practices:
Learning from Leading Organizations
Federal managers are seeking guidance on developing and implementing
agency IT performance management systems.6 To assist agencies, we
examined how certain leading organizations approach IT performance
management, studying practices of both public and private sector
organizations recognized by peers and independent researchers for their
IT performance efforts. (A more detailed description of our case study
selection methodology is found in appendix 2.)

The private sector companies used as case studies were

•   Xerox Corporation,
•   Eastman Kodak Company,
•   Texas Instruments,
•   Motorola Semiconductor Products Sector, and
•   American Express Travel Related Services Company.

In the public sector, we studied

• Oregon Department of Transportation,
• Sunnyvale, California, and
• Phoenix, Arizona.

We also selectively included the U.S. Immigration and Naturalization
Service, the U.S. General Services Administration Information Technology
Service, and the U.S. Department of Agriculture to assess




6
 Recent efforts include The Connection: Linking IRM and Mission Performance, a
resource paper sponsored by the Association for Federal Information Resources
Management, September 1995; Practitioner's Guide to I.S. Performance Measurement, a
guide issued by the Society for Information Management's Advanced Practices Council,
March 1995; Information Management Performance Measures, a report for the U.S.
Department of Defense issued by the National Academy of Public Administration,
January 1996, Performance-Based Management: Eight Steps to Develop and Use
Information Technology Performance Measures Effectively, a guide prepared by the
General Services Administration, December 1996; and Guide for Managing Information
Technology (IT) as an Investment and Measuring Performance, Department of Defense
guidance issued by Assistant Secretary of Defense for Command, Control,
Communications, and Intelligence, March 3, 1997.

Page 8                        GAO/AIMD-97-163 IT Performance Measurement Guide
early federal practices.7 We gathered additional information from general
and IT performance management literature and reports.

We preceded the organizational research by an extensive review of the
generic performance management and IT performance management and
measurement literature, guides, and reports. We also consulted with
experts involved in IT performance management and measurement
efforts. We collected organizational data through interviews and
documentary analysis, not direct observation. Case study organizations
reviewed our results for accuracy and completeness. We also gave
briefings to federal officials to discuss our results. Appendix II provides
a more detailed description of our scope and methodology.

Our guide contains key practices we found from our organizational
research and key concepts and practices extracted from available
literature, guides, and reports. They supplement recent GAO reports and
testimonies.8 Much still remains to be learned. This guide is an initial
effort in an area very much in its infancy. Without exception, those in
the organizations we studied noted that IT performance management and
measurement practices are not completely defined. However, their
experiences did translate to the key practices we describe in this guide.

Understanding the Context of IT
Performance Management
We have found from our research that there is not one "best" approach
to IT performance management. How IT performance management is
designed, implemented, and sustained in each organization depends on a
multitude of contextual factors, such as


7
 Other organizations we consulted were the Florida Legislature Joint Committee on
Information Technology Resources, the Oregon Department of Administrative Services,
the Oregon Secretary of State Audits Division, the Office of the Texas State Auditor, the
Minnesota Office of the Legislative Auditor, the Minnesota Department of
Transportation, the Federal Emergency Management Agency, the city of Portland, the
Treasury Board of Canada, the United Kingdom's Government Centre for Information
Systems and Royal Mail, the Society for Information Management, and the MITRE
Corporation.

8
 These include Government Reform: Goal-Setting and Performance (GAO/AIMD/GGD-95-
130R, March 27, 1995); Managing for Results: Experiences Abroad Suggest Insights for
Federal Management Reform (GAO/GGD-95-120, May 2, 1995); Managing for Results:
Critical Actions for Measuring Performance (GAO/T-GGD/AIMD-95-187, June 20, 1995);
Managing for Results: Achieving GPRA's Objectives Requires Strong Congressional Role
(GAO/T-GGD-96-79, March 6, 1996); The Government Performance and Results Act:
1997 Governmentwide Implementation Will Be Uneven (GAO/GGD-97-106, June 2, 1997);
Managing For Results: The Statutory Framework for Improving Federal Management
and Effectiveness (GAO/T-GGD/AIMD-97-144, June 24, 1997).

GAO/AIMD-97-163 IT Performance Measurement Guide                                      Page 9
Page 10   GAO/AIMD-97-163 IT Performance Measurement Guide
•   whether the organization's culture--leadership, decision-making,
    appraisal and reward systems--supports IT performance management;
•   how important IT is for program (mission) delivery;
•   how widespread IT is used in the organization;
•   what IT activities are centralized, dispersed, or decentralized; and
•   the availability of resources such as skills and tools to support
    performance management.

Factors such as these, taken together, provide a unique environment
impacting IT performance. However, as with most things important to
organizational health, another significant contextual factor is top
management ownership, involvement in, and use of IT performance
information. What the top leaders pay attention to, what messages they
send about IT performance management, and what action they take
based on the measures tells the true story of IT performance
management acceptance in any organization.

In the organizations we studied, there is strong management attention on
IT measures and their rigorous use in decision-making at all management
levels to improve IT performance. A second significant factor for
effective IT performance is the partnership forged between IT and the
enterprise and operational customers. In a large sense, the enterprise
and operational customers "co-produce" IT results because they are the
consumers and users of IT products and services. The starting point for
IT objectives is organizational goals and objectives. What IT does and
what is measured must directly align with those organizational goals and
objectives. While IT management and staff serve as business consultants
on how current and emerging IT can aid mission objectives, they must
rely on the ongoing engagement of organizational customers in defining
how IT can facilitate mission accomplishment.




GAO/AIMD-97-163 IT Performance Measurement Guide                    Page 11
Fundamental Practices: The Foundation of
IT Performance Management
______________________________________________________________________________

            Our case study research clearly indicated that knowing what
            performance management is and is not is the starting point for
            developing an IT performance management system. In simple terms, any
            performance management system assesses how well an organization
            delivers expected products and services directly that are tied to its goals
            and objectives. It also incorporates the products and services from
            enterprise and program support functions such as IT, financial
            management, or human resources management. Within this context, IT
            performance management and measures are considered subsets of
            overall performance management systems.

            Our IT performance management approach includes several important
            distinguishing characteristics that are discussed in greater depth in
            different parts of this guide. These characteristics include:

            •   differentiating between IT''s impact on intermediate versus final
                program outcomes,
            •   using a good balance of different kinds of IT measures,
            •   understanding that measures may differ by management tier within an
                organization, and
            •   evaluating both the overall performance of the IT function within an
                organization and the outcomes for individual IT investments.

            Our approach suggests three distinct practice areas that involve
            aligning IT systems with agency missions, goals, and programs;
            constructing measures that determine how well IT is supporting
            strategic, customer, and internal business needs; and implementing
            performance measurement mechanisms at various decision-making levels
            within an organization.

            Two supporting practice areas are important to keep the overall IT
            measurement process working. Data collection and analysis
            capabilities must effectively support the performance management
            system being used in such a way that performance data is accessible,
            reliable, and collected in the least burdensome manner. The benefit of
            effective automated data and management information systems is that
            performance information can be effectively and efficiently used to make
            strategic, managerial, and day-to-day operational decisions. In addition,
            a constant focus on strengthening the processes and practices being
            used to deliver IT products and services is essential for building and
            maintaining effective IT organizations.

            Page 12                   GAO/AIMD-97-163 IT Performance Measurement Guide
Figure 1 shows the generic model produced by our case study research
on IT performance measurement.


Figure 1: IT Performance Management Approach


                                  ALIGN


                              Practice Area 1
                                   Use an
                                  IT Results
                                    Chain
 IMPLEMENT                                                  CONSTRUCT
                              Practice Area 4
         Practice Area 3          Build data        Practice Area 2
         Target Measures      Collection/analysis       Follow a
               at                 Capability            Balanced
         Decision-making                                Scorecard
             Levels           Practice Area 5           Approach
                                 Improve IT
                                 Processes


                               REINFORCE




   Practice Area 1:    Follow an IT "results chain"

Leading organizations build and enforce a disciplined flow from goals to
objectives to measures and individual accountability. They define
specific goals, objectives, and measures, use a diversity of measure types,
and develop a picture as to how IT outputs and outcomes directly impact
operational customer and enterprise (agency) program delivery
requirements. The IT performance management system does not
optimize individual customer results at the expense of an enterprise
(agency) perspective. Operational customer goals and measures meet IT
department or unit objectives that are matched to enterprise strategic
directions or goals.



GAO/AIMD-97-163 IT Performance Measurement Guide                      Page 13
   Practice Area 2:    Follow a balanced scorecard approach

Leading organizations use an IT goal, objective, and measure approach
that translates organizational strategy and IT performance expectations
into a comprehensive view of both operational and strategic measures.
Four generic goal areas include meeting the strategic needs of the
enterprise, meeting the needs of individual operational customers,
addressing internal IT business performance, and addressing ongoing IT
innovation and learning.

   Practice Area 3:    Target measures, results, and accountability
                       at different decision-making tiers

For the balanced scorecard areas, leading organizations match measures
and performance results to various decision-making tiers or levels.
These tiers cover enterprise executives, senior to mid-level managers
responsible for program or support units, and lower-level management
running specific operations or projects. The organizations we studied
place IT goals and measures in widely distributed IT performance
improvement plans. Individual appraisals tie IT performance to
incentives.

   Practice Area 4:    Build a comprehensive measure, data
                       collection, and analysis capability

Leading organizations give considerable attention to baselining,
benchmarking, and the collection and analysis of IT performance
information. They use a variety of data collection and analysis tools and
methods which not only keep them on top of IT performance
management, but reduce the burden of collection and analysis. They
also periodically review the appropriateness of their current measures.

   Practice Area 5:    Improve performance of IT business
                       processes to better support mission goals

In the leading organizations, IT performance improvement begins and
ends with IT business processes. The organizations map their IT
business processes and select those processes which must be improved
to support an enterprise and operational customers' business processes.




Page 14                   GAO/AIMD-97-163 IT Performance Measurement Guide
Measurement Maturity: Start With the Basics
and Increase Sophistication Over Time
Developing performance measures that demonstrate the impact of
information technology on mission performance requires management
commitment, experience in constructing and evaluating measures, and a
constant learning environment. Many of the organizations we talked to
indicated that had attempted to develop strategic or mission impact
measures without first realizing that they had to demonstrate strong
capability and sound performance in the basics of IT management. In
short, if an IT unit was not being very successful in providing quality
products and services to the rest of the organization, it had little
credibility in measuring strategic contributions to mission or business
results.

As such, several IT managers emphasized the need to start with the
basics by assessing the quality and effectiveness of existing internal IT
operations. This evaluation and early measurement construction exercise
can focus on such things as (1) delivery of reliable, cost-effective, high
quality IT products and services, (2) adherence to industry standards for
systems design, cost estimation, development, and implementation, (3)
internal customer satisfaction, (4) staff productivity, and (5) technical
skills and capability. All of these factors are important dimensions in
providing effective IT support to business operations and improving
overall organizational performance.

Starting with measures of internal IT operations offers some advantages,
even though they should be viewed as a substitutes for other
measurements of IT contribution to specific program and mission area
results. First, it preempts the problem of IT organizations waiting for
development and consensus of mission or business specific performance
measures. Second, it provides the IT organization with valuable
experience in performance measurement construction and evaluation
which is easily transferable to mission-related measures. Third,
constructing performance measures for IT operations conforms with a
balanced scorecard approach which emphasizes the need for a diversity
of measures in examining IT performance. And fourth, introducing
performance measurement maturity over time is a critical factor affecting
the overall success of implementing performance management in an
organization--an issue we discuss in greater detail in the final section of
this report.




GAO/AIMD-97-163 IT Performance Measurement Guide                      Page 15
IT Performance Is Essential for
Strategic Information Management
Performance measurement is not an end, but rather the means to
achieving better management results. In our May 1994 Executive Guide
on strategic information management,9 we noted that leading
organizations use performance measures to objectively evaluate mission,
business, and project outcomes. These organizations (1) focused
performance measures on gauging service to key management processes,
(2) embedded performance measures in key management processes, (3)
used internal and external benchmarks to assess relative performance,
and (4) tailored performance measures to gauge whether information
technology made a difference in improving performance.

As shown in figure 2 performance measurement is a cornerstone practice
that GAO advocates as part of an integrated strategic information
management approach.

Figure 2: Performance Measurement--A Strategic Information
          Management Best Practice




          Decide to   Communicate the urgency to change IT practices
          Change      Get line management involved and create ownership
                      Take action and maintain momentum

                      Define customer needs and mission goals
            Direct    Measure the performance of mission processes
           Change     Focus on process improvement
                      Manage IT projects as investments
                      Integrate planning, budgeting, and evaluation

                      Establish customer/supplier relationships
           Support    between managers and IT professionals
           Change     Create a Chief Information Officer
                      Upgrade IT skills and knowledge




9
GAO/AIMD-94-115.

Page 16                     GAO/AIMD-97-163 IT Performance Measurement Guide
In June 1996, GAO issued a companion Executive Guide on a suggested
performance measurement implementation approach for the Results
Act.10 The approach, depicted in figure 3 identifies certain key steps and
associated practices that agencies may find useful for implementation of
the Results Act. The approach is based on the actions taken by certain
organizations that have successfully undertaken performance
improvement initiatives similar to that required by the act.

Figure 3: Implementing The Results Act -- Key Steps and
          Critical Practices


                                       STEP 1
                                 Define Mission and
                                 Desired Outcomes
                                 Practices:
                                 1. Involve stakeholders
                                 2. Assess environment
                                 3. Align activities,
                                    core processes,
                                    and resources




            STEP 3                                                  STEP 2
                                    Reinforce GPRA            Measure Performance
       Use Performance
       Information                  Implementation            Practices:
                                 Practices:                   4. Produce measures at
       Practices:                9. Devolve decision-making      each organizational
                                    with accountability          level that
       6. Identify performance                                  demonstrate results,
          gaps                   10. Create incentives
                                                                are limited to a vital
       7. Report information     11. Build expertise
                                                                few,
       8. Use information        12. Integrate management
                                                                respond to multiple
                                     reforms                    priorities, and
                                                                link to responsible
                                                                programs
                                                              5. Collect data




Each organization GAO studied set its agenda for management reform
according to its own environment, needs, and capabilities. In striving to
become more results oriented, they commonly took three steps in
implementing a performance based management approach. First, they
defined clear missions and desired outcomes. Second, they measured
performance to gauge progress. Third, they used performance
information as a basis for decision-making.


10
 GAO/GGD-96-118.

GAO/AIMD-97-163 IT Performance Measurement Guide                                         Page 17
Page 18   GAO/AIMD-97-163 IT Performance Measurement Guide
Along with these steps, certain practices proved especially important to
the success of their efforts. Taken together, these steps and practices
were useful in making changes necessary for these organizations to
become results oriented. These fundamental steps and practices are
consistent with the Results Act requirements and provide a useful
framework for federal agencies to adopt in implementing key provisions
of the law.

The IT performance management approach outlined is this guide works
in tandem with GAO's Results Act implementation model and
demonstrates how IT performance measurement can be implemented
within an overall performance management framework. Most
importantly, mission goals, objectives, and strategies must be understood
in order to evaluate how IT contributes to performance improvements.

In the sections that follow, each practice area in our approach is
explained in detail, listing specific characteristics and providing case
study examples to illustrate how they are implemented. A final section
discusses key steps involved in implementing an IT performance
management system.




GAO/AIMD-97-163 IT Performance Measurement Guide                      Page 19
Practice Area 1:                                                 ALIGN


Follow An                                                    Practice Area 1
                                                                 Use an

Information                            IMPLEMENT
                                                                IT results
                                                                  chain
                                                                                         CONSTRUCT
Technology                                 Practice Area 3
                                                             Practice Area 4
                                                                 Build data        Practice Area 2

"Results Chain"                            Target measures
                                                 at
                                           decision-making
                                                             collection/analysis
                                                                 capability
                                                                                      Follow a
                                                                                      balanced
                                                                                      scorecard
                                               levels        Practice Area 5          approach
                                                                Improve IT
                                                                processes

                                                             REINFORCE




  Practice Area Characteristics:

  1. Directly map IT goals and measures to organizational mission
     goals, objectives, and measures.
  2. Prepare a chain of events and evidence to understand IT's
     contribution to enterprisewide and operational customer
     objectives
  3. Use a diversity of measures to evaluate IT performance




Practice Area Overview
To maximize the results of IT investments, leading organizations ensure
that IT programs align with and directly support high-level organizational
missions, goals, and objectives. This practice provides an approach for
linking organizational goals and objectives to the "vital few" IT
performance measures needed to manage for effective results. This
framework, formal or informal, follows a systematic movement through
what can be called an "IT results chain." The results chain approach
provides discipline for aligning performance expectations and measures
at all levels.

Any effort to measure IT performance must begin with clearly defined
organizational and programmatic goals and objectives. In other words,
an organization cannot properly define its IT goals and objectives (much
less measure to determine the degree of success in meeting them) unless
it has clearly defined goals and objectives for the programs that IT
supports. The resulting IT goals and measures must, in all cases, map
back to program or strategic (enterprise-level) goals. To help understand
the relationships between operational programs and the IT contribution

Page 18               GAO/AIMD-97-163 IT Performance Measurement Guide
to their success, many organizations prepare a chain of events and
evidence to show how programs work and how success might be
measured. Finally, a diversity of qualitative and quantitative measures
are used for the inputs, outputs, and outcomes of IT programs.

In short, a results chain approach

•   defines what the organization is attempting to accomplish,
•   allows an organization to identify success, and
•   links IT projects directly to business goals and objectives.

As shown in figure 4, the chain shows the links from organizational goals
and objectives to IT performance measures. In the organizations we
studied, measuring IT's contribution begins by defining organizational
goals and objectives--for the enterprise and for internal and external
customers. These goals and objectives should be based on defined
organizational mission statements.

Figure 4: An IT Results Chain
                                                 M
                                                  ea




      Organizational
                                                    su a onv and
                                          di



                                                      re nd erg me
                                             v




      Goals & Objectives
                                             e          rg


                                                        m A e as
                                                           e/


                                                           en lig go ur
                                                             c



                                                               t D nm als es




                   IT Purpose
                                                                  ev en , ob



                   (Mission/Vision)
                                                                    el t jec
                                                                      op
                                                                        m



                                      IT Goals
                                                                          en
                                                                             t
                                                                               tiv
              C




                                                                                   es




                                                     Vital Few
               C ctio
               ol




                                                                                      ,
                M mu An
                om n




                                                     IT Objectives
                 le


                  ea ni al
                    su ca ys
                      re tio is,




                                                                     Vital Few
                       ,




                                                                     IT Measures
                            n, U




                                                                                 IT Management
                                                                                     and Staff
                                 se




                                                                                       Performance
                                                                                         Expectations




GAO/AIMD-97-163 IT Performance Measurement Guide                                             Page 19
Page 20   GAO/AIMD-97-163 IT Performance Measurement Guide
As for other programs, IT management and staff can then develop a
purpose statement that specifically defines how IT products and services
will be used to support the achievement of organizational and customer
goals. The purpose statement is then translated into IT goals, objectives,
and measures.

IT goals and objectives should be consistent with the IT purpose
statement and clearly linked to organizational goals and objectives.
Leading organizations focus on a "vital few IT objectives" to demonstrate
results in selected key areas. Similarly, the number of measures for each
IT goal should be limited to the "vital few." These should be limited to
the key IT performance dimensions that will enable the IT organization
to assess accomplishments, make decisions, realign processes, and assign
accountability. Attempts to manage an excess number of IT measures
increase risks of confusing excess data with key IT performance issues.
Lastly, management and staff performance evaluations are linked to the
performance measures as a way of comparing achievements with
planned results.

As part of a results-oriented management approach, IT performance
measurement must be used in the decisionmaking process. Measurement
development and alignment involves consideration of all organizational
goals and objectives and converging on the vital few IT goals, objectives,
and measures. Effective measurement must be supported by sound data
collection and analysis methods and communication of results to
management and staff.

Practice Area Characteristics
1. Directly Map Information Technology and Management Goals
   and Measures to Strategic Goals

Use of an IT results chain is only as good as the clarity and specificity of
the overall organizational goals and objectives. Leading organizations
build consensus among program managers, IT managers, customers,
stakeholders, and staff to establish joint ownership for performance
management. They work together to achieve a common understanding
of goals, objectives, measures, and anticipated outcomes. As a practical
matter, those who will judge the success of programs and the supporting
functions should agree on the links in the results chain from IT's purpose
to the vital few measures. 11



11
 For related guidance, see Agencies' Strategic Plans Under GPRA:
Key Questions to Facilitate Congressional Review (GAO/GGD-
10.1.16, May 1997).

GAO/AIMD-97-163 IT Performance Measurement Guide                       Page 21
             In the organizations we examined, IT goals and measures flow directly
             from strategic goals. IT managers and staff do not develop performance
             management systems that optimize operational customer results without
             considering an enterprisewide perspective. IT goals and measures in
             support of individual operational customers must meet IT department or
             unit objectives. In turn, IT department or unit objectives must map
             directly to both programmatic and enterprisewide strategic directions or
             goals. The result is that IT goals and measures track in a seamless
             fashion back to enterprise strategic directions or goals. If such mapping
             is not obvious when comparing measures and high-level goals, the IT
             function is probably not measuring the right things.


Case Study 1:
Linking Measures to Specific IT Goals and Objectives

The city of Phoenix's Information Technology Department (ITD) wanted a set of business-oriented
customer-based measurements to measure the effectiveness of its ITD operations. One manager said,
"We set forth to come up with a 'net result measuring scheme.' It would (1) focus on customer
satisfaction, (2) be employee friendly, (3) not affect program work, (4) improve customer satisfaction,
and (5) give the customer the ability to understand the value our organization gives to them, and let
them suggest how to help us get better."

The department wanted to (1) lay the groundwork for continuous monitoring and improving customer
satisfaction with ITD service, (2) better define the value-added of ITD services from a business
perspective, (3) identify continuous improvement needs in internal ITD operations, and (4) strengthen
ITD relationships with other city departments and manage customer expectations using hard facts and
measures.

The department identified measures based on input from key selected customers and staff and the ITD
objectives. Objectives were developed using the City of Phoenix Strategic Directions Report and the IT
architecture vision. Generally, the strategic directions were to (1) create and maintain a well-informed
and involved community with a clean and safe environment, efficient transportation systems, a quality
education system, and economic opportunity for all citizens, (2) provide cost-effective, high-quality
community services through a productive, efficient organization and an empowered work force, and (3)
generate and maintain desirable revenue flows and sound financing within a system of rational resource
allocation. Goals for the architecture vision were to make all applications integrated and fully
compatible, using a city-wide deployed IT architecture supported by commonly shared data repositories.
The department then revised the measures to meet all of the ITD objectives and correlated the measures
to customer-defined goals of time, cost, and customer satisfaction.

The table on the following page shows how Phoenix measures progress towards achieving department
objectives. These measures focus on timeliness (service delivery), cost (producing city services at
acceptable or reduced cost), or customer satisfaction (evidence that internal and external customers are
satisfied and that ITD's contribution can be quantified).




             Page 22                       GAO/AIMD-97-163 IT Performance Measurement Guide
Case Study 1:
The City of Phoenix: Relating Performance Metrics to IT Objectives
IT Department Objectives      Net Results Metrics (Examples)         Type of        IT Objectives
for Meeting City                                                     Measure        Linked To
Strategies                                                                          Measures


1   Build partnerships with   Successful delivery of ITD products    Time           2, 3, 4, 5, 6, 11
    city departments.         or services on time v. goal.
2   Provide enabling          Reduced delivery time v. goal.         Time           3, 4, 5, 6, 11
    technology to city
    departments.
3   Increase customer         Problem responsiveness v. goal.        Time           1, 3, 4, 5, 6, 8, 12
    satisfaction.
4   Reduce cycle times in     Reduction attributed to ITD v. goal.   Cost           3, 9, 10, 11, 12
    delivery of ITD
    products and services.
5   Reduce service delivery   Customer satisfaction and              Customer       all
    costs.                    relationships v. goal.                 satisfaction
6   Improve service           Reliability of products and services   Customer       3, 6, 8, 12
    delivery processes.       v. goal.                               satisfaction
7   Implement citywide        Systems using citywide                 Customer       1, 3, 4, 7, 8, 9, 10
    technology                architecture/new emerging              satisfaction
    architecture.             technologies v. goal.


8   Improve citizen access    ITD staff trained in customer          Customer       1, 2, 3, 9, 10, 11
    to information and        service skills and new technology v.   satisfaction
    services through          goal.
    technology.
9   Increase resource         New ideas received and adopted         Customer       1, 3, 6, 9, 10, 11
    versatility.              from ITS staff v. goal.                satisfaction
10 Improve leadership         Employee satisfaction v. goal.         Customer       3, 9, 10, 11
   skills.                                                           satisfaction
11 Increase employee          Effective communications v. goal.      Customer       1 ,2 ,3, 4, 11, 12
   confidence.                                                       satisfaction
12 Achieve world class        ITD involvement in departmental        Customer       1 ,2, 3, 4, 7, 11, 12
   results                    technology planning v. goal.           satisfaction




               GAO/AIMD-97-163 IT Performance Measurement Guide                                          Page 23
Case Study 2:
Connecting Goals, Strategies, Objectives and Measures

To better meet customer needs and respond to a changing market environment, in the
mid 1990's Xerox developed a Xerox 2000 business strategy. In response to the new
strategy, the information management organization--Global Process and Information
Management (GP&IM)--developed four strategies and three breakthrough goals to align
IM programs to support achievement of the outcomes anticipated by the corporate level
business drivers. Senior executives ultimately decided on eight measures to determine
whether IM programs were supporting corporate-level strategies.

GP&IM concluded that five factors were significantly influencing how it developed a new
IT strategy for Xerox. These were (1) the push to reengineer Xerox business processes, (2)
knowledge that the current information management environment was not the
foundation for the future of Xerox, (3) information systems investment decisions were
currently driven by "entitlements," not a strong selection strategy, (4) renewal of the
existing infrastructure was too costly, and (5) the cycle time for information management
was not meeting business requirements.

GP&IM then developed four strategies to respond to these five IT strategy "drivers": (1)
reduce and redirect information spending through global outsourcing, reducing legacy
system spending, and consolidating and sharing resources, (2) infrastructure
management including asset management, (3) leverage worldwide information resources
of people, hardware, and software, and (4) develop business process-driven solutions.

As shown in the figure on the following page, GP&IM has several outcomes identified for
each of the four strategies. GP&IM also developed three proposed high-level
"breakthrough" goals covering the four strategies, shown on page 24. The goals are
defined as follows:

Manage IM spending suggests that IM2000 must be to managed to a flat ceiling for
base spending while doubling the amount of investment in new development.

Deliver infrastructure renewal determines the penetration of the new infrastructure in
conjunction with some renewal of the old infrastructure (known as Globalview).

Deliver business process solutions covers reengineering process deployment and drives
operational optimization and the development of a baseline for day-to-day work. The
baseline is for comparison with Electronic Data Systems (EDS) outsourcing
arrangements. Each breakthrough goal area has measure examples.

Xerox top management decided that the eight measures cited in figure 8 provided the right
information on whether their IT strategies were being achieved. For example, the measures told
them how well they were performing in actually redirecting IT spending into new applications and
systems development, retiring the old infrastructure and replacing it with more companywide
solutions, reducing operations and maintenance costs, and whether new process-driven IT solutions
were providing timely, high-quality products near estimated costs.




            Page 24                      GAO/AIMD-97-163 IT Performance Measurement Guide
Case Study 2
Xerox IM 2000: Connecting Strategies With Outcomes




      Strategies                             Expected Outcomes
 GAO
                              Orderly retirement of older infrastructure and applications
 Reduce/Redirect IM           Increased IT investment into new applications and infrastructure
     Spending                 development and deployment
                              Smaller Xerox IM staff focused on high value IT projects



                              Dollar savings from reduction in acquisition costs
                              Better workstations and productivity tools
 Infrastructure               Movement from central to distributed processing
  Management                  Increased interoperability
                              Increased use of company-wide software/applications




                              Better alignment of IT spending with business priorities/needs
 Leverage Worldwide           Increased IM productivity
    IM Resources              Improved IT resource allocation via a new solutions delivery
                              process




                              IT resources focused on business process needs
 Business Process             Increased speed in delivering new solutions
 Driven Solutions             Tighter IT spending focus
                              Old applications replaced and legacy spending eliminated




         GAO/AIMD-97-163 IT Performance Measurement Guide                                 Page 25
Case Study 2:
Xerox IM2000: Breakthrough Goals and Related Measures




IM2000 Strategies         Breakthrough Goals            Measures Used
                                                         IM spending as % of
Reduce/Redirect IM              Manage IM                revenues reduced
    Spending                    Spending                 % [investment fund/total
                                                         IM spending]



  Infrastructure                                         % [new infrastructure
   Management                                            installed/total eligible
                                   Deliver               sites]
                               Infrastructure            % old infrastructure
                                                         removed
                                  Renewal
Leverage Worldwide                                       % [approved reengineered
                                                         processes deployed/total
   IM Resources                                          business processes]
                                                         % compliance to
                                                         applications retirement
                                                         plan
                             Deliver Process             % reuse code for in-house
  Business Process
                             Driven Solutions            deployed applications
  Driven Solutions                                       % compliance to approved
                                                         solution deployment




        Page 26           GAO/AIMD-97-163 IT Performance Measurement Guide
2. Prepare a Chain of Events and Evidence

A common measurement problem is determining what impact support
functions such as IT have on operational customer or enterprise
outcomes in comparison to other factors. Since final enterprise or
program outcome measures are difficult to measure for a support
function like IT, it is useful to think in terms of a "chain of events and
evidence."

If given careful consideration, a chain of events and evidence can help IT
managers understand how IT resources are being used to support
mission or specific program-related outcomes. In some cases, IT is not
directly related to final program results. However, there are intermediate
outcomes that must occur before the program or business outcome can
be achieved. In contemporary organizations, IT can play a pivotal role in
achieving these intermediate outcomes.

A chain of events is a theory of how inputs in an action chain move to
outputs and finally to end outcomes. A corresponding chain of evidence
shows the evidence and measures that match to the chain of events. The
chains of events and evidence recognize that agency programs set in
motion a sequence of events which are expected to achieve desired
goals. The sequence of events bridges inputs to outputs to outcomes.

The hypothetical example presented in figure 5 illustrates a simple chain
of events, evidence, and measures for an environmental water quality
program. At the beginning point of the program results chain, a series of
simple events are mapped out to indicate steps essential to achieving the
end outcome of a desired level of environmental quality. Certain types of
evidence would then be used as indicators of whether the steps are
occurring. IT provides essential support for several of the steps in the
results chain which can indirectly affect the achievement of the program
outcome. Measuring how well IT is supporting these intermediate steps
is necessary to demonstrate how well it is performing from a mission
perspective.




GAO/AIMD-97-163 IT Performance Measurement Guide                        Page 27
Figure 5: A Hypothetical Chain of Events/Evidence for Achieving
          Environmental Quality




Chain of Events

Companies              Target industry         Industry
participate in water                                                       Pollutants       Quality of the
                       group reached to        discharge
quality                                                                    reduced;         environment
                       reduce discharges       practices changed
environmental                                                              water quality    achieved
                                               to meet new
monitoring program                                                         increased
                                               standards



Chain of Evidence
                        Number of                  Number of                Percent          End outcome:
                        firms                      firms                    pollutants in    improved
                        participating              conforming               water quality    health/
                                                   to standards                              welfare



IT Support Chain                    INTERMEDIATE                OUTCOMES                     END   OUTCOMES


                                              Providing computer           More efficient
                       Timely,                program software                              Effective information
                                                                           and effective    support for mission
                       accurate help desk     applications                 program
                       assistance             meeting program                               accomplishment
                                                                           operations
                                              needs


IT Measures
                           Improvement in             Percent                                      PROGRAM
                                                                            PROGRAM
                          timeliness of              customers                                     OUTCOME
                                                                            PERFORMANCE
                         assistance due             reporting
                                                                                                   MEASURE
                                                   satisfactory use         MEASURE
                        to systems
                        support                    of applications




In an iterative process, organizations examine the chain of events and
evidence for enterprise and operational customer goals and objectives.
The chains explain how an enterprise or operational customer produces
its results and just what those results are. The assumption in using the
chains of events and evidence is that there is a clear understanding of
just how a program, for example, is supposed to work in producing the
expected results.

In turn, a support function such as IT needs to define its own chains of
events and evidence, demonstrating how its products and services affect
enterprise and operational customer chains. For IT, the operational
customer is interested in having IT applications that help achieve
efficient and effective program operations and service delivery. In
essence, final IT outcomes are often efficient and effective program
operations-- actually an operational customer's goal. The organization in


Page 28                             GAO/AIMD-97-163 IT Performance Measurement Guide
large part gauges IT failure or success by how well IT supports the chain
of events the operational customer has in place to achieve its desired
effect.

Building chains of events and evidence in partnership with enterprise and
organizational customers can be a difficult process, but it helps IT
managers understand exactly how IT products and services support
customers and how IT performance should be measured. Using a chain
of events approach also enhances customer understanding of just how IT
can contribute to the eventual enterprise or program outcome.

3. Use a Diversity of Measures to Evaluate IT Performance

An effective IT performance management system should have a diversity
of measures, matched to the right organizational need and level of
decision-making, and action taken on the measure. These measures can
capture performance at the individual, organizational, program, and
process levels. Generically, measures--both qualitative and quantitative--
are often categorized into four main types:

   Input Measures are assessments of the resources used to carry out
   a program or activity over a given time period with the purpose of
   achieving an outcome or output. Input measures can include number
   of IT managers and employees, labor hours, IT funds, computer and
   telecommunications equipment or facilities, or supplies.

   Output Measures are assessments of the actual level of work
   accomplished or services provided over a given time period. They are
   often used to control resources. Number of reports issued, number of
   projects completed, number of answers on hot line services, and
   function points delivered are examples of output measures. These,
   too, are often process measures.

   Outcome Measures assess the actual results, effects, or impacts of a
   program or support function compared to its intended purpose.
   Outcomes can be difficult to measure because results may not be
   immediately evident or several organizational units and external
   suppliers or customers are involved and it is difficult to assign relative
   contributions to them. Outcome measures may be the level of
   customer satisfaction with IT services or cycle time reduction
   attributable to automated work processes.

   Combinations of Single Measures combine single output, outcome,
   and/or input measures into measures designed to demonstrate
   improvements in efficiency or effectiveness. An efficiency measure is
   output over input, such as number of PC applications installed per
   number of IT staff. An effectiveness measure may compare actual

GAO/AIMD-97-163 IT Performance Measurement Guide                        Page 29
    results to estimated or expected results or compare existing levels
    of performance (output) to accepted industry standards or target
    performance goals.

A wide range of measures provide a balance for different decisionmaker
needs. Input and output measures assess workload for an enterprise or
specific program and how much in demand are its products and services.
Combination measures assess efficiency and effectiveness. Outcome
measures assess results compared to expectations. The key point is that
the right measure is used at the right time and for the right reason.
Input and output measures are absolutely vital for measuring how well a
process to deliver IT products and services is performing.

Use Contextual Information to Augment
Performance Measures
The organizations we studied track contextual or explanatory information
to use with their IT performance measures. The contextual or
explanatory information describes the broader environment that
surrounds IT activities which can influence IT inputs, outputs, and
outcomes. For example, measuring changes in mainframe to client
server use, policy changes that impact performance, or changes in IT
organizational structure could be important explanatory information.

Capturing contextual and explanatory information can help managers
understand the measures, assess performance, and evaluate the
significance of underlying factors that may affect reported performance.
Managers can also capture actions that have been taken or are being
taken to in response to reported information, particularly for
unexpectedly high or low performance. Often, IT performance data
displays show the contextual or explanatory information either as a
footnote to the performance data or in an explanatory page attached to
the performance information.




Page 30               GAO/AIMD-97-163 IT Performance Measurement Guide
  How to Get Started
  To proceed with the development and use of an IT results chain
  approach, organizations should:

  • clarify--with top management participation and stakeholder
    involvement--major organizational goals and objectives;

  • establish a simple chain of events and evidence for key mission
    areas of the organization;

  • create supporting IT outcome and process measures for each
    organizational mission goal area;

  • use one of the organizational goals and objectives to develop IT
    goals, specific objectives, and related performance measures
    for that IT goal;

  • examine existing IT measures, categorize them as input, output,
    or outcome measures, and decide on the combination of
    measures that best provides performance results; and

  • test the performance measurement system and make revisions
    based on initial lessons learned.




GAO/AIMD-97-163 IT Performance Measurement Guide                      Page 31
Practice Area 2:                                                     ALIGN

                                                                 Practice Area 1
Follow A "Balanced                                                    Use an
                                                                     IT results
                                                                      chain

Scorecard" Approach                        IMPLEMENT                                         CONSTRUCT
                                                                 Practice Area 4
                                               Practice Area 3       Build data        Practice Area 2
                                               Target measures   collection/analysis      Follow a
                                                     at              capability           balanced
                                               decision-making                            scorecard
                                                   levels        Practice Area 5          approach
                                                                    Improve IT
                                                                    processes

                                                                 REINFORCE




        Practice Area Characteristics

        1. Develop IT goals, objectives, and measures in operational
           and strategic areas
        2. Focus on the most important "vital few" objectives and
           measures in four IT goal areas:
           • Achieving the strategic needs of the enterprise
           • Satisfying the needs of individual customers
           • Fulfilling IT internal business performance
           • Accomplishing IT innovation and learning




      Practice Area Overview
      A second best practice is to use a balanced scorecard approach to IT
      performance measurement. The approach attempts to create a
      measurement balance across the overall performance management
      framework. A balanced approach to measuring the contribution of IT to
      mission outcomes and performance improvement recognizes the broad
      impact of IT's supporting role. By measuring IT performance across four
      goal areas that are critical to overall IT success, the scorecard forces
      managers to consider measurement within the context of the whole
      organization. This limits the possibility of overemphasizing one area of
      measurement at the expense of others. In addition, measuring IT
      performance from different perspectives helps strengthen the analysis of
      intangible and tangible benefits attributable to technology.




      GAO/AIMD-97-163 IT Performance Measurement Guide                                                Page 31
In the four IT goal areas discussed in this section, we present three or four
key objectives that were common among the organizations we examined.
Corresponding to each objective we provide some sample measures that
come from our case study research and from supporting literature. Our
purpose is to illustrate possible types of measures, not to prescribe a
definite list of measures that all organizations should be using. Some of the
measures are very basic, but they are clearly related to the objectives. Also,
many of the measures are percentages or ratios. This is important because
successful organizations begin with good baseline data on performance and,
therefore, can accurately measure progress against the baseline as they
move forward.

In several of the organizations we studied, management is developing
measures across key areas covering both long- and short-term strategies and
activities. This approach is best captured in Robert Kaplan and David
Norton's balanced scorecard approach, which many of the organizations
either used directly or incorporated into the development their own
approaches.11 The Kaplan and Norton scorecard evaluates performance in
four areas: financial (how does the organization look to shareholders?),
customer (how do customers see performance?), internal business (at what
must the organization excel?), and innovative and learning (can the
organization continue to improve and create value?).

In order to summarize the IT performance methods being used by the
organizations we studied, we have adopted a balanced scorecard approach
similar to the Kaplan and Norton framework. However, a balanced
scorecard is just one approach available for agencies to adopt in conducting
IT performance management and measurement. Other approaches such as
the value management framework, critical success factor analysis, and
information economics also offer useful IT performance measurement
methodologies.12 Like other methodologies, a balanced scorecard approach
translates organizational strategy into specific measurable objectives,
operating from several key concepts:

•    no single measure provides clear performance targets or places attention
     on critical mission areas,
•    goal, objective, and measure areas should give a comprehensive view of
     all levels of activities, from the project level to the strategic level,
•    limiting the number of measures used minimizes information overload,
     and

11
  Kaplan and Norton published a series of articles in the Harvard Business Review which explain
the concept and its application. The articles are listed in the bibliography.

12
  A complete description of IT performance measurement approaches can be found in the Society
for Information Management's Practitioner's Guide to I.S. Performance Measurement, referenced in
appendix I.

Page 32                     GAO/AIMD-97-163 IT Performance Measurement Guide
GAO/AIMD-97-163 IT Performance Measurement Guide   Page 33
•   a scorecard guards against optimizing one goal area at the expense of
    others.



Practice Area Characteristics
1. Develop IT Goals, Objectives, and Measures in
   Operational and Strategic Areas

For IT, measures cover a great diversity of value-added activities, including
those for projects, a portfolio of applications, and infrastructure
development. Organizations should know about success in all of them. As
shown in figure 6, an IT results chain can be translated into a scorecard
framework that looks at goals, objectives, measures (tiered for various
decision-making levels), and accountability in key goal areas. The key
starting point in developing a balanced scorecard is the question of purpose
from the IT results chain--"What is the current and future purpose of IT?
Then to meet that purpose, the IT organization must answer the goal
question--"If we succeed, how will we differ?" in terms of specific goals.

Figure 6: An IT Balanced Scorecard Approach


                 Enterprise and Operational Customer
                         Goals and Objectives

                                        IT Purpose
               In meeting the     In meeting the    In internal IT   In the ability to
               strategic needs        needs of        business        innovate and
                    of the           individual    performance?           learn?
                 enterprise?       customers?



Enterprise
level

Program
 level

Project                          The Balanced Scorecard
 level



Page 34                 GAO/AIMD-97-163 IT Performance Measurement Guide
The IT goals and objectives of the organizations we studied most often
focused on the following:

•   customer commitments and satisfaction,
•   cycle and delivery time,
•   quality,
•   cost,
•   financial management,
•   IT infrastructure availability,
•   internal IT operations,
•   IT skill availability, and
•   customer business process support.

For example, Motorola's Semiconductor Products Sector (SPS) focuses its
goals and objectives on four areas: (1) delivering reliable products (quality
environment, best-in-class staff, worldwide resource optimization,
communication), (2) providing integrated IT solutions (integrated data and
systems architecture, technology roadmap and migration planning,
distributed computing effort), (3) building client partnerships (client
involvement, lead the information technology community), and (4) achieving
competitive advantage (prioritize the projects that benefit SPS, deploy
resources for maximum impact, speed of execution).

We developed four balanced scorecard goal areas, objectives, and measures
for IT that were among the most common across the organizations we
studied. As such, the four goal areas illustrate an approximate consolidation
of the performance management efforts of the organizations involved in our
research. The four balanced scorecard goal areas are designed to measure
how well IT is

•   achieving the strategic needs of the enterprise as a whole, in contrast to
    specific individual customers within the enterprise,
•   satisfying the needs of individual customers with IT products and
    services,
•   fulfilling internal IT business performance that delivers IT products and
    services for individual customers and the enterprise, and
•   accomplishing ongoing IT innovation and learning as IT grows and
    develops its skills and IT applications.

The first two goals address whether IT is providing the right products and
services for the enterprise and individual customers. The latter two goal
areas address how well IT is performing in its own capability to deliver
those products and services. The strategic and customer perspectives are
key for linking to mission planning requirements in the Results Act, the
Chief Financial Officers Act, the Paperwork Reduction Act, and the Clinger-
Cohen Act.


GAO/AIMD-97-163 IT Performance Measurement Guide                       Page 35
Managers in our case study organizations emphasized that "balance" does
not necessarily mean "equality." Use of a balanced approach only means the
consideration of several goal areas and the development of objectives and
measures in each. For example, Kodak managers liked the balanced
scorecard approach because it was a multivariable approach. Before using
it, the IT organization was very cost conscious and tended to judge
investments in new applications or skills largely from a cost perspective.
The balanced scorecard examines short-term cost goals and potential
business value in the context of various other nonfinancial operating
parameters.

2. Focus on the "Vital Few" Objectives and Measures

Each leading organization customizes a set of measures appropriate for its
organizational goals and, for IT, how IT fits into the enterprise's strategic
direction and mission delivery plans. The organizations concentrate their IT
performance management efforts on a vital few objectives and measures
within the goal areas. The organizations did not severely limit the number
of measures developed at the beginning. But, over time, and with
experience, the organizations became more focused in the measures they
used. However, use of a balanced scorecard approach gets rid of "safety
net" measures which organizations often collect but do not use for decision-
making, resource allocation, or oversight reporting purposes.

As is explained in the sections that follow, the measure examples illustrate
the need for diversity. Within some of our case study organizations, similar
measures are being used, but the measures remain under development,
requiring more refinement and documentation. The measures presented
here do not represent the full universe of what an organization might use.
Also, in practice, the goal and objective areas may be more specific than
those presented on the following pages. For example, one of our goal areas
centers on the customer perspective. One objective of this goal area is
customer satisfaction. In practice, an actual customer objective statement
might be stated as "This fiscal year, at least 98 percent of customers will be
satisfied with IT products, services, and processes." In short, the following
sections discuss a general categorization of IT goals, objectives, and sample
measures.


  Balanced Scorecard Goal Area 1:
  Achieving the Strategic Needs of the Enterprise




IT strategic measures are designed to evaluate the aggregate impact of IT
investments on the organization. In short, these measures provide insights
into impacts made by the organization's entire portfolio of IT investments.

Page 36                GAO/AIMD-97-163 IT Performance Measurement Guide
This goal area focuses on ways to measure how IT supports the
accomplishment of organizational strategies. The strategic perspective
recognizes that in successful organizations, all components, including IT,
must align with enterprise goals and directions.

When evaluating the impact of IT in terms of strategic needs, the following
questions should be considered:

•   How well integrated are our IT strategies with business needs?
•   How well is the overall portfolio of IT investments being managed?
•   Is IT spending in line with expectations?
•   Are we consistently producing cost-effective results?
•   Are we maximizing the business value and cost effectiveness of IT?

IT managers and staff often attempt to satisfy individual operational
customers without a check against enterprise interests. Having this goal
area prevents targeting IT efforts for individual operational customers which
may be very counter-productive to enterprise IT needs and expectations.
Doing so is difficult, as one manager said, "It has been a cultural plan to
look at [the company] as a whole versus maximizing for individual business
partners. The reward and incentive systems are set up to emphasize that all
senior managers must succeed together."

The four IT strategic enterprise objectives presented in figure 7 reflect
several key objective areas of the organizations we studied. These
objectives cover enterprise strategic planning and goal accomplishment,
enterprise management of the portfolio of IT applications, IT financial and
investment performance, and use of IT resources across the enterprise.

The first objective in this goal area addresses how well IT plans and efforts
reflect enterprise mission goals. This objective area assumes the enterprise
has defined its mission goals and can make the clear link to how IT
supports those goals. The sample measures capture the contribution of IT
solutions and services, compare what was planned for IT benefits and IT
strategies against what actually happened, and compare IT strategies and
planning and enterprise strategies and planning. The overall measurement
thrust is to make sure that enterprise mission goals direct IT activities.

The second objective, portfolio analysis and management, is a growing
concern among the organizations we studied. Leading organizations want to
make sure they have the right portfolio of IT applications either planned or
in place that will enhance business or mission performance.




GAO/AIMD-97-163 IT Performance Measurement Guide                      Page 37
Figure 7: Balanced Scorecard-IT Strategic Measures



          Objectives                        Sample Measures
  Enterprise              percent mission improvements (cost, time, quality, quantity)
                          attributable to IT solutions and services
  mission goals
                          percent planned IT benefits projected v. realized


  Portfolio analysis      percent IT portfolio reviewed and disposed
                          percent old applications retired
  and management          percent applications retirement plan achieved
                          percent reusability of core application modules
                          percent new IT investment v. total IT spending
  Financial and           percent and cost of services provided in-house v. industry
                          standard
  investment
  performance             IT budget as percent of operational budget and compared to
                          industry average

                          net present value, internal rate of return, return on investment,
                          return on net assets


  IT resource usage       percentconsolidated/shared resources across units
                          percentcross-unit shared databases and applications
                          percenthardware/software with interoperability capabilities




Kodak defines an application portfolio as a comprehensive inventory of computer
applications that were developed or purchased to manage an organization's
processes and information. The inventory contains detailed data relative to each
application's size and characteristics, effectiveness in meeting business needs,
potential for growth, and development and maintenance costs. The application
portfolio forms the foundation for an overall IT investment strategy.

Xerox has defined its IT inventory in a similar manner and made IT portfolio
management a key objective area as part of its overall strategic enterprise
strategies. As described in an earlier case study describing its IM2000 strategy,
Xerox wanted to improve information management spending, deliver IT
infrastructure renewal, and deliver process-driven IT solutions to customers. A key
part of the overall IT strategy was to evaluate existing IT applications and
determine how they supported, if at all, the IM2000 strategy. Xerox ran each of
its existing applications through a rigorous analysis process, categorizing each into
one of nine "disposition" categories ranging from stopping those of low usage and
value to keeping others as corporatewide applications.



Page 38                  GAO/AIMD-97-163 IT Performance Measurement Guide
For Xerox, the IT portfolio strategy helps accomplish several performance goals.
The strategy reduces unnecessary operational costs and increases support
productivity; identifies and consolidates similar applications and eliminates low
value applications; identifies and retires legacy applications, data, and infrastructure
as new solutions are deployed; and identifies consolidation and sharing
opportunities. The principle is to view application systems as corporate assets.13

The third objective in this goal area examines financial and investment
performance. While the two objectives above cover mission goals and portfolio
management, this objective addresses management of IT costs and returns. The
sample measures capture costs in major IT financial categories such as hardware
and software. They also provide information on the balance of spending between
legacy and new development applications and between in-house and outsourced
operations. Another sample measure compares the IT budget to the enterprise
operational budget, and how that compares to industry standards.

Sample measures also look at the return on the IT investments, offering several
different methodologies such as rate of return and net present value. Much of this
information is traditionally benchmarked with other IT organizations of similar size
and IT penetration. These measures are tied to customer and enterprise strategic
perspectives to assess if scarce resources are being invested wisely. This is an
especially important area in the federal government with the emphasis on cost
reduction and the best possible use of existing resources.

Lastly, IT resource usage as an objective targets how well the organization can
leverage and share its IT resources across the enterprise. The measures evaluate
factors such as what resources can be shared, what has been consolidated, and
employee access to computing services. From a strategic perspective, this
objective recognizes the need for shared, enterprisewide applications and the use of
an IT infrastructure and architecture for the entire organization.




13
  The Office of Management and Budget and GAO issued a guide, Evaluating Information
Technology Investments: A Practical Guide, in November 1995 to assist federal agency and
oversight staff in evaluating a portfolio of information technology investments in a similar manner.
The approach is also an underlying feature of GAO's guide Assessing Risks and Returns: A Guide
for Evaluating Federal Agencies' IT Investment Decision-making (GAO/AIMD-10.1.13, February
1997).

GAO/AIMD-97-163 IT Performance Measurement Guide                                           Page 39
    Balanced Scorecard Goal Area 2:
    Satisfying the Needs of Individual Customers


IT customer measures are designed to measure the quality and cost effectiveness of
IT products and services. When evaluating the impact of IT on customer
satisfaction, the following questions should be considered:

•   How well are business unit and IT staff integrated into IT systems
    development and acquisition projects?
•   Are customers satisfied with the IT products and services being delivered?
•   Are IT resources being used to support major process improvement
    efforts requiring information management strategies? If so, are the IT projects
    delivering the expected share of process improvement?

The purpose of the second goal area is to meet the needs of individual operational
customers. The three objectives shown in figure 8 capture the key objective areas
we found in our research.

Figure 8: Balanced Scorecard-IT Customer Measures


          Objectives                      Sample Measures

     Customer             percent projects using integrated project teams
     partnership and      percent joint IT customer/supplier service level agreements
     involvement
     Customer             percent customers satisfied with IT product delivery
     satisfaction         percent customers satisfied with IT problem resolution

                          percent customers satisfied with IT maintenance and support

                          percent customers satisfied with IT training

                          percent products launched on time

                          percent service-level agreements met

     Business process     percent IT solutions supporting process improvement projects
     support              percent users covered by training to use new IT solutions

                          percent new users able to use applications unaided after initial
                          training




Page 40                  GAO/AIMD-97-163 IT Performance Measurement Guide
Two of the objective areas, customer satisfaction and business process
support, address direct IT support. Customers were especially interested in
time, cost, quality, overall customer satisfaction, and business process
support. One official we talked to said, "[Our IT organization] looks at the
business process characteristics of our customers. IT personnel ask are
there better ways to support product innovation and development? How
does IT support that? The question is the effectiveness of IT in supporting
business processes--not cranking out function points."14

The first objective area, customer partnership and involvement, stresses a
mutual partnership between the IT organization and customers in developing
the best possible IT products and services. The sample measures examine a
variety of areas, ranging from cooperation and joint development to
involvement in project management.

Customer satisfaction measures assess how well customers are satisfied
with many IT activities. Sample measures also cover the accomplishment of
system design requirements, complaints, problem resolution, error and
defect rates, timeliness, and service-level agreement accomplishments.

The business process support objective area emphasizes the importance of
business process improvement as organizations streamline and reengineer.
Business process improvement is a central objective area for many of the
organizations we studied. The sample measures capture how well IT
supports business process improvement plans and process analysis. They
also examine the adaptability of IT solutions, training for new IT solutions
and the effectiveness of the training, and costs in moving applications to
new hardware.



     Balanced Scorecard Goal Area 3:
     Addressing IT Internal Business Performance


Internal IT business measures are designed to evaluate the operational
effectiveness and efficiency of the IT organization itself. The ability of the
IT shop to deliver quality products and services could have a direct impact
on decisions to outsource IT functions. When evaluating internal IT
business functions, the following questions should be considered:

•     Are quality products delivered within general industry standards?


14
  A function point measures an IT application in terms of the amount of functionality it provides
users. Function points count the information components of an application, such as external inputs
and outputs and external interfaces.

GAO/AIMD-97-163 IT Performance Measurement Guide                                         Page 41
•   Are quality products being delivered using accepted methods and tools?
•   Is our infrastructure providing reliable support for business needs?
•   Is the enterprise architecture being maintained and sustained?

One manager we interviewed said, "There are two dimensions of [IT]
performance. One is the dominant or visible component--the use of IT in the
context of [customer] business processes. The other is transparent--the
functional excellence of IT." The first two goal areas stress the use of IT as
it supports enterprise and operational customers. On a day-to-day basis, it is
the functional excellence of IT internal business processes which delivers
that support. Figure 9 shows four objective areas and sample measures we
synthesized from our case study organizations and the general IT literature.


Figure 9: Balanced Scorecard-IT Internal Business Measures


     Objectives                        Sample Measures

Applications           number of function points delivered per labor hour
                       number of defects per 100 function points at user acceptance
development and        number of critical defects per 100 function points in production
maintenance            percent decrease in application software failures, problems
                       mean time to resolve critical defects
                       cycle time for development

Project                percent projects on time, on budget
                       percent projects meeting functionality requirements
performance            percent projects using standard methodology for systems
                       analysis and design



Infrastructure         percent computer availability
                       percent communications availability
availability           percent applications availability
                       on-line system availability

Enterprise             number of variations from standards detected by review and
                       audit per year
architecture           percent increase in systems using architecture
standards              percent staff trained in relevant standards
compliance



IT managers and staff, along with enterprise senior management, decide
which of the many IT processes truly must excel for meeting customer and
enterprise goals in the short and long term. For example, is the IT process
for identifying the right technology for customer applications the best it can
be? IT managers and staff set specific goals for improvement of internal IT
business processes.

Page 42                GAO/AIMD-97-163 IT Performance Measurement Guide
The first objective covers IT's performance in developing and maintaining
applications. The sample measures include dollars expended per function
point, average application development cycle time, and cost. The second
objective area examines the performance in delivering projects, capturing
traditional measurements on project time, budget, functionality, and use of
widely accepted methods and tools. Measures also capture backlogs in both
development and enhancement or maintenance of applications.

The third objective area addresses IT infrastructure availability in a variety
of areas, as well as response time and transactions. Many of the
organizations we studied stressed the importance of infrastructure
availability, an area totally transparent to the customer until something goes
wrong. These measures keep managers on top of infrastructure
performance where there is little tolerance for down time. The last
objective area covers architectural standards. 15 The measures assess how
well IT is meeting set standards, most often developed for interconnectivity
and interoperability and efficient IT support.

Many of the traditional IT measures fall into the internal business
performance goal area, often focusing on the efficiency of computing and
communications hardware and software. The measures in this goal area
frequently are used for individual manager and staff IT accountability, as
described in a later practice.

Some of the organizations we studied were using the Software Engineering
Institute's five-level capability maturity model to guide their IT process
improvement efforts. The objective areas and measures are, in contrast to
some of the other balanced scorecard areas, highly integrated. For example,
project performance relies on effective applications development and
maintenance.



     Balanced Scorecard Goal Area 4:
     Addressing Innovation and Learning


Innovation and learning measures evaluate the IT organization's skill levels and
capacity to consistently deliver quality results. This goal area recognizes that
without the right people with the right skills using the right methodologies, IT

15
  IT architectures explicitly define common standards and rules for both data technology, as well as
mapping key processes and information flows. A complete IT architecture should consist of both
logical and technical components. The logical architecture provides the high-level description of
the organization's mission, functional requirements, information requirements, systems components,
and information flows among the components. The technical architecture defines the specific IT
standards and rules that will be used to implement the logical architecture.

GAO/AIMD-97-163 IT Performance Measurement Guide                                          Page 43
performance will surely suffer. Measures in this goal area should be used to
answer the following questions:

•   Do we have the right skills and qualified staff to ensure quality results?
•   Are we tracking the development of new technology important to our
    business/mission needs?
•   Are we using recognized approaches and methods for building and managing
    IT projects?
•   Are we providing our staff the proper tools, training, and incentives to perform
    their tasks?

The four objective areas shown in figure 10 include workforce competency and
development, advanced technology use, methodology currency, and employee
satisfaction and retention.

Figure 10:     Balanced Scorecard-IT Innovation and Learning
               Measures


      Objectives                         Sample Measures

Workforce                percent staff trained in use of new technologies and techniques
                         percent staff professionally certified
competency and           percent IT management staff trained in management skills
development              percent IT budget devoted to training and staff development



Advanced                 percent employees skilled in advanced technology applications
technology use           number of dollars available to support advanced technology
                         skill development


Methodology              currency of application development methods in use
                         percent employees skilled in advanced application development
currency                 methods
                         percent projects developed using recognized methods and tools


Employee                 percent employee satisfaction with the capability of the existing
                         technical and operating environment to support mission
satisfaction and
retention                percent employee turnover by function




This goal area develops the continuous improvement aspect of IT activities. It
speaks to capabilities of bringing new technologies to bear on customer problems,
practicing the best methodologies, and retaining and developing the best
employees. The first objective area stresses the importance of having a capable

Page 44                  GAO/AIMD-97-163 IT Performance Measurement Guide
and competent workforce. In particular, the organizations we studied were very
concerned with workforce competence and development. Key measures included
training hours and skill development. Most were transitioning from core
competencies in operations and maintenance to business process improvement and
reengineering, new business solutions, and technical direction of applications
development done by others.

The second and third objectives, advanced technology use and methodology
currency, speak to the ability to recognize and deploy advanced technologies and
methodologies in doing IT's work. The last objective, employee satisfaction and
retention, measures how well employees themselves are satisfied with the quality
of their work environment and general IT strategies and accomplishments.




  How to Get Started
      To begin developing a balanced scorecard approach for IT,
      organizations should:

      •   get agreement among business and IT management on the
          approach that will be used for developing IT-related
          performance indicators and measures,

      •   using the agreed upon approach, define and develop the key
          goal areas and objectives for the IT organization,

      •   develop a full set of measures in one or two priority IT goal
          areas, then expand out to other goal areas, and

      •   test the performance measurement system and make revisions
          based upon initial lessons learned.




GAO/AIMD-97-163 IT Performance Measurement Guide                           Page 45
Practice Area 3:                                                      ALIGN


Target Measures,                                                  Practice Area 1
                                                                       Use an
                                                                      IT results

Results, and                                IMPLEMENT
                                                                       chain

                                                                                              CONSTRUCT

Accountability at                               Practice Area 3
                                                Target measures
                                                                  Practice Area 4
                                                                      Build data
                                                                  collection/analysis
                                                                                        Practice Area 2
                                                                                           Follow a

Decision-making Tiers                                 at
                                                decision-making
                                                    levels
                                                                      capability

                                                                  Practice Area 5
                                                                                           balanced
                                                                                           scorecard
                                                                                           approach
                                                                     Improve IT
                                                                     processes

                                                                  REINFORCE




        Practice Area Characteristics:

           1. Track IT measures and appropriate reports to each decision-making
              level
           2. Align measures from the bottom to the top
           3. Directly link tiered measures to the balanced scorecard
           4. Align individual accountability to IT scorecard goals




      Practice Area Overview
      Organizations in our study targeted measures and performance reports at
      specific decision-making levels or tiers. These tiers cover the enterprise
      (agency) level, senior to mid-level management (program) level, and
      specific operations or project level. This approach offers several
      advantages, including (1) enhanced communication and understanding of
      performance measurement throughout the organization, (2) systematic
      links among measures and enterprise, program, project, and individual
      objectives, and (3) alignment of measures with mission results. IT
      performance information should drive management actions and decisions
      that support the attainment of organizational goals.

      Practice Area Characteristics
      1. Track IT Measures and Reports to Decision-making Levels

      As shown in figure 11, performance measures and reports at each tier
      have specific purposes. At the enterprise tier, IT performance and
      measures focus on mission results, or how well IT is meeting its purpose
      in supporting enterprisewide goals and objectives. Information on final
      and intermediate outcomes of programs facilitated by IT projects and
      investments would be shown. A summary report may be prepared for

      GAO/AIMD-97-163 IT Performance Measurement Guide                                             Page 45
external reporting to stakeholders and the general public. Reports may
be prepared on an annual or quarterly basis and highlight IT policy-
oriented information showing areas of progress, problems, and contextual
or explanatory information to supplement the performance data.


Figure 11: Performance Measurement Tiers
At the IT program or support unit level, senior to mid-level managers




                                                                              Measurement focus and
                                                                                     timing
                            ENTERPRISE
                                                                           Cyclical (e.g., annual or quarterly)
                                                                           information that focuses on mission
                               Executive
                                                                           results used for policy decisions and
                              Information                                  strategies
                                              inc
                                               Me ases
                            Mission Results
                                                  re
                                                   asu fro
                      m
           to t n fro




                                                      rem m t
                             PROGRAM                                       Periodic (e.g., quarterly or monthly)
               op
       tom lig




                                                                           information that focuses on unit results
                                                         ent op t
    bot ures a




                                                                           associated with management and
                             Management                     gra o bo
                                                                           operational improvements
                             Information
                                                               nul tto
    eas




                             Unit Results
                                                                  arit m
IT M




                                                                      y



                             PROJECT                                       Immediate (e.g., daily or weekly)
                                                                           information that focuses on activity
                            Activity/Task                                  and task level data used to make
                            Information                                    tactical decisions and execute
                          Workplace Results                                management decisions




want to know how specific units or processes are performing.
Measurement most often covers specific IT business processes such as
applications development or lines of business (or core business areas).
At this tier, more detailed performance information is used for
management and the improvement of operations and integrating activities
across IT processes or programs.

In the third tier, or bottom level, the measurement emphasis is generally
at the project level and individual systems performance. Highly detailed
tactical and execution information supports immediate and day to day
decision-making on funding, contract development and monitoring,


Page 46                              GAO/AIMD-97-163 IT Performance Measurement Guide
            project priorities, and possible adjustments in program operating
            procedures. Here the emphasis is on input and output measures.

            Tiering in this manner can help assign performance accountability and
            determine where IT measurement ownership lies. The organizations
            decide where pieces of IT performance accountability rest, such as user
            or operational departments, central IT organizations, and/or department
            or operational IT units. In decentralized organizations, many people are
            involved in the IT processes that deliver products and services.


Case Study 3:
Using Performance Measures for Different Management Tiers

Xerox information management operational measures are targeted at three distinct
organizational tiers. As shown in the following figure, Tier 1 consists of corporatewide
metrics, targeted for senior managers and executives. Tier 2 focuses on measures used by
specific organizations, divisions, and core business areas. These measures assess what is
happening below the enterprise level and actually roll up Tier 3 metrics collected at the
project level.




 Case Study 3:
 Xerox Tiered Information Management Matrix

               Tier 1: Global Information Management
            IT spending as % of revenues reduced (quarterly)
            Total IT spending as % of IT investment funding (quarterly)
            Customer satisfaction scores (annually)                          Consolidate
                                                                             and analyze

                                                                             Identify
            Tier 2: Organizations, Divisions, Core Areas                     trends
           % projects on time (actual versus planned)
           costs/benefits for IT investments (projects and overall)          Communicate
           overall IT spending profiles (all IT investments)                 results

                                                                             Benchmark

          Tier 3: Project Level (contractor performance)
            $ per function point delivered    % projects on time
            % software delivery re-use        productivity (function point
            function point defects            per month)




            GAO/AIMD-97-163 IT Performance Measurement Guide                                Page 47
Case Study 4:
Decision-making Levels Use Different Types of Measures

American Express' Technologies group is also developing a tiered measurement process for
decision-making at all management levels. Tier 1 measures consist of executive
information that represents the Technologies group's overall effectiveness against such goals
as (1) achieving world class time-to-market, (2) developing new business, and (3)
enabling business partners. Specific measures include development cycle time, time
requirements for ongoing operations, cost of ongoing operations, cost of quality, leading
quality indicators, and elimination of root causes of failures.

Tier 2 measures consist of management information for senior and mid-level managers
and have direct links to tier 1 measures. To illustrate, tier 2 measures supporting the tier
1 measure of development cycle time include "elapsed time per function point" and "effort
hours per function point."

Tier 3 measures are operational information used for project development, operations, and
project leaders. Tier 3 information links directly to tier 2 and forms the basis for
management decisions, root cause analysis, and continuous process improvement
evaluations. Specific performance measures are used to evaluate individual projects and
applications.




            2. Align Measures From the Bottom to the Top of the
               Organization

            A key performance feature found in the organizations we studied is the
            notion of aligning--but not necessarily "rolling-up"--measures from the
            bottom to the top of the organization. IT measures used at the lowest
            tier--the specific operations or project level--must align upwards with the
            subsequent tiers. In other words, the IT input and output information
            collected for tactical and execution management must directly relate to
            unit results needs and then upwards to the enterprise level. This
            alignment helps to ensure that performance measures and information at
            the lowest level directly support policy and mission decisions and
            strategies.

            Only rarely will an organization have a single performance measure
            appropriate for all three levels, or, in other words, those that can "roll-up"
            to the top of the pyramid. A temptation in performance measures is to
            layer measures from lower levels on top of each other and pass the
            information along to higher-level officials. This approach may provide an
            overload of information not easily understandable or digestible by top
            executives or even the public. It also creates the potential of hiding
            embarrassing details in a mountain of data, or can promote the self-
            selection of favorable data. A few of the bottom tier measures may "roll

            Page 48                        GAO/AIMD-97-163 IT Performance Measurement Guide
up" into strategic measures of interest at the other two tiers. However,
the type and formatting of IT information and measures and timing of
performance reporting appropriate at one tier may not be appropriate for
others. Use of all three tiers gives the organization a comprehensive
picture of the value of IT and if the individual IT products, services, and
processes were worth the investment.

3. Directly Link Tiered Measures to a Balanced Scorecard

Once the IT organization has agreement on balanced scorecard goals and
objectives, then it would develop limited tiered measures to address
specific operational or project measures, program and support unit
measures, and enterprise-level measures. The tiering of measures across
the balanced scorecard area facilitates the use of measures for decision-
making and performance improvement action. Otherwise, if measures
are not used, they lose their decision-making impact, which results in less
effort to collect and use them and, in turn, leads to less decision-making
impact.

One tier is not more important than the other two. As one manager
noted, an operational customer will never ask for reports containing
performance data found in the bottom tiers, such as mainframe
availability or application availability. But this operational data provides
vital information about how well IT is supporting program operations and
is indirectly linked to customer satisfaction measures.

The use of performance measurement tiers is not a novel concept, but
reflects a change in how measurement information is used by an
organization. Traditional IT performance measures are used to examine
lines of code generated, number of reports issued, data center downtime,
transactions, and the number of projects on time. These can be
considered bottom tier measures. More recently, management emphasis
has shifted towards performance-oriented customer requirements,
productivity and quality improvements, selection of strategic projects,
and weighing value of delivered systems.

Using some specific objectives under the balanced scorecard goal areas
discussed in Practice 2, figure 12 provides some hypothetical examples of
IT measures that might be used at different organizational tiers. A
combination of input, output, and outcome measures are sprinkled
throughout the tiers to accommodate different management information
decision-making and reporting needs.




GAO/AIMD-97-163 IT Performance Measurement Guide                       Page 49
  Figure 12: A Tiered Performance Scorecard Example




      Scorecard                               Tiered Measures
                              Specific                                Enterprise
                                                  Program/
                              Operational
  Goals      Objectives       or Project
                                                  Support Unit        Level
                                                                      Measures
                                                  Measures
                              Measures

  Strategic   Portfolio                   % compliance to            % IT portfolio
                         % reusability of  approved IT
 Enterprise analysis and core modules                                reviewed and
                                             solution                  disposed
 Perspective management
                                                  deployment
   Customer                  # complaints to     % customers            % planned
               Customer
                                help desk       satisfied with IT    system design
  Perspective satisfaction
                                                  application         requirements
                                                     design            realized vs.
                                                                       deliverables
   Internal  Applications    # defects per       % decrease in
                                                   application      % projects on
  Business development        100 function
                                                                  time, on budget
 Perspective    and          points at user     software failures
              maintenance     acceptance

 Innovation & Workforce                             % staff            % IT budget
                          # staff by skill
   Learning competency &                         professionally         devoted to
                               area
  Perspective development                           certified       training and staff
                                                                      development




4. Align Individual Accountability to IT Goals

The last question that ties together the IT balanced scorecard and the IT
results chain is "Who is accountable for results and how are they held
accountable?" The leading organizations have learned that managing
performance well depends on making the connection between program
and IT purpose, goals, and objectives, and responsible teams and
individual staff.

Alignment begins when organizational and program goals and objectives
are translated into action plans for the improvement of IT products,
services, systems, processes, and practices. Just as measure
development involves many staff, IT performance plans and targets must
be communicated to all levels of the organization. Smaller components
develop goals and objectives that support the higher level. In turn,
individual accountability is based on the actions that each individual can
take to contribute to the organization's goals. For example, the system
of accountability must recognize that the final outcomes of IT support

Page 50                      GAO/AIMD-97-163 IT Performance Measurement Guide
            activities are usually intermediate outcomes for the higher level program
            goals and targets.

            Actual levels of performance should be tracked against targets, and both
            IT and program managers should be held accountable for the results.
            Measure and results reviews should report on the actual outcomes and
            individual performance appraisals should link IT performance to merit
            pay, bonuses, promotional opportunities, and other incentives.



Case Study 5:
Aligning Individual Performance with Organizational Goals

Texas Instruments Information Systems and Services (IS&S) business excellence improvement process
has many events involving measures and results. Operational assessments include quarterly metrics
and project reviews. Information sharing events include a presentation of annual results and goals in
January and quarterly communication meetings and results reports. The assessments focus on
sharing lessons learned, identifying opportunities to transfer best practices, uncovering potential risks,
and assigning corrective action. IS&S holds quarterly communication meetings open to all IS&S staff
to discuss current topics and answer general questions. IS&S leaders use quarterly department
meetings to further disseminate information and plans. Several electronic information sharing
sources are available to IS&S staff.

IS&S uses a performance evaluation and development process to allow IS&S personnel the
opportunity to detail their previous year's accomplishments and identify short and long-term job and
career goals. The employee uses the process to align his/her goals to those of the organization. The
process is key to promotions, bonuses, and selection for a technical ladder--a designation recognizing
technical staff for outstanding technical contributions and quality leadership. IS&S also uses
recognition actions to support quality and performance improvement. For example, recognition
display boards are located in highly visible areas. These display individual and team pictures of
IS&S personnel honored for technical and quality contributions, as well as for involvement in team
problem solving activities and community projects.

Each year, IS&S leadership sends a memorandum to Texas Instruments' top management on policy
deployment and key performance measures. Policy deployment is Texas Instruments' term for
aligning the individual to overall corporate business objectives and for continuous or dramatic
improvement of key performance measures.

The memorandum's performance areas match the Texas Instruments strategic directions described in
an earlier chapter--customer satisfaction, continuous improvement, people involvement, and cycle time
improvement. Texas Instruments chose cycle time as a measure to drive dramatic improvement in a
specific IT process (solutions provisioning). Improving this one process as a starting point addresses
IT structural problems that affect cycle time. The IS&S leadership team has a business excellence
improvement plan. This plan leads to action plans for each IS&S division supporting the strategic
business units and other key organizational units, to team action plans to individual contributions.




            Although this approach is controversial in the pubic sector, leading
            private sector organizations have found that failure to translate
            organizational strategy into program, team, and individual IT goals can

            GAO/AIMD-97-163 IT Performance Measurement Guide                                    Page 51
result in a focus on short-term and tactical issues rather than the
accomplishment of strategic goals.




   How to Get Started
   To begin targeting measures, results, and accountability at
   different decision-making levels, organizations should:

   •   identify enterprise tier IT performance information
       requirements,

   •   use the enterprise tier information to develop measures across
       one balanced scorecard area,

   •   begin adjusting management performance review processes at
       the unit, team, and individual level to reflect the tiered
       approach, and

   •   test the performance measurement system and make revisions
       based on initial implementation efforts.




Page 52                    GAO/AIMD-97-163 IT Performance Measurement Guide
Practice Area 4:                                                         ALIGN


                                                                    Practice Area 1
Build a Comprehensive                                                    Use an
                                                                        IT results
                                                                         chain

Measurement, Data                             IMPLEMENT                                         CONSTRUCT
                                                                    Practice Area 4

Collection,                                       Practice Area 3
                                                  Target measures
                                                        at
                                                                        Build data
                                                                    collection/analysis
                                                                        capability
                                                                                          Practice Area 2
                                                                                             Follow a
                                                                                             balanced

and Analysis Capability                           decision-making
                                                      levels        Practice Area 5
                                                                       Improve IT
                                                                       processes
                                                                                             scorecard
                                                                                             approach




                                                                     REINFORCE




       Practice Area Characteristics:

          1.   Use data collection tools
          2.   Develop and use baseline and benchmark information
          3.   Assess maturity and develop complete performance
               definitions
          4.   Utilize concise, understandable performance reporting
          5.   Conduct measure reviews and audits



      Practice Area Overview

      Building a balanced scorecard and tiered measures is only one step in
      designing an effective IT performance management system. In the
      organizations we studied, management paid careful attention to the "back
      end" of performance management: data collection and analysis. As one
      manager explained, "You need a lot of analysis to figure out the key
      drivers for performance, like what will it take to raise customer
      satisfaction one or more percentage points. And is that goal good for
      other measures such as cost and time? What is the gain for the effort we
      must put in and is it worth it?"

      Most organizations began by benchmarking existing performance against
      different IT units within the organization, external IT organizations in
      other businesses, or industry benchmarks published by research
      organizations. Next, performance definitions were agreed upon and
      existing data used to "baseline" existing performance and identify
      information gaps that needed to be addressed with additional data
      collection and analysis.

      Performance data are needed at all tier levels. Even so, the collection
      and reporting of this information should not impose an unnecessary

      GAO/AIMD-97-163 IT Performance Measurement Guide                                             Page 53
            burden on management and staff. Data collection should utilize efficient
            manual or automated methods. The organizations we studied developed
            a clear rationale for new and continued data collection and specifications
            for accuracy, reliability, timeliness, and use before setting out
            measurement reporting requirements. Most organizations we examined
            designed efficient and effective ways to present the performance
            information to management so that it could facilitate better decision-
            making. Finally, the organizations regularly conducted reviews of their
            performance measurement systems and revised or updated measures in
            accordance with management feedback or changing business needs.

            Practice Area Characteristics

            1. Use Data Collection Tools

            For each data collection requirement--whether qualitative or quantitative--
            most of our case study organizations developed manual and automated
            tools to reduce the burden of collecting IT performance information.
            These included personal observation, formal performance measure
            reports, customer satisfaction surveys and interview questions, reviews
            of records and documents, and automated hardware and software
            productivity data collection tools.




Case Study 6:
Collecting IT Performance Information Using Multiple Techniques
Across Different Performance Dimensions

The Immigration and Naturalization Service's Office of Information Resources Management
(OIRM) collects information on both its Information Technology Partnership (ITP) contract,
the primary IT support services contract, and for program-level performance regarding
mission outcomes. As shown in the following figure, the ITP contract tasks measure within
budget, on-time, customer satisfaction, and technical (such as quality of project
completeness, use of resources, and completeness and quality of documentation) performance.
The program-level performance measures cover cost, efficiencies, and quality in the
improvements realized.

The program level covers all major operational areas, such as the Border Patrol. INS has
specifically designated tools for its data collection under the Information Technology
Partnership mentioned earlier. These tools include a performance measures report, a
customer satisfaction survey, and a technical performance evaluation form. For mission-
critical tasks, quantitative methods, ratios, and performance reports are the tools that are
used for data collection and analysis.




            Page 54                        GAO/AIMD-97-163 IT Performance Measurement Guide
Case Study 6:
INS ITP Performance Measurement Plan


                                                                            All ITP Contract Tasks
           The Vision                                                    Within budget                   Performance measures
                                                                                                         report
  The INS seeks a contractor to perform as




                                                 Performance Measures
                                                                         On-time
  a full partner with INS in conceptualizing,                                                       T    Performance measures
  planning, developing, implementing, and                                Customer satisfaction           report




                                                      ITP Contract
                                                                                                   O
  maintaining advanced, cost-effective, state-
                                                                         Technical performance           ITP customer satisfaction
  of-the-art technological solutions to the                                                        O     survey
  business processes of the INS. The goal
                                                                                                    L
  of the partnership is optimal achievement                                                              ITP technical performance
  of the INS mission. In addition, the INS                                                          S    evaluation form
  technology partner will provide visionary,
                                                                                                         ITP self assessment report
  leading-edge support while participating
  in the planning to meet the INS information
  technology needs of the 21st century.
                                                                        High Impact/Mission Critical Tasks
                                                                         Cost (improvements             Quantitative methods
                                                 Performance Measures




            Methodology                                                  realized)                 T    **Return on Investment
                                                                                                        **Net Present Value
                                                     Program-Level




    Identify customer needs                                              Efficiencies (improvements O
                                                                         realized)                      Process improvement ratios
                                                                                                   O
    Document current operating baseline                                                                 Service-level Agreements
                                                                         Quality (improvements     L
                                                                         realized)
    Set goals for business process                                                                      Performance measures
                                                                                                   S    report
    improvements

    Develop service-level agreement
                                                                                               ITP
    Monitor key indicators
                                                                                      "Return on Investment"




             Most of the organizations we studied feature customer surveys,
             interviews, and focus groups as important sources of performance
             information. These data collection exercises are well-designed, tailored
             to IT performance measurement ownership, and fit into management
             processes to take action on the results. For example, Motorola uses a
             companywide survey that asks for ratings ranging from "superior" to
             "poor" on such issues as

             •     availability of Information Systems (IS) personnel,
             •     responsiveness of IS personnel,
             •     effectiveness of IS in project management communication,
             •     reliability of IS in meeting commitments,


             GAO/AIMD-97-163 IT Performance Measurement Guide                                                                         Page 55
•   IS cycle time for completing projects,
•   quality of IS work,
•   rate of improvement during the past year,
•   alignment of IS resources with business needs,
•   overall credibility of information systems support, and
•   IS performance with respect to total customer satisfaction.

In all cases, the organizations started with what they considered a
primitive data collection strategy, beginning with data definitions and
working with customers and IT staff to gain understanding and
agreement. In the early stages, some of the data collection may all be
manual, later supplemented or replaced by automated systems. All of
the organizations cautioned that IT should never wait for automated data
collection systems, but start small and with measures meaningful to
enterprise and operational customers, such as IT cycle time and
customer satisfaction.




Page 56                    GAO/AIMD-97-163 IT Performance Measurement Guide
Case Study 7:
Using Multiple Data Collection Mechanisms and Ensuring System
Integrity

Texas Instruments' Information Systems & Services (IS&S) gathers, prioritizes, and
communicates customer and supplier concerns through a series of teaming activities. IS&S
conducts annual strategic intent conferences and objective reviews with top-level customer
management. Several times a year senior customer managers on an Internal Information
Systems Leadership Team meet with IS&S managers to offer strategic planning and
guidance. Other activities include ongoing executive one-on-one interviews and
management-level and project-level customer surveys, employee development, reengineering
efforts, customer focus workshops, steering teams, quality improvement teams, leadership
teams, and quality steering teams. Senior IS&S management uses these opportunities to
confer with the customer on business priorities, critical success factors, and operational
plans. A service-focused survey is sent to end users of IS&S products and services. Each
Texas Instruments organization has a metrics coordinator who handles the manual and
automated databases.

Much of Texas Instruments' IS&S performance information has been tracked for at least 5
years, with the majority of the information available online. A combination of software
error checking, password control, and independent audits ensures the reliability,
accessibility, and integrity of the performance databases. IS&S promotes system
consistency and maximum data sharing through standardized data formats and system
interfaces. IS&S uses an electronic data interchange network to communicate electronically
with suppliers and customers. An online complaint system assists customers in registering
complaints about IS&S products and services.




        While there is widespread use of annual customer surveys, most
        organizations note that they are very limited in the information they
        provide that can tie to corrective action strategies. One manager said his
        organization was putting customer surveys on hold until the organization
        could develop questions which would provide answers for concrete
        corrective action. Another manager said he planned to do "just in time"
        surveys on specific projects or IT processes instead of a survey at the
        end of the year. His rationale was that year end surveys only capture
        problems or feelings, while he wanted to capture data that the IT
        organization could take action on, that is linked to organizational
        activities, and where he can see trends in particular areas and focus on
        skills and problem areas. His point was that asking general questions is
        no way to develop actionability or determine significant trends.

        Managers in the organizations we studies emphasized that it is important
        to have consistency of some hard data collection from year to year.
        Some performance information such as computer and communications
        availability, customer satisfaction percentages, and software capability

        GAO/AIMD-97-163 IT Performance Measurement Guide                                Page 57
maturity levels are relatively durable and comparable over longer time
periods. These measures track trends in areas where products and
services do not change significantly. One manager suggested that
measures should not be policy-based as policy can change from year to
year.

2. Develop and Use Baseline and Benchmark Information

The organizations we studied spent considerable time and effort on
baselining and benchmarking, two entirely different activities. They
assessed what performance information they had for the measures they
had selected (baselining) and how that information might compare to
that of other organizations or similar processes within their organization
if there were discrete IT units (benchmarking).

In baselining, available IT performance information, or information that
will have to be collected, becomes part of the performance baseline for
each scorecard objective. The current performance becomes the
"baseline" against which further performance is measured. Without
baselining, there is no standard to measure progress. In fact, one of the
initial tasks in IT performance management is determining current
performance using the measures designed for the balanced scorecard or
a similar approach. The organizations we studied found that
performance data within a balanced scorecard cannot be a set of data on
top of all the IT performance data that was previously collected.

To set actual IT performance targets, organizations often do
benchmarking, an activity that is much different from baselining.
Benchmarking was done with other IT organizations in the enterprise,
with IT organizations outside the enterprise, or with similar processes
but in other industries. For example, Kodak benchmarks with companies
in markets where it competes (competitive benchmarking), with leading
or best of class organizations within any industry (functional
benchmarking), and among the operating units within Kodak (internal
benchmarking). For example, handling customer support phone lines
could be benchmarked against mail order telephone operations.

Another organization we studied, Xerox Corporation, was outsourcing
many of its IT operations to Electronic Data Systems (EDS). Xerox is
benchmarking EDS service content, service delivery, and pricing against
the best in many countries. Initially, Xerox established a price and
service baseline for comparison of EDS services and prices against the
best organizations.

Many of the organizations we studied have made benchmarking an
integral part of their IT performance management activities.
Benchmarking information is often required in decision-making packages

Page 58                   GAO/AIMD-97-163 IT Performance Measurement Guide
sent forward to senior executives. Most have devoted at least one staff
member to benchmarking. Texas Instruments, for example, has a
corporate office of benchmarking and best practice sharing.

Some organizations we studied cautioned that benchmarking requires
focus. Benchmarking often requires looking at process improvements
versus strategic value. While an organization will know how an
organization does a particular IT process or set of activities, that
knowledge may do little to improve outcomes. If the benchmarking
focus is on key measures, such as reducing cycle time or how well IT
business process support customers, then benchmarking realizes a
strategic potential. The organizations use baselining and benchmarking
information to identify performance gaps between current IT
performance and desired achievement levels. Leading organizations
recognize that improvement goals must flow from a fact-based analysis
of IT performance aligned to organization mission. At least one of the
organizations we studied believed that goals, such as six sigma, make
benchmarking irrelevant.16 The standard for performance is already set
at a "zero defect" goal. One manager noted that "the key to performance
is how the [IT] organization supports the business, not how the IT
organization compares to other IT organizations." However, others
believed that benchmarking provides comparison data on exemplary
organizations and sets "stretch" performance standards for the IT
organization.

3. Assess Performance Maturity and Develop Complete
   Performance Definitions

A common stumbling block for organizations is the tendency to struggle
to develop "perfect" measures instead of thinking in terms of improving
measures over time. But as one manager told us, in performance
measurement, you simply cannot be good at everything right away which
forces you to undertake a phased approach. As an organization moves
gains more performance management experience, better and more
appropriate goals will be defined so the supporting measures will, in
turn, be modified. For many measures, the definitions, data collection
techniques, and reporting will need to be refined over time.

Measure maturity assessment can be a part of the measure definitions
that organizations develop. Generally, these definitions cover what the
measure is intended to show and why it is important, how performance
data are generated, who is responsible for collecting it, how the measure

16
 Six sigma is a measure defining the quality level of a product, service, or process. This
organization defined six sigma is 99.9997 percent perfect or 3.4 defects per million
opportunities to create a defect. If set as a goal, it basically calls for a virtual "zero defect"
standard.

GAO/AIMD-97-163 IT Performance Measurement Guide                                          Page 59
            is specifically calculated, any limitations placed on the measurement data
            (for example, factors beyond the organization's control), and whether the
            data is cumulative or noncumulative. As this process is repeated and
            refined over time, there is increased confidence that the measure
            accurately describes performance in the relevant area.




Case Study 8:
Gaining Experience with Fundamental Measures and then Expanding Out

Kodak is one organization that is systematically defining the maturity of each measure it
plans to use in its balanced scorecard. Kodak categorizes measure maturity as either
fundamental, growing, or maturing. Established indicators are considered as fundamental.
Growing measures are evolving from the fundamental, but are not the best they can be.
Maturing measures are defined as best-in-class for whatever they are measuring. For example,
for internal performance a fundamental measure is to meet all service-level agreements, a
growing measure is information delivery excellence, and a maturing measure is defect-free
products and services. Kodak believes it is important to build the right fundamental practices
first in developing an initial IT performance management system.

As part of its development of measures for EDS services, Xerox is starting with what it calls
"primitive metrics." These include size, measured in function points; effort, measured in work
hours; defects, measured by number of defects; changes, measured by number of changes; and
duration, measured by elapsed days. Over time, quarterly performance reviews and
examination of measures are expected to result in revisions to these measures.




            4.   Utilize Concise, Understandable Performance Reporting

            Leading organizations take great care in designing IT performance
            reports that are concise, easy to understand, and tailored to various
            management needs and audiences. Executive managers, in particular,
            often require data presentations and displays that focus on bottom line
            performance results. When presented in this manner, they can quickly
            digest information, focus on problem areas, seek pertinent follow-up
            data, and be more efficient in making or recommending project or
            program decisions.

            Performance reports should be tailored for the audience for which the
            information is intended. Operational managers may need more details
            and supporting, contextual information while external stakeholders may
            require far less.




            Page 60                       GAO/AIMD-97-163 IT Performance Measurement Guide
Case Study 9:
Presenting Performance Data in Usable Formats

The Immigration and Naturalization Service's Office of Information Resources Management
(OIRM) collects information on both its Information Technology Partnership contract and on
program-level performance regarding mission outcomes. The contract is run by INS in
partnership with Electronic Data Systems (EDS).

The following figure shows how overall contract-level performance is reported on a quarterly
basis for each of the four key critical success factors: (1) customer satisfaction, (2) technical
performance, (3)on time, and (4) within budget. These quarterly measures can then be plotted
to show a performance trend over the life of the contract. Gathering data such as this will also
allow INS to provide documented and objective feedback on a contractor’s past performance.




 Case Study 9:
 INS/ITP Quarterly Performance Reporting




   Percent                                            1st QTR.     Score     Weight   Weighted
                                                   Performance                         Score
   120                                               Measures
                                                  Customer              75      .30       22.5
   100                         89                 Satisfaction
                                    85
               75
    80               70                           Within Budget         70      .20       14.0


    60
                                                  On-Time               89      .20       17.8

    40
                                                  Technical             85      .30       25.5
    20                                            Performance

     0
                    1st Qtr.

              Cust.Sat    Budget                 ITP Performance Index 79.8
              On-Time     Tech. Perf.




             GAO/AIMD-97-163 IT Performance Measurement Guide                                    Page 61
Case Study 10:
Effective Presentation of Performance Data and Contextual Information

The Minnesota Department of Administration provides business management and
administrative services to Minnesota agencies. Most of the Department's operations are fee-
based operations in areas such as data processing, printing, vehicle rental, and the sale of
office supplies. The Department's InterTechnologies Group (InterTech) provides services in
managing and operating information technology resources.

Figure 21 illustrates one objective for customer service performance. The objective is written
with a specific performance rating standard and the use of a specific survey for data
collection. The performance data are reported in this format, with the objective, definition,
rationale, and data source. The data also include a discussion of past performance and the
plan to achieve targets. These data and reports on other measures are given to the Minnesota
Legislature in a formal performance report.




Case Study 10:
Minnesota Department of Administration's Customer Service
Performance Data Template

OBJECTIVE 1: InterTech will receive
above-average customer service rating (i.e. a                      Actual performance   Fiscal    Fiscal
rating of 3.0 or better on a scale of 1.0-4.0) for                                      Year 1    Year 2
key services, as rated by customers on an
annual customer service survey.
                                                                   Telecom. Services
                                                                     Perform. Targets       3.0       3.3
DEFINITION, RATIONALE, DATA SOURCE:                                  Actual. Perform.       3.1

Definition: Key services are defined as those for                  Data Processing
which customers' ratings averaged 3.0 or greater,                  Services
indicating important or very important. In the FY 1                 Perform. Target         3.0       3.4
survey, 27 services were measured with                              Actual Perform.         3.2
responses from 259 customers. Customers rated
15 of 21 data processing services as important
and 8 of 11 telecommunications services as
important. The reported satisfaction ratings are             DISCUSSION OF PAST PERFORMANCE:
based on these services.
                                                             Prior to FY 1, InterTech periodically surveyed
Rationale: Above-average satisfaction is defined             customer agencies to determine their level of
as customer ratings averaging 3.0 or greater,                satisfaction with key services. This effort was
indicating satisfied or very satisfied. The future           expanded; surveys are now conducted annually.
year goal is to improve any services rated below
3.0 by at least 10% each year and to improve any
services rated 3.0 or above by at least 5% each              PLAN TO ACHIEVE TARGETS:
year, until a perfect score of 4.0 is achieved.
                                                             Targets in future years will be based on the
Data sources: Annual customer satisfaction                   previous year's percent improvement.
survey.




              Page 62                                 GAO/AIMD-97-163 IT Performance Measurement Guide
5. Conduct Measure Reviews and Audits

Most organizations we studied regularly assess their measures to see if
they are still appropriate for measuring results and assigning
accountability. Most of these organizations do regular external audits of
IT measure appropriateness and data collection efficiency. One manager
said, "We believe any successful organization needs a good basic
independent look at measures, challenging what are the right questions."
The reviews and audits serve as an oversight of measures and the data,
challenging the necessity of the measures (i.e., are they being used?) and
their linkage to enterprise strategic plans and individual performance
expectations (i.e., are they having an impact?). Reviewers consider the
accuracy, completeness, timing, match to actual conditions, and technical
features such as the method of results calculations.

For example, as new technology is introduced, measures can become
obsolete or less important. Or customer measures supplant or replace
traditional measures. In one organization, IT used thousands of lines of
code per calendar month as a measure of software productivity. From
the customer point of view, the measure was ineffective, and the
measure was changed to cycle time. Over time the organization will
develop more reliable measures.

The reviews also analyze key performance drivers in each goal area.
They question what a stress in one area will do to the results in other
areas. They also can question what performance gain might be achieved
for a certain level of performance effort, and if the performance results
are worth that effort.


  How to Get Started

  To build a comprehensive measure, data collection, and analysis
  capability, organizations should

  • designate specific IT staff to gain skills in measurement, data
    collection, and analysis,

  •   review existing data collection and reporting and determine
      what is still appropriate and what should be changed or deleted
      to match possible measurement scorecard areas, and

  •   determine what are preliminary interdependencies among
      scorecard goal areas, objectives, and measures.




GAO/AIMD-97-163 IT Performance Measurement Guide                      Page 63
Page 64   GAO/AIMD-97-163 IT Performance Measurement Guide
Practice Area 5:                                                    ALIGN

Strengthen                                                      Practice Area 1
                                                                     Use an

IT Processes to                                                     IT results
                                                                     chain



Improve Mission                           IMPLEMENT

                                              Practice Area 3
                                                                Practice Area 4
                                                                                            CONSTRUCT

                                                                    Build data        Practice Area 2
Performance                                   Target measures
                                                    at
                                              decision-making
                                                                collection/analysis
                                                                    capability
                                                                                         Follow a
                                                                                         balanced
                                                                                         scorecard
                                                  levels        Practice Area 5          approach
                                                                   Improve IT
                                                                   processes

                                                                REINFORCE




        Practice Area Characteristics:

        1. Define the IT business processes that produce IT products and
           services meeting mission goals
        2. Using IT performance information, prioritize IT business
           processes essential for improving mission performance.




      Practice Area Overview

      In many of the organizations we studied, business process improvement
      is a high priority enterprise strategy. Often, enterprise and operational
      customer business process improvement can only be accomplished using
      IT products and services. That means that IT must ensure that its own
      business processes are the best they can be. Simply put, IT internal
      business processes deliver IT products and services which the enterprise
      and operational customers depend on to support their missions. If IT
      does not have the capability to deliver high quality products and services,
      then organizational goals can suffer. This is an important reason why
      the balanced scorecard approach includes internal IT business processes.

    Practice Area Characteristics

      1. Define IT Business Processes That Produce Products
         and Services Critical for Meeting Mission Goals

      All leading organizations define their key IT business processes. This
      helps the IT organization focus on primary activities, identify IT
      competencies, eliminate processes which do not add value, and facilitate
      IT process innovation. Several of the organizations we studied noted

         Page 64                       GAO/AIMD-97-163 IT Performance Measurement Guide
that process measures are tightly linked with tasks or activities. Should
organizational structures change and tasks and activities move within the
IT organization, measures can more easily move. In other words, IT
processes and subprocesses, once defined, are "portable" from one part
of the organizational structure to another.

Some of the organizations developed their IT process orientation based
on work done by the Ernst and Young Center for Information
Technology and Strategy and later published by a workgroup of the
Society for Information Management (SIM).17 SIM's Information Systems
Process Architecture (ISPA) process framework is a model of how a
typical organization obtains and applies information systems and
technology. In describing its initial framework issued in September 1993,
SIM indicated that its process model (1) defines strategically important
IT processes in an overall framework, (2) communicates with IT
stakeholders on the value, activities, and organization of IT, and (3)
provides a basis for allocating IT resources that is congruent with
activity-based costing.

As shown in figure 13, SIM's ISPA version 2.0 process framework, issued
in March 1996, includes eight IT processes which overlap. This
framework, like the earlier version, provides an example of a way to
organize IT, determine core competencies, and identify process owners.

Figure 13: ISPA Version 2.0 Process Framework


                      Perform customer relations




                      Align       Manage      Develop & Deliver &
            Market
                     business    enterprise    deploy    support
              IT
                      and       architecture products/ products/

                       IT                     services   services




                              Plan IS organization


                            Manage IS organization




17
  See the Society for Information Management Working Group on ISPA
process architecture documents listed in the bibliography.

GAO/AIMD-97-163 IT Performance Measurement Guide                     Page 65
As explained below, each process in this suggested framework has a
specific purpose and suggested metrics (measures).

• Perform Customer Relations
This process is used to develop and maintain working relationships with
the customers of IT products and services. Performance metrics focus
on the business impact of IT, customer satisfaction, and product and
service measurements, such as the accuracy and response time for
solving problems or number of viable IT projects identified.

• Market IT
This process is used to ensure that customers want, need, and buy the
products and services offered. Sample metrics focus on the increased
business value of IT, IT return on investment, customer satisfaction, and
product and service effectiveness measures.

• Align Business and IT
This process is used to incorporate IT into strategic business change
activities in a way that captures opportunities from current and emerging
technologies, and to promote process innovation leadership using IT as
the catalyst and using proven visioning and change management
techniques. Sample metrics focus on the business value of IT, net
present value of projects approved by the strategic business unit, and
improvements to business processes.

• Manage Enterprise Architecture
This process is used to provide a framework for delivering consistent
products and services. Sample metrics focus on IT architecture design
and implementation, the number of viable IT projects identified, how
service chargebacks align with customer views of services, and the
degree of technology standardization.

• Develop and Deploy Products and Services
This process is used to acquire, develop, deliver, and implement new
information services for the organization. Sample metrics might include
the percent of projects on-time and with the desired functionality,
projects completed within budget, customer satisfaction, and degree of
technology usage in conducting core business processes.

• Deliver and Support the Products and Services
This process is used to ensure that the products and services are
deployed in the most effective and efficient manner. Metrics can focus
on customer satisfaction survey ratings, increased demand for IT
information, adequacy of equipment and facilities, availability and
accessibility of data, number of problems received and requests satisfied,
and mean time to data recovery.


   Page 66                       GAO/AIMD-97-163 IT Performance Measurement Guide
GAO/AIMD-97-163 IT Performance Measurement Guide   Page 67
        • Plan the IS Organization
        This process is used to shape and support business unit strategies,
        establish strategy and vision for long-term IS use, develop tactical plans
        and development and infrastructure resources over a 12 to 18 month
        horizon, and design key processes within the IS organization. Sample
        metrics could include employee satisfaction with the IS vision and
        knowledge about technology plans.

        • Manage IS Organization Business
        This process is used to manage the processes within IS that deal with the
        health and state of the IS organization, its employees, and vendors.
        Sample metrics include an employee satisfaction and commitment index,
        team efficiency and effectiveness measures, product and service
        measurements, and an index of employee skills.




Case Study 11:
Texas Instruments IT Process Segmentation Effectively Supports Business
Needs

At Texas Instruments, many IT customer business process reengineering activities established
a clear need for rapid provisioning of IT solutions, flexibility with respect to business rules
and work flows, and cost effective use of advanced technology. In response, the Information
Systems and Services Group (IS&S) developed its IT process map with five processes, as
shown in the following figure.

The process map defines activities that "map" into both the operational customer and IT
organization sphere. For example, strategy management as a process is the joint
responsibility of both the business customers and IS&S (the enterprise IT group). Strategy
management sets the vision and timing of all the elements of IT needed to support Texas
Instruments. Solution provisioning is the mechanism for assembling the hardware and
software pieces together to form solutions for IT customers. The service and product support
area deploys and maintains IT products and services. The research and architecture
definition area does research, experiments with and sets standards and methodologies for
current and future architectures. The component provisioning area provides reusable
applications, building blocks, assembly tools, and integration expertise.




            Page 68                         GAO/AIMD-97-163 IT Performance Measurement Guide
Case Study 11:
Texas Instruments' Information Technology Process Map

                                         Concept
                                                                   Provision
  Business IT




                                                                                                                   Use


                                       Strategy             IT Needs
                                      Management                         Solution
                                                                                                            Solutions            Service and
                                                                       Provisioning
                                                                                                                                  Product
                                                                                                                                  Support




                                                                                             Compon ent s
                                      Enterprise Principles,
                                      Architectures, and Methods
   Enterprise IT




                                        Research and
                   IT Market Trends




                                        Architecture                     Component
                                         Definition                      Provisioning
                                                                                                                                      Enablers

                                                                                                                    Capability
                                                                              Compo nents
                                             IT




                                                     Commercial          IT                 Marketplace



                   2. Using IT Performance Information, Prioritize IT Business
                      Processes Essential for Improving Mission Performance

                   Given the many IT business processes that an IT organization manages,
                   which ones are the most important for improvement? In the
                   organizations we studied, customer business process improvement
                   strategies and performance requirements frequently identify and drive
                   major improvements in IT business process. They key is to determine
                   the most efficient and effective manner to organize roles, responsibilities,
                   and delivery of IT products and services within the organization. This
                   requires a clear understanding of what IT functions are the domain of
                   business units, unit-organized IT groups, and the corporate or
                   enterprisewide IT organization. Knowing how effective is this
                   arrangement is in providing effective IT support, which IT functions are
                   performing well, and where service improvements are needed is critical
                   to targeting IT management attention.


                   GAO/AIMD-97-163 IT Performance Measurement Guide                                                                              Page 69
Case Study 12:
Using Business Process Focus to Shape IT Service Delivery

Kodak is deciding what IT processes it should pursue to support the enterprise and operational
customers' business processes. The focus is to look at each business process and its use of IT and
determine if the information systems organization identified the right technology for the business
needs.

For Kodak, enterprise goals drive a process focus for strategic business units and support functions
such as IT. The company's enterprise goals are three fold:

   (1) a 10 time defect reduction every 3 years,
   (2) reaching 6 sigma by the year 2000, and
   (3) a 10 time cycle time reduction in 3 years.

Kodak defines a defect as a variation in a product or service which, if not caught, prevents the
company from meeting the needs of its customers. Cycle time is the total time to move a unit of
work from the beginning to the end of a process. Kodak believes that maximizing business
processes is the only way to achieve these dramatic improvement goals.




      How to Get Started
      To begin focusing resources on IT process improvement,
      organizations should

          •   use a model such as that suggested by the Society for
              Information Management to define IT business processes,

          •   determine which IT business processes are the most critical
              to meeting enterprise and operational customer
              performance goals and requirements, and

          •   choose one or more critical IT business processes to
              baseline existing performance and benchmark against
              industry standards or leading public/private organizations.




          Page 70                            GAO/AIMD-97-163 IT Performance Measurement Guide
Key Lessons Learned for Effective
Implementation
______________________________________________________________________________


            Agency managers just starting to develop IT performance management
            systems or those who want to enhance existing ones have a formidable
            task. They often are faced with resource constraints, demands for
            immediate IT support and solutions as program areas reduce staff and
            reengineer their business processes, and skepticism about the value of
            performance management.

            From the organizations we studied and the experiences of other
            organizations described in the literature, three key activities are essential
            in putting the practices in place. These are assessing organizational
            readiness for a successful IT performance management system and
            staging the system development, following a simple measure selection
            process, and recognizing system maturity will change over time.

            Assess Organizational Readiness

            The leading organizations find that assessing organizational readiness for
            a successful IT performance management system is essential. Here, the
            organizations look for the involvement, commitment, and day-to-day
            support of enterprise senior managers. They also determine if they have
            adequate resources, including staff allocation, skills, time, tools, and use
            of consultants or technical assistance if needed. A manager
            characterized a good performance management system as one that "has
            complete buy-in from top management, involves front line employees in
            system design, and results in front line employees understanding what
            they do and why they are doing it."

            Organizational readiness also means making sure that existing planning
            and decision making structures can accept performance results so they
            can be used. As the introductory chapter to this guide explained,
            performance measures are a central piece of alignment around mission
            planning, budgeting, and evaluation. These are essentially separate
            processes linked by performance measures. The organization needs the
            capability to specify clear goals and objectives to set the focus and
            direction of IT performance, creating an IT performance improvement
            plan and revisiting it every one or two years. The IT organization has to
            understand the business of operational customers and make sure IT
            measures are consistent with business measures. That means the
            capability to develop a "theory" of how IT supports enterprise and


                          Page 70      GAO/AIMD-97-163 IT Performance Measurement Guide
operational customers so the organization can build the chain of IT
events and evidence described in practice 1.

The organizations also determine if they have the support of other
stakeholders and funding sources, such as legislative staff. As mentioned
under practice 1, stakeholders are one of the parties which have to reach
a common understanding of IT goals, objectives, appropriate measures,
and anticipated outcomes.

Lastly, organizational readiness means paying attention to organizational
culture -- is it receptive to data collection, measurement, and analysis and
accountability for performance and decisions as part of an overall
performance improvement system? The organization should have a
philosophy that is positive towards performance management and
measurement and views it as a way to focus on quality and operational
customer satisfaction. That means the organization should be willing to
assess organizational values, principles, and how they are working. That
is the key to success, one manager explained, "We have the culture to
support this -- people get to where they track and report metrics, they
think about quality goals. [IT performance management] is a hard sell
where there is not a total quality culture, where the focus is on
technology, not satisfying the customer."

Follow a Simple Measure
Selection Process

Selecting and implementing IT performance measures is extremely
complex. Each enterprise and its operational customers have a mission
and goals that often differ significantly from other agencies. A set of IT
performance measures that work for one organization likely will not
completely work for another organization. The performance measures
differ on what is valued most in terms of IT performance. To illustrate,
IT goals and objectives which stress cost savings would have more
measures related to efficiency versus effectiveness. IT goals and
objectives stressing customer satisfaction and other service-oriented
goals might have fewer efficiency measures. However, well-managed IT
activities should take a balanced approach to goals, objectives, and
related measures.

In the organizations we studied and in the literature we reviewed, one
element of success is sifting through the many possible goals, objectives,
and measures before finalizing them in a balanced scorecard or similar
approach. The sifting process identifies potential objectives and
measures for all parts of IT activities. It assesses which measures will be
valuable for which purposes and to whom and eliminates measures
which are not relevant to customer and stakeholder needs. And it

GAO/AIMD-97-163 IT Performance Measurement Guide                       Page 71
eliminates measures for which good quality data cannot be practically
obtained. One manager said, "We want to put the best measures in
place. We want them simple, collectable, repeatable, and concrete."

A good example of the selection process is the measurement "roadmap"
followed by the Minnesota Department of Transportation. The roadmap
examines and reduces the total potential to a vital few number of goals,
objectives, and related measures. The roadmap, shown in figure 14 and
enhanced with additional information from the United Kingdom's Royal
Mail, is a general performance measure selection framework that all
program and support functions follow. In the roadmap, the word
"organization" can refer to an agency, a program, or a support function.

Figure 14: IT Measurement Implementation Roadmap

 Stage I Clarify Goals and Objectives
  INPUTS                              DIVERGENCE
                                      Full listing of                    CONVERGENCE
  Vision/Mission
                                      everything the                     Determine "vital
  Strategic/Annual Plans                                                 few" goals and
  Legislation/Policies                enterprise and
                                      programs are trying                specific objectives
  Performance                         to achieve
  Expectations


 Stage II Create Supporting Outcome and Process Measures
                     DIVERGENCE                                                         Finalize measurement
                                             Review existing     CONVERGENCE            system:
  Select a vital     Ask 1) What             process and         Select "vital few"
  goal and           has to happen?                                                     -diversity of measures
                                             outcome             measures for
  objectives         2) How will we          measures:                                  -clarify linkages from
                                                                 each objective         the enterprise to the
                     know?                   keep? discard?
                                                                                        task level
                           Repeat for all Goals and Objectives



 Stage III Put Measures to Work
                                                                      Reinforce                Continually
 Communicate          Establish                 Tackle
                                                                      progress:                improve
 -measures            baselines and             improvement
                                                opportunities         -data display            measurement
 -purpose/benefit     do
                                                                      -meaningful              system
 -owner               benchmarking              -outcome
                                                                      feedback                 -drive
 -actionability       -gap analysis             -process
                                                                      -reinforcement           improvement

Source: Minnesota Dept. of Transportation and United Kingdom's Royal Mail




In stage 1, the organization clarifies its overall goals and objectives, first
by fulling listing all possibilities, and then converging on the vital few
goals and objectives. In stage II, the organization would develop its
measures and make sure its measurement system has a diversity of
measures and follows a tiering approach. In the last stage, the
organization uses the measures, establishing baselines and targets,


                    Page 72              GAO/AIMD-97-163 IT Performance Measurement Guide
comparing against benchmarks, and monitoring progress for continual
improvement.

For support functions such as IT, two roadmaps are in play. One
roadmap develops the objectives and measures for an agency or program
area; another roadmap takes that agency or program information and
develops objectives and measures for IT. This is the intent of the
balanced scorecard approach discussed in practice 2.

Develop IT Performance Management
System Maturity Over Time

In discussing the development of an IT performance management system,
we attempted to determine if there was a maturity aspect to system
development. Several of the managers we talked to believe there is a
staging in performance management emphasis and expectations although
staging is not precise or formal. Figure 15 shows possible stages in
performance management and their linkage to the balanced scorecard
areas, drawing on work done for the city of Phoenix and discussed in the
literature. While shown as discrete stages for purposes of illustration,
the distinction between them often is not clear-cut as they overlap over
time.




GAO/AIMD-97-163 IT Performance Measurement Guide                    Page 73
Figure 15: Stages in IT Performance Management




                                 deliver reliable, cost-effective operations
   1. Traditional                focus on internal IT measurement          Internal
     Internal IT                 determine adherance to standards Business
                                 focus on technical factors such as       Emphasis
    Operations                   quality, productivity, and impact on Innovation &
                                 customer service                          Learning
                                 build on competencies to consistently Emphasis
 2. Linking to the               deliver new and improved systems,
 MIssion with New                solutions on time and within budget
  and Improved                   eliminate root causes of defects;
                                 improve processes
     Services                    link key technical indicators to mission
                                 performance
                                                                          Customer
                                 apply IT expertise to the mission         Needs
                                 create an enabling and evolving IT       Emphasis
     3. Mission/                 architecture
   Results Oriented              have measures of IT outcomes in          Strategic
                                                                           Needs
                                 mission terms, based on customer
                                                                          Emphasis
                                 needs and in "customer" terminology
 Source: Howard Rubin, Ernst & Young, City of Phoenix




The premise is that IT organizations have to be good at the basics in
stage one -- traditional internal IT operations -- and then move to stages
two (linking enterprise and customer mission with new and improved IT
services) and three (be mission-results oriented). In other words, an IT
organization that is viewed as a failure in day-to-day operations will not
have the credibility nor support of the rest of the organization to play a
strategic role directly tied to mission results.

In stage one, the IT organization is developing and implementing a
performance management system that examines internal operations
against standards and acceptable levels of performance. Traditional,
activity-based measures such as number of reports issued or mainframe
availability are used. In stage two, the IT organization is eliminating root
causes of defects, building its competencies to consistently deliver new
and improved systems and solutions on time and within budget, and
linking operational measures to mission performance. The goal is to
improve IT processes and prevent defects. For both stage one and two,
most of the measures fall in the internal business goal and innovation

               Page 74         GAO/AIMD-97-163 IT Performance Measurement Guide
and learning goal of the balanced scorecard approach discussed in
practice 2.

In stage three, the IT organization has the capability of applying IT
expertise to enterprise and operational customer mission requirements
and putting its IT outcomes in those mission terms. Measures are based
on customer needs and benefits, expressing measures in terms customers
understand, such as business outcomes. Stage three IT organizations
also specify who is responsible for corrective action. In this stage, the
measures found in the strategic and operational customer goals of the
balanced scorecard approach are prominent.

The staging or maturity perspective suggests that organization should
consider using the balanced scorecard approach in a building block
approach. The organization might initially develop major goals,
objectives, and measures in each of the four areas. However, it is
possible that it likely must perform well in stages one and two, reflecting
the balanced scorecard areas of internal business and innovation and
learning, before it can perform well in strategic and operational customer
areas.

A Final Note

The organizations we studied cautioned that the practice of IT
performance management and measurement is in its infancy. Most of the
organizations we studied have worked on their IT performance
management systems for several years and most of those efforts are part
of strong performance goals at the enterprise level. The cities of
Phoenix and Sunnyvale, for example, have long-standing reputations for
being well-managed and their general and specific IT measurement
approaches have evolved over many years.

In both these organizations, as with others we studied, there is a strong
performance management culture. The organizations share many similar
performance management values and management objectives that stress
IT results and accountability. For them, IT measures make a valued and
positive difference in mission and business performance.




GAO/AIMD-97-163 IT Performance Measurement Guide                       Page 75
              APPENDIX I                                               APPENDIX I


                                        Selected Bibliography
Allen, Dan. "Performance Anxiety," Computerworld, February 15, 1993.

Allen, Dan. "Take a Good Look at Yourself," Computerworld, February 15, 1993.

Association for Federal Information Resources Management. The Connection: Linking IRM and
Mission Performance. Washington, DC: Association for Federal Information Resources
Management, September 1995.

Baatz, E. B. "Altered Stats," CIO, October 15, 1994.

Bjorn-Andersen, Niels and Davis, Gordon B. (eds.). Information Systems Assessment: Issues
and Challenges. Amsterdam: North-Holland, 1986.

Brizius, Jack A. and Campbell, Michael D. Getting Results. Washington, DC: Council of
Governors' Policy Advisors, 1991.

Carlson. W. and McNurlin, B. Measuring the Value of Information Systems. Rockville, MD:
I/S Analyzer, 1989.

Davis, Dwight B. "Does Your IS Shop Measure Up?" Datamation, September 1, 1992.

Earl, Michael J. and Feeny, David F. "Does the CIO Add Value?" Information Week, May 30,
1994.

Financial Management Service, Department of Treasury. Performance Management Guide.
Washington, DC: Financial Management Service, November 1993.

Gaskin, Barbara and Sharman, P. "IT Strategies for Performance Measurement," CMA
Magazine, April 1994.

Glover, Mark. A Practical Guide for Measuring Program Efficiency and Effectiveness in Local
Government. Tampa, FL: The Innovation Groups, 1994.

Gold, Charles L. IS Measures -- A Balancing Act. Boston: Ernst & Young Center for
Information Technology and Strategy, 1992.

Government Centre for Information Systems. Benchmarking IS/IT. London: HMSO, 1995.




              Page 76                 GAO/AIMD-97-163 IT Performance Measurement Guide
              APPENDIX I                                               APPENDIX I

Governor's Office of Budget and Planning, Legislative Budget Board, and State Auditor's
Office. Guide to Performance Measurement for State Agencies. Austin, TX: State Auditor's
Office, 1995.

Harrington, H. James. Total Improvement Management. New York: McGraw-Hill, Inc., 1995.

Jackson, Peter and Palmer, Bob. First Steps in Measuring Performance in the Public Sector.
London: Public Finance Foundation, 1989.

Kaplan, Robert S. and Norton, David P. "The Balanced Scorecard -- Measures that Drive
Performance," Harvard Business Review, Vol. 70, No. 2 (January-February 1992).

Kaplan, Robert S. and Norton, David P. "Putting the Balanced Scorecard to Work," Harvard
Business Review, Vol. 71, No. 5 (September-October 1993).

Kaplan, Robert S. and Norton, David P. "Using the Balanced Scorecard as a Strategic
Management System," Harvard Business Review, Vol. 74, No. 1 (January-February 1996).

Klinger, Daniel E. "A Matter of Marketing," CIO, September 15, 1995.

LaPlante, Alice. "IT's Got What It Takes," Computerworld, October 3, 1994.

LaPlante, Alice and Alter, Allan E. "IT All Adds Up," Computerworld, October 31, 1994.

Lingle, John H. and Schieman, William A. "From Balanced Scorecard to Strategic Guages: Is
Measurement Worth It?," Management Review, March 1996.

Moad, Jeff. "New Rules, New Ratings as IS Reengineers," Datamation, November 1, 1993.

National Academy of Public Administration. Information Management Performance Measures.
Washington, DC: National Academy of Public Administration, January 1996.

Parker, Marilyn M., Benson, Robert J., and Trainor, H.E. Information Economics: Linking
Business Performance to Information Technology. Englewood Cliffs, NJ: Prentice Hall, 1988.

Pastore, Richard. "Benchmarking Comes of Age," CIO, November 1, 1995.

Quinn, James B. and Baily, Martin N. "Information Technology: The Key to Service
Performance," The Brookings Review, Summer 1994.

Rubin, Howard. "Measurement: Shifting the Focus to Business Value," Capacity Management
Review, Vol. 19, No. 1 (January 1991).


              GAO/AIMD-97-163 IT Performance Measurement Guide                       Page 77
              APPENDIX I                                               APPENDIX I


Rubin, Howard. "Measure for Measure," Computerworld, April 15, 1991.

Rubin, Howard A. "In Search of the Business Value of Information Technology," Application
Development Trends, November 1994.

Sankaran, Chandran. and Taylor, Henry D. "Taking New Measures," CIO, October 1, 1993.

Saunders, Carol S. and Jones, Jack W. "Measuring the Performance of the Information Systems
Function," Journal of Management Information Systems, Vol. 8, No. 4 (Spring 1992).

Society for Information Management Working Group on ISPA. Information Systems Process
Architecture 1.0. Chicago: Society for Information Management, September 1993.

Society for Information Management Working Group on ISPA. Information Systems Process
Architecture 2.0. Chicago: Society for Information Management, March 1996.

Society for Information Management Advanced Practices Council. Practitioner's Guide to I. S.
Performance Measurement. Chicago: Society for Information Management, March 1995.

Strassmann, Paul A. The Business Value of Computers. New Canaan, CT: The Information
Economics Press, 1990.

Strassmann, Paul A. and others. Measuring Business Value of Information Technologies.
Washington, DC: International Center for Information Technologies, 1988.




              Page 78                 GAO/AIMD-97-163 IT Performance Measurement Guide
APPENDIX II                                                                      APPENDIX II


                   Objectives, Scope, and Methodology
The objectives of our research were to (1) identify information technology performance
management practices used by leading private and public sector organizations with
demonstrated success in developing and using information technology solutions and
infrastructure to improve mission or business performance and outcomes, and (2) share our
results with federal agencies to help improve overall mission performance.

Scope

Our research focused on information technology performance management practices used by
management teams and staff in five private sector companies, three state and local governments,
and three federal agencies. The organizations were chosen purposefully, not at random or to
ensure representation of a larger group. We selected the private, state, and local organizations
based on (1) recognition by other organizations and independent researchers for their progress
in successfully developing and using information technology performance management systems,
(2) recognition by professional publications for their performance management systems, and (3)
willingness to participate as a case study organization. The federal agencies were selected
based on their recognition by federal information resources management officials for initiating
comprehensive work on information technology performance management. Because our work
often involved data that these organizations regarded as proprietary or sensitive, we agreed not
to disclose any data they wished to protect.

To supplement our findings from these private and public sector organizations, we gathered
additional information from other federal, state, and local organizations. This information
included both generic and information technology performance management information,
ranging from guides to specific practices.

Methodology

Our research was conducted with an illustrative case study approach using open-ended and
focused interviews and documentary analysis, not direct observations. In conducting the case
studies, we interviewed senior executives, line managers, and information technology
professionals to learn how the organization measured and managed the contribution of
information technology towards organizational goals. Interview information was supplemented
by documentary analysis of each organization's information technology performance
management approach.

For quality assurance, we conducted a meeting of case study participants to obtain group
comments on an initial draft of this guide, followed by individual case study participant reviews
of a subsequent draft. We also distributed the draft to other experts on information technology
performance management, the Office of Management and Budget, and the General Services
Administration. We also made numerous presentations as the guide was developed to test our

GAO/AIMD-97-163 IT Performance Measurement Guide                                         Page 79
APPENDIX II                                                                      APPENDIX II


preliminary findings and the applicability to the federal government with Federal executives,
managers, and staff, representing both line and information technology functions. We have
incorporated changes where appropriate.




Caveats

Information technology performance management and measurement is very much in its infancy
in both public and private sectors. As an initial step, this guide presents a framework that
begins to document the state of the practice drawn from our analysis of a relatively small
number of case studies. Much more research and analysis remains to be done. The practices
we have presented can serve as starting point for any organization, tailored to the strategic
directions and performance requirements unique to each organization.




Page 80                                  GAO/AIMD-97-163 IT Performance Measurement Guide
APPENDIX III                                                                  APPENDIX III


              Case Study Organizations and Participants
We would like to acknowledge the invaluable assistance of the following individuals who
serviced as key contacts for the case study organizations.

American Express Travel Related
Services Company, Inc.
Cliff Shoung (Project Leader,                        City of Phoenix
    Benchmarking)                                    Joe Motto (Information Technology
Paula Bouthillier (Project Leader, Strategic             Director)
    Advantage)                                       Bob Wingenroth (Deputy City Auditor)

U.S. General Services Administration                 City of Sunnyvale
Jon Desenberg (Management Analyst)                   Shawn Hernandez (Director, Information
                                                        Technology)
Immigration and Naturalization Service               Marilyn Crane (Information Technology
Janet Keys (Director, Systems Policy and                Administrative Services Manager)
    Planning)
J. T. Lazo (Senior Systems Manager, EDS,             Texas Instruments
    INS ITP Contract)                                Gary Pollard (Total Quality Director)
                                                     Vicki Flewelling (Total Quality
Eastman Kodak Company                                   Management)
Jeff Duell (Technology Director, IS
    Measurement and Benchmarking)                    U.S. Department of Agriculture
                                                     Joseph Ware (Director, IT Planning and
Motorola Semiconductor Products                          Program Management)
Sector
Dorothy Hines (Controller, Sector Quality            Xerox Corporation
   and Support Operations)                           M. Catherine Lewis (Manager, IM2000
Bret Wingert (Manager, SEI Projects)                    Integration Planning)
                                                     Margie Tomczak (Manager, Global
Oregon Department of Transportation                     Applications)
Craig Holt (formerly Chief Information
   Officer)




GAO/AIMD-97-163 IT Performance Measurement Guide                                      Page 81
APPENDIX IV                                                              APPENDIX IV


                     Major Contributors to This Report

Accounting and Information Management Division
IRM Policy and Issues Group

Christopher W. Hoenig, Director
Dr. David L. McClure, Senior Assistant Director
Bernard R. Anderson, Senior Analyst

Los Angeles Office

Dr. Sharon Caudle, Project Director
Barbara House, Senior Analyst




Page 82                                GAO/AIMD-97-163 IT Performance Measurement Guide