oversight

Information Technology Dashboard: Opportunities Exist to Improve Transparency and Oversight of Investment Risk at Select Agencies

Published by the Government Accountability Office on 2012-10-16.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

               United States Government Accountability Office

GAO            Report to Congressional Requesters




October 2012
               INFORMATION
               TECHNOLOGY
               DASHBOARD
               Opportunities Exist to
               Improve Transparency
               and Oversight of
               Investment Risk at
               Select Agencies




GAO-13-98
                                              October 2012

                                              INFORMATION TECHNOLOGY DASHBOARD
                                              Opportunities Exist to Improve Transparency and
                                              Oversight of Investment Risk at Select Agencies
Highlights of GAO-13-98, a report to
congressional requesters




Why GAO Did This Study                        What GAO Found
In June 2009, OMB launched the                Chief Information Officers (CIO) at six federal agencies rated the majority of their
federal IT Dashboard, a public website        information technology (IT) investments as low risk, and many ratings remained
that reports performance data for over        constant over time. Specifically, CIOs at the selected agencies rated a majority of
700 major IT investments that                 investments listed on the federal IT Dashboard as low risk or moderately low risk
represent about $40 billion of the            from June 2009 through March 2012; at five of these agencies, these risk levels
estimated $80 billion budgeted for IT in      accounted for at least 66 percent of investments. These agencies also rated no
fiscal year 2012. The Dashboard is to         more than 12 percent of their investments as high or moderately high risk, and
provide transparency for these                two agencies (Department of Defense (DOD) and the National Science
investments to aid public monitoring of       Foundation (NSF)) rated no investments at these risk levels (see table). Over
government operations. It does so by          time, about 47 percent of the agencies’ Dashboard investments received the
reporting, among other things, how            same rating in every rating period. For ratings that changed, the Department of
agency CIOs rate investment risk.             Homeland Security (DHS) and Office of Personnel Management (OPM) reported
GAO was asked to (1) characterize the         more investments with reduced risk when initial ratings were compared with
CIO ratings for selected federal              those in March 2012; the other four agencies reported more investments with
agencies’ IT investments as reported          increased risk. In the past, the Office of Management and Budget (OMB)
over time on the Dashboard, (2)               reported trends for risky IT investments needing management attention as part of
determine how agencies' approaches            its annual budget submission, but discontinued this reporting in fiscal year 2010.
for assigning and updating CIO ratings
vary, and (3) describe the benefits and       Average Composition of CIO Ratings for Agencies’ Major IT Investments, June 2009
challenges associated with agencies’          through March 2012
approaches to the CIO rating.
To do so, GAO selected six agencies                                                                                            Agency
spanning a range of 2011 IT spending                                                                        DOD        DHS      HHS     DOI     OPM    NSF
levels and analyzed data reported for          High risk and moderately high risk investments                    0%     11%       5%      4%    12%     0%
each of their investments on the               Medium risk investments                                           15%    38%      21%     13%    22%     0%
Dashboard. GAO also interviewed                Low risk and moderately low risk investments                      85%    51%      74%     83%    66%    100%
agency officials and analyzed related          Range in the number of investments during
documentation and written responses            the period                                                    49-87     63-83    55-81   33-48    6-9    3-5
to questions about ratings and                Source: GAO analysis of data downloaded from OMB’s IT Dashboard.
evaluation approaches, as well as             Note: Table does not include downgraded or eliminated investments. HHS is the Department of
agency views on the benefits and              Health and Human Services. DOI is the Department of the Interior.
challenges related to the CIO rating.
                                              Agencies generally followed OMB’s instructions for assigning CIO ratings, which
What GAO Recommends                           included considering stakeholder input, updating ratings when new data become
GAO is recommending that OMB
                                              available, and applying OMB’s six evaluation factors. DOD’s ratings were unique
analyze agencies’ investment risk over        in reflecting additional considerations, such as the likelihood of OMB review, and
time as reflected in the Dashboard’s          consequently DOD did not rate any of its investments as high risk. However, in
CIO ratings and present its analysis          selected cases, these ratings did not appropriately reflect significant cost,
with the President’s annual budget            schedule, and performance issues reported by GAO and others. Moreover, DOD
submission, and that DOD ensure that          did not apply its own risk management guidance to the ratings, which reduces
its CIO ratings reflect available             their value for investment management and oversight.
investment performance assessments
and its risk management guidance.             Various benefits were associated with producing and reporting CIO ratings. Most
Both OMB and DOD concurred with               agencies reported (1) increased quality of their performance data, (2) greater
our recommendations.                          transparency and visibility of investments, and (3) increased focus on project
                                              management practices. Agencies also noted challenges, such as (1) the effort
View GAO-13-98. For more information,         required to gather, validate, and gain internal approval for CIO ratings; and (2)
contact David A.Powner at (202) 512-9286 or   obtaining information from OMB to execute required changes to the Dashboard.
pownerd@gao.gov.
                                              OMB has taken steps to improve its communications with agencies.
                                                                                                          United States Government Accountability Office
Contents


Letter                                                                                      1
               Background                                                                   3
               CIOs Rated Most IT Investments as Low Risk or Moderately Low
                 Risk; Agencies Had Mixed Results in Reducing Higher Risk
                 Levels                                                                   14
               Most Agencies Established Approaches for CIO Ratings That
                 Follow OMB’s Instructions                                                20
               CIO Ratings Present Benefits and Challenges for Agencies’
                 Investment Management                                                    25
               Conclusions                                                                27
               Recommendations for Executive Action                                       28
               Agency Comments and Our Evaluation                                         28

Appendix I     Objectives, Scope, and Methodology                                         30



Appendix II    Risk Levels for Investments at Selected Agencies, as of March
               2012                                                                       33



Appendix III   CIO Ratings and Net Changes for Major IT Investments at Selected
               Agencies, June 2009 through March 2012                                     34



Appendix IV    Comments from the Department of Defense                                    47



Appendix V     GAO Contacts and Staff Acknowledgments                                     49



Tables
               Table 1: Dashboard Variance and Rating Colors                                8
               Table 2: Investment Evaluation Factors Identified by OMB for
                        Assigning CIO Ratings                                             10
               Table 3: IT Dashboard CIO Rating Colors, Based on a Five-Point
                        Scale for CIO Ratings                                             11
               Table 4: Average Composition of CIO Ratings for Agencies’ Major
                        IT Investments from June 2009 through March 2012                  15


               Page i                               GAO-13-98 Information Technology Dashboard
          Table 5: Number of Major IT Investments with CIO Ratings That
                   Changed Compared to Those That Remained Constant,
                   Initial Rating through March 2012                                16
          Table 6: Net Changes in Investment Risk, Based on Comparison of
                   Initial Rating to March 2012                                     18
          Table 7: Data Sources and Derivation of CIO Ratings by Selected
                   Agencies                                                         21
          Table 8: CIO Ratings and Budget Totals for Selected Agencies’
                   Major IT Investments as of March 2012                            33


Figures
          Figure 1: Example of an Agency Portfolio Page as Reported on
                   OMB’s IT Dashboard Website, July 2012                              6
          Figure 2: Example of an Agency’s IT Investment Page from the IT
                   Dashboard Website, July 2012                                     13
          Figure 3: CIO Ratings and Associated IT Budgets for Fiscal Year
                   2012 Dashboard Investments from Selected Agencies, as
                   of March 2012                                                    14
          Figure 4: Percentages of Major IT Investments at Selected Agencies
                   with CIO Ratings That Changed Compared to Ratings That
                   Remained Constant, Initial Rating through March 2012             16
          Figure 5: Percentages of Major IT Investments at Selected Agencies
                   with Changes in CIO Ratings, Initial Rating Compared to
                   March 2012                                                       18
          Figure 6: CIO Ratings for Major IT Investments at the Department
                   of Defense, as Reported on the IT Dashboard from July
                   2009 through March 2012                                          35
          Figure 7: Changes to Risk Levels for Major IT Investments at the
                   Department of Defense, Initial CIO Rating through March
                   2012                                                             36
          Figure 8: CIO Ratings for Major IT Investments at the Department
                   of Homeland Security, as Reported on the IT Dashboard
                   from July 2009 through March 2012                                37
          Figure 9: Changes to Risk Levels for Major IT Investments at the
                   Department of Homeland Security, Initial CIO Rating
                   through March 2012                                               38
          Figure 10: CIO Ratings for Major IT Investments at the Department
                   of Health and Human Services, as Reported on the IT
                   Dashboard from July 2009 through March 2012                      39




          Page ii                             GAO-13-98 Information Technology Dashboard
Figure 11: Changes to Risk Levels for Major IT Investments at the
         Department of Health and Human Services, Initial CIO
         Rating through March 2012                                         40
Figure 12: CIO Ratings for Major IT Investments at the Department
         of the Interior, as Reported on the IT Dashboard from July
         2009 through March 2012                                           41
Figure 13: Changes to Risk Levels for Major IT Investments at the
         Department of the Interior, Initial CIO Rating through
         March 2012                                                        42
Figure 14: CIO Ratings for Major IT Investments at the National
         Science Foundation, as Reported on the IT Dashboard
         from June 2009 through March 2012                                 43
Figure 15: Changes to Risk Levels for Major IT Investments at the
         National Science Foundation, Initial CIO Rating through
         March 2012                                                        44
Figure 16: CIO Ratings for Major IT Investments at the Office of
         Personnel Management, as Reported on the IT Dashboard
         from July 2009 through March 2012                                 45
Figure 17: Changes to Risk Levels for Major IT Investments at the
         Office of Personnel Management, Initial CIO Rating
         through March 2012                                                46




Page iii                             GAO-13-98 Information Technology Dashboard
Abbreviations

CIO                        chief information officer
DEAMS                      Defense Enterprise Accounting and Management
                           System
DOD                        Department of Defense
DOI                        Department of the Interior
DHS                        Department of Homeland Security
GCSS-Army                  Global Combat Support System-Army
GFEBS                      General Fund Enterprise Business System
HHS                        Department of Health and Human Services
IT                         information technology
NSF                        National Science Foundation
OMB                        Office of Management and Budget
OPM                        Office of Personnel Management
TechStat                   TechStat Accountability Sessions




This is a work of the U.S. government and is not subject to copyright protection in the
United States. The published product may be reproduced and distributed in its entirety
without further permission from GAO. However, because this work may contain
copyrighted images or other material, permission from the copyright holder may be
necessary if you wish to reproduce this material separately.




Page iv                                      GAO-13-98 Information Technology Dashboard
United States Government Accountability Office
Washington, DC 20548




                                   October 16, 2012

                                   The Honorable Thomas R. Carper
                                   Chairman
                                   The Honorable Scott P. Brown
                                   Ranking Member
                                   Subcommittee on Federal Financial Management,
                                   Government Information, Federal Services,
                                       and International Security
                                   Committee on Homeland Security
                                       and Governmental Affairs
                                   United States Senate

                                   Spending on information technology (IT) represents a significant portion
                                   of the federal budget—estimated at $80 billion for fiscal year 2012. 1 More
                                   than 700 major investments account for approximately $40 billion of this
                                   IT spending. The Clinger-Cohen Act of 1996 charges the Director of the
                                   Office of Management and Budget (OMB) with responsibility for
                                   analyzing, tracking, and evaluating the risks and results of all major IT
                                   investments as part of the federal budget process, and reporting to
                                   Congress on the performance benefits achieved by these investments. 2
                                   The act also places responsibility for managing investments with the
                                   heads of agencies and establishes chief information officers (CIOs) to
                                   advise and assist agency heads in carrying out this responsibility.

                                   OMB launched the Federal IT Dashboard in June 2009 as a public
                                   website that reports performance and supporting data for the major IT
                                   investments. The Dashboard is to provide transparency for these
                                   investments in order to facilitate public monitoring of government
                                   operations and accountability for investment performance by the federal
                                   CIOs who oversee them. In January 2010, OMB began using the
                                   Dashboard as one of several tools to identify troubled investments. These
                                   investments became the focus of joint OMB-agency TechStat
                                   Accountability Sessions (TechStats)—evidence-based reviews intended
                                   to improve investment performance through concrete actions. In
                                   December 2010, OMB reported that these sessions resulted in $3 billion


                                   1
                                    As reported by agencies to the Office of Management and Budget.
                                   2
                                    40 U.S.C. § 11302(c).




                                   Page 1                                    GAO-13-98 Information Technology Dashboard
in reduced life-cycle costs and subsequently incorporated the TechStat
model into its 25-point plan for reforming federal IT management. 3 With
this plan, agency CIOs became responsible for leading TechStat sessions
at the department level, analyzing investments using data from the
Dashboard, and terminating or turning around at least one-third of
underperforming IT projects within 18 months. OMB reported its progress
on the plan, improvements to the Dashboard, and results of TechStat
sessions in the analytical perspectives it provided for the President’s 2012
and 2013 budget submissions. 4

In response to your request, our objectives for this review were to (1)
characterize the CIO ratings for selected federal agencies’ IT investments
as reported over time on the Dashboard, (2) determine how agencies’
approaches for assigning and updating CIO ratings vary, and (3) describe
the benefits and challenges associated with agencies’ approaches to the
CIO rating.

To establish the scope of our review, we selected six agencies that
spanned a range of IT spending for fiscal year 2011, including the three
highest spending agencies, two of the lowest, and an agency in the
middle. Collectively, these agencies accounted for approximately $51
billion, or 65 percent, of 2011 spending on IT investments. The six
agencies are the Department of Defense (DOD), Department of
Homeland Security (DHS), Department of Health and Human Services
(HHS), Department of the Interior (DOI), National Science Foundation
(NSF), and Office of Personnel Management (OPM).

To address our objectives, we downloaded CIO ratings and related data
reported for investments on the Dashboard and analyzed these data for
the period June 2009 to March 2012. 5 We did not independently evaluate
the ratings as reported by the agencies, but determined that they were
sufficiently complete and accurate for our analyses. We interviewed
agency officials, including CIOs where possible, and obtained written


3
 U.S. Chief Information Officer, 25 Point Implementation Plan to Reform Federal
Information Technology Management, The White House (Washington, D.C.: Dec. 9,
2010).
4
 Office of Management and Budget, Budget of the United States Government, Analytical
Perspectives. Fiscal Years 2012, 2013.
5
http://www.itdashboard.gov.




Page 2                                    GAO-13-98 Information Technology Dashboard
             responses and supporting documents, related agency policies,
             procedures, reported data, artifacts, as well as agency views on the
             benefits and challenges associated with performing these ratings and
             reporting them to the Dashboard. We also utilized recent GAO and DOD
             Inspector General reviews of DOD’s major IT investments and compared
             findings in these reports to the CIO ratings that the department submitted
             to the Dashboard. In addition, we analyzed OMB documentation and
             interviewed OMB staff to update our information on how the Dashboard
             has evolved, identify the guidance agencies received about CIO ratings,
             determine the efforts OMB has under way to improve the Dashboard, and
             describe the ways in which OMB is using the data to improve IT
             management.

             We conducted this performance audit from January 2012 through
             September 2012 in accordance with generally accepted government
             auditing standards. Those standards require that we plan and perform the
             audit to obtain sufficient, appropriate evidence to provide a reasonable
             basis for our findings and conclusions based on our audit objectives. We
             believe that the evidence obtained provides a reasonable basis for our
             findings and conclusions based on our audit objectives. Further details of
             our objectives, scope, and methodology are provided in appendix I.


             The Clinger-Cohen Act of 1996 requires OMB to establish processes to
Background   analyze, track, and evaluate the risks and results of major capital
             investments in information systems made by federal agencies and report
             to Congress on the net program performance benefits achieved as a
             result of these IT investments. 6 Further, the act places responsibility for
             managing investments with the heads of agencies and establishes CIOs
             to advise and assist agency heads in carrying out this responsibility.

             OMB established the Management Watch List in 2003 to help carry out its
             oversight role. The Management Watch List included mission-critical
             projects that needed to improve performance measures, project
             management, IT security, or overall justification for inclusion in the
             President’s budget submission. Further, in August 2005, OMB
             established a High-Risk List, which consisted of projects identified by
             federal agencies, with OMB’s input, as requiring special attention from



             6
             40 U.S.C. § 11302(c).




             Page 3                                GAO-13-98 Information Technology Dashboard
                               oversight authorities and the highest levels of agency management.
                               Between 2005 and 2009, OMB described its efforts to monitor and
                               manage risky federal IT investments in the annual budget submission.

                               Over the past several years, we have reported and testified on OMB’s
                               initiatives to highlight troubled IT projects, justify investments, and use
                               project management tools. 7 For instance, in 2006 we recommended that
                               OMB develop a single aggregated list of high-risk projects and their
                               deficiencies and use that list to report to Congress on progress made in
                               correcting high-risk problems. 8 As a result, OMB started publicly releasing
                               aggregate data on its Management Watch List and disclosing the
                               projects’ deficiencies. Moreover, between 2007 and 2009, the President’s
                               budget submission included an overview of investment performance over
                               several budget years, including the number of federal IT projects in need
                               of management attention. Such information helped Congress stay better
                               informed of high-risk projects and make related funding decisions. 9 With
                               the advent of its IT Dashboard in 2009, OMB discontinued this type of
                               reporting in the fiscal year 2010 budget submission.


OMB’s Dashboard                In June 2009, OMB deployed a public website to further improve the
Provides Visibility into the   transparency and oversight of agencies’ IT investments, replacing the
Performance of Federal IT      Management Watch List and High-Risk List. Known as the IT
                               Dashboard, 10 this site displays federal agencies’ cost, schedule, and
Investments                    performance data for over 700 major federal IT investments at 27 federal
                               agencies that are responsible for about $40 billion of the federal budget.
                               According to OMB, these data are intended to provide a near-real-time
                               perspective on the performance of these investments, as well as a
                               historical perspective. Further, the public display of these data is intended


                               7
                                GAO-09-624T; GAO, Information Technology: Treasury Needs to Better Define and
                               Implement Its Earned Value Management Policy, GAO-08-951 (Washington, D.C.: Sept.
                               22, 2008); and Air Traffic Control: FAA Uses Earned Value Techniques to Help Manage
                               Information Technology Acquisitions, but Needs to Clarify Policy and Strengthen
                               Oversight, GAO-08-756 (Washington, D.C.: July 18, 2008).
                               8
                                GAO, Information Technology: Agencies and OMB Should Strengthen Processes for
                               Identifying and Overseeing High Risk Projects, GAO-06-647 (Washington, D.C.: June 15,
                               2006).
                               9
                                Office of Management and Budget, Budget of the United States Government, Analytical
                               Perspectives. Fiscal Years 2005, 2006, 2007, 2008, 2009.
                               10
                                   http://www.itdashboard.gov.




                               Page 4                                    GAO-13-98 Information Technology Dashboard
to allow OMB; other oversight bodies, including Congress; and the
general public to hold the government agencies accountable for progress
and results.

OMB reported on plans and implementation progress for this
management tool in the “Analytical Perspectives” section of the
President’s budget submissions for fiscal years 2012 and 2013, 11
including planned updates to the Dashboard during 2012 to support
closer executive oversight and intervention to prevent schedule delays,
cost overruns, and failures in delivering key functionality needed by
federal programs. For example, it reported using the Dashboard to
identify investments for TechStat reviews.

The Dashboard visually presents performance ratings for agencies overall
and for individual investments using metrics that OMB has defined—cost,
schedule, and CIO evaluation. The website also provides the capability to
download certain data. Figure 1 is an example of an agency’s (OPM)
portfolio page as recently depicted on the Dashboard.




11
 Office of Management and Budget, Budget of the United States Government, Analytical
Perspectives. Fiscal Years 2012, 2013.




Page 5                                    GAO-13-98 Information Technology Dashboard
Figure 1: Example of an Agency Portfolio Page as Reported on OMB’s IT Dashboard Website, July 2012




                                       The Dashboard’s data spans the period from June 2009 to the present,
                                       and is based, in part, on each agency’s exhibit 53 and exhibit 300




                                       Page 6                                  GAO-13-98 Information Technology Dashboard
submissions 12 to OMB, as well as on agency assessments and
supporting information on each investment. Over the life of the
Dashboard, OMB has issued guidance to agencies on, among other
things, what data to report, how those data need to be structured and
formatted for upload to the Dashboard, and procedures for using the
Dashboard’s submission tools. For instance, OMB instructed agencies to
update and submit investment cost and schedule data monthly. OMB has
made various changes to the organization, available data, and features of
the Dashboard over time, including

•    improvements to Dashboard calculations to incorporate the variance
     of “in progress” milestones rather than just “completed” milestones;

•    web pages containing data on historical ratings and rebaselines of
     eliminated and downgraded investments;

•    added data on awarded contracts, with links to USAspending.gov;

•    release of IT Dashboard source code and documentation to an open
     source hosting provider;

•    enhancements to baseline history, which give users the ability to see
     field-by-field changes for each rebaseline;

•    a mechanism for OMB analysts to provide feedback to agencies on
     investment submissions; and

•    mobile-friendly formatting of Dashboard displays.

Once OMB has received agency-reported investment data, it converts
these into investment performance ratings for display on the dashboard
according to calculations and protocols described on its website. OMB
assigns cost and schedule performance ratings by using data submitted
by agencies to calculate variances between the planned cost or schedule
targets and the actual or projected cost or schedule values. OMB
converts these variances to percentages, and assigns the ratings to be


12
  Exhibit 53s list all of the IT projects and their associated costs within a federal
organization. An exhibit 300 is also called the Capital Asset Plan and Business Case. It is
used to justify resource requests for major IT investments and is intended to enable an
agency to demonstrate to its own management, as well as to OMB, that a major project is
well planned.




Page 7                                       GAO-13-98 Information Technology Dashboard
                          presented on the Dashboard within three ranges, red, yellow, and green,
                          as shown in table 1.

                          Table 1: Dashboard Variance and Rating Colors

                           Project level cost and schedule variance rating               Rating color
                           ≥ 30%                                                         Red
                           ≥ 10% and < 30%                                               Yellow
                           < 10%                                                         Green
                          Source: OMB’s IT Dashboard.


                          Although the thresholds for assigning cost and schedule variance ratings
                          has remained constant over the life of the Dashboard, the cost and
                          schedule data agencies are required to submit have changed in several
                          ways, as have the variance calculations. For example, in response to our
                          recommendations (further discussed in the next section), OMB changed
                          how the Dashboard calculates the cost and schedule ratings in July 2010,
                          to include “in progress” milestones rather than just “completed” ones for
                          more accurate reflection of current investment status.


GAO Has Previously        We have previously reported that OMB has taken significant steps to
Reported on the           enhance the oversight, transparency, and accountability of federal IT
Dashboard’s Value, Data   investments by creating its IT Dashboard, and by improving the accuracy
                          of investment ratings. We also found issues with the accuracy and data
Quality, and Recent       reliability of cost and schedule data, and recommended steps that OMB
Improvements              should take to improve these data.

                          In July 2010, we reported 13 that the cost and schedule ratings on OMB’s
                          Dashboard were not always accurate for the investments we reviewed,
                          because these ratings did not take into consideration current
                          performance. As a result, the ratings were based on outdated information.
                          We recommended that OMB report on its planned changes to the
                          Dashboard to improve the accuracy of performance information and
                          provide guidance to agencies to standardize milestone reporting. OMB
                          agreed with our recommendations and, as a result, updated the
                          Dashboard’s cost and schedule calculations to include both ongoing and




                          13
                            GAO-10-701.




                          Page 8                                    GAO-13-98 Information Technology Dashboard
                            completed activities. Similarly, in March 2011, we reported 14 that OMB
                            had initiated several efforts to increase the Dashboard’s value as an
                            oversight tool, and had used its data to improve federal IT management.
                            We also reported, however, that agency practices and the Dashboard’s
                            calculations contributed to inaccuracies in the reported investment
                            performance data. For instance, we found missing data submissions or
                            erroneous data at each of the five agencies we reviewed, along with
                            instances of inconsistent program baselines and unreliable source data.
                            As a result, we recommended that the agencies take steps to improve the
                            accuracy and reliability of their Dashboard information, and that OMB
                            improve how it rates investments relative to current performance and
                            schedule variance. Most agencies generally concurred with our
                            recommendations; OMB agreed with our recommendation for improving
                            ratings for schedule variance. It disagreed with our recommendation to
                            improve how it reflects current performance in cost and schedule ratings,
                            but more recently made changes to Dashboard calculations to address
                            this while also noting challenges in comprehensively evaluating cost and
                            schedule data for these investments.

                            More recently, in November 2011, we reported 15 that the accuracy of
                            investment cost and schedule ratings had improved since our July 2010
                            report because OMB had refined the Dashboard’s cost and schedule
                            calculations. Most of the ratings for the eight investments we reviewed
                            were accurate, although we noted that more could be done to inform
                            oversight and decision making by emphasizing recent performance in the
                            ratings. We recommended that the General Services Administration
                            comply with OMB’s guidance for updating its ratings when new
                            information becomes available (including when investments are
                            rebaselined) and the agency concurred. Since we previously
                            recommended that OMB improve how it rates investments, we did not
                            make any further recommendations.


CIO Ratings Are Important   Unlike the Dashboard’s cost and schedule ratings, which are derived by
for OMB’s IT Reform and     OMB based on agency-submitted data, the “Investment Evaluation by
Management Initiatives      Agency CIO” (also called the CIO rating) is determined by agency
                            officials; OMB translates the agency’s numerical assignment for an


                            14
                             GAO-11-262.
                            15
                             GAO-12-210.




                            Page 9                              GAO-13-98 Information Technology Dashboard
investment into a color for depiction on the Dashboard. An OMB staff
member from the Office of E-Government and Information Technology
noted that the CIO rating should be a current assessment of future
performance based on historical results and is the only Dashboard
performance indicator that has been defined and produced the same way
since the Dashboard’s inception. According to OMB’s instructions, a CIO
rating should reflect the level of risk facing an investment on a scale from
1 (high risk) to 5 (low risk) relative to that investment’s ability to
accomplish its goals. Each agency CIO is to assess their IT investments
against a set of six preestablished evaluation factors identified by OMB
(shown in table 2) and then assign a rating of 1 to 5 based on his or her
best judgment of the level of risk facing the investment. According to an
OMB staff member, agency CIOs are responsible for determining
appropriate thresholds for the risk levels and for applying them to
investments when assigning CIO ratings.

Table 2: Investment Evaluation Factors Identified by OMB for Assigning CIO
Ratings

 Evaluation factor            Supporting examples
 Risk management              Risk management strategy exists
                              Risks are well understood by senior leadership
                              Risk log is current and complete
                              Risks are clearly prioritized
                              Mitigation plans are in place to address risks
 Requirements                 Investment objectives are clear and scope is controlled
 management                   Requirements are complete, clear and validated
                              Appropriate stakeholders are involved in requirements definition
 Contractor                   Acquisition strategy is defined and managed via an Integrated
 oversight                    Program Team
                              Agency receives key reports, such as earned value reports, current
                              status, and risk logs
                              Agency is providing appropriate management of contractors such
                              that the government is monitoring, controlling, and mitigating the
                              impact of any adverse contract performance
 Historical                   No significant deviations from planned cost and schedule
 performance                  Lessons learned and best practices are incorporated and adopted
 Human capital                Qualified management and execution team for the IT investments
                              and/or contracts supporting the investment
                              Low turnover rate
 Other                        Other factors that the CIO deems important to forecasting future
                              success
Source: OMB’s IT Dashboard.




Page 10                                              GAO-13-98 Information Technology Dashboard
OMB recommends that CIOs consult with appropriate stakeholders in
making their evaluation, including Chief Acquisition Officers, program
managers, and other interested parties. Ultimately, CIO ratings are
assigned colors for presentation on the Dashboard, according to the five-
point rating scale, as illustrated in table 3.

Table 3: IT Dashboard CIO Rating Colors, Based on a Five-Point Scale for CIO
Ratings

 Rating (by agency CIO)                                         Color
 5-Low risk                                                     Green
 4-Moderately low risk                                          Green
 3-Medium risk                                                  Yellow
 2-Moderately high risk                                         Red
 1-High risk                                                    Red
Source: OMB’s IT Dashboard.


OMB has made the CIO’s evaluation and rating a key component of its
larger IT Reform Initiative and 25 Point Plan. In its plan, OMB reported
that it used agencies’ CIO ratings to select investments for the TechStat
review sessions it conducted between 2010 and 2011. These sessions
are data-driven assessments of IT investments by agency leaders that
are intended to result in concrete action to improve performance. OMB
reported that the TechStats it conducted on selected investments resulted
in approximately $3 billion in reduced costs. Building on the results of
those sessions, the plan articulates a strategy for strengthening IT
governance, in part, through the adoption of the TechStat model by
federal agencies. In conducting TechStats, agencies are to rely, in part,
on CIO ratings from the IT Dashboard. The TechStat Toolkit, developed
by OMB and a task force of agency leads, provides sample questions
regarding an investment’s CIO rating and associated risks for use in
TechStat sessions.

Furthermore, OMB issued guidance in August 2011 16 that stated, among
other things, that agency CIOs shall be held accountable for the
performance of IT program managers based on their governance process
and the data reported on the IT Dashboard, which includes the CIO



16
  Chief Information Officer Authorities, Memorandum for Heads of Executive Departments
and Agencies, M-11-29 (Washington, D.C.: Aug. 8, 2011).




Page 11                                    GAO-13-98 Information Technology Dashboard
rating. According to OMB, the addition of CIO names and photos on
Dashboard investments is intended to highlight this accountability and link
it to the Dashboard’s reporting on investment performance. Figure 2
illustrates the CIO rating information presented on the Dashboard for an
example IT investment.




Page 12                              GAO-13-98 Information Technology Dashboard
Figure 2: Example of an Agency’s IT Investment Page from the IT Dashboard Website, July 2012




                                        Page 13                                 GAO-13-98 Information Technology Dashboard
                       As of March 2012, CIO ratings for most investments listed on the
CIOs Rated Most IT     Dashboard for the six agencies we reviewed indicated either low risk or
Investments as Low     moderately low risk (223 out of 313 investments across all the selected
                       agencies). High risk or moderately high risk ratings were assigned to
Risk or Moderately     fewer investments (12 out of 313 investments across all the selected
Low Risk; Agencies     agencies). Figure 3 presents the total number of IT investments rated on
Had Mixed Results in   the Dashboard for each of the selected agencies according to their risk
                       levels, as of March 2012, and illustrates the predominance of low risk
Reducing Higher Risk   investments for the agencies in our review. The figure also reports
Levels                 agencies’ budgets for their major IT investments for fiscal year 2012, as
                       presented on the Dashboard.


                       Figure 3: CIO Ratings and Associated IT Budgets for Fiscal Year 2012 Dashboard
                       Investments from Selected Agencies, as of March 2012




                       Note: Figure does not include downgraded or eliminated investments. Budget figures represent
                       spending on major IT investments for fiscal year 2012.

                       Historically, over the life of the Dashboard from June 2009 to March 2012,
                       low or moderately low risk ratings accounted for at least 66 percent of all
                       ratings at five of the six agencies (the exception is DHS with 51 percent).


                       Page 14                                          GAO-13-98 Information Technology Dashboard
                                           Medium risk ratings accounted for between 0 to 38 percent of all reported
                                           ratings across agencies during this period. The maximum percentage of
                                           ratings in the high risk or moderately high risk categories for any agency
                                           during this 34-month period was 12 percent, with two agencies—DOD
                                           and NSF—reporting no high risk investments. 17 DOD stated in written
                                           comments that this was because they did not deem any of their
                                           investments to be high risk. (DOD’s investment risks are further
                                           discussed in the next section.) An NSF official from the Division of
                                           Information Systems stated that there were no high risk investments
                                           because most of their investments were in the operations and
                                           maintenance phase. Table 4 presents the average composition of ratings
                                           for each agency during the reporting period of June 2009 to March 2012.
                                           Appendix III depicts each agency’s CIO ratings by risk level on a monthly
                                           basis during the reporting period.

Table 4: Average Composition of CIO Ratings for Agencies’ Major IT Investments from June 2009 through March 2012

                                                                                     DOD           DHS        HHS        DOI      OPM     NSF
High risk and moderately high risk investments                                         0%           11%         5%        4%      12%      0%
Medium risk investments                                                              15%            38%       21%        13%      22%      0%
Low risk and moderately low risk investments                                         85%            51%       74%        83%      66%    100%
Range in the number of major IT investments rated on the
Dashboard during this period                                                        49-87          63-83     55-81      33-48      6-9     3-5
                                           Source: GAO analysis of data from OMB’s IT Dashboard.


                                           Note: Table does not include downgraded or eliminated investments.

                                           Overall, the CIO rating remained constant for 147 of 313 investments that
                                           were active as of March 2012 (about 47 percent of the investments we
                                           reviewed). These investments were rated at the same risk level during
                                           every rating period (see fig. 4).




                                           17
                                             CIO ratings for investments that were downgraded (no longer defined as a major
                                           investment) or eliminated during this period are not included in these percentages.




                                           Page 15                                                    GAO-13-98 Information Technology Dashboard
Figure 4: Percentages of Major IT Investments at Selected Agencies with CIO
Ratings That Changed Compared to Ratings That Remained Constant, Initial Rating
through March 2012




Note: Figure does not include downgraded or eliminated investments.


Four of the six agencies did not change the CIO rating for a majority of
their investments (excluding any investments that were downgraded or
eliminated) during the time frame we examined. In contrast, the other two
agencies—OPM and HHS—changed the CIO rating for more than 70
percent of their investments at least once between the investment’s initial
rating and the rating reported as of March 2012. Table 5 lists the number
of each agency’s investments whose ratings were constant and changed
over time.

Table 5: Number of Major IT Investments with CIO Ratings That Changed Compared
to Those That Remained Constant, Initial Rating through March 2012

                                             DOD        DHS     HHS       DOI     OPM       NSF
 Remained constant                               51      42       17       31         2        4
 Changed                                         36      41       64       17         7        1
 Total                                           87      83       81       48         9        5
Source: GAO analysis of data from OMB’s IT Dashboard.


Note: Table does not include downgraded or eliminated investments.




Page 16                                                  GAO-13-98 Information Technology Dashboard
Agencies offered several reasons for why many investments had no
changes in their CIO ratings during their entire time on the Dashboard.
Five of the six selected agencies indicated that many investments were in
a steady-state or operations and maintenance phase with no new
development. One agency reported that their investments’ CIO ratings
remained constant because the investments consistently met all
requirements and deadlines and were using project management best
practices.

The agencies we reviewed showed mixed results in reducing the number
of higher risk investments during the rating period. 18 For investments
whose rating changed at least once during the period, 40 percent (67
investments) received a lower risk rating in March 2012 than they
received initially, 41 percent of investments (68 investments) received a
higher risk rating, and the remaining 19 percent (31 investments) received
the same rating in March 2012 as they had initially received, despite
whatever interim changes may have occurred (i.e., there was no “net”
change to their reported risk levels). (See fig. 5.)




18
  A reduction in risk level is indicated by a higher CIO rating. An increase in risk level is
indicated by a lower CIO rating.




Page 17                                         GAO-13-98 Information Technology Dashboard
Figure 5: Percentages of Major IT Investments at Selected Agencies with Changes
in CIO Ratings, Initial Rating Compared to March 2012




Note: Figure does not include downgraded or eliminated investments. Percentage represents initial
CIO rating compared to the CIO rating as of March 2012 and does not represent any fluctuations that
may have occurred in between that time frame.


Two agencies—DHS and OPM—reported more investments with reduced
risk in March 2012, as compared with initial ratings. The other four
agencies reported more investments with increased risk. Table 6 presents
net changes in risk levels at each of the selected agencies (among
investments that were not downgraded or eliminated). Appendix III
graphically summarizes these data for all six agencies.

Table 6: Net Changes in Investment Risk, Based on Comparison of Initial Rating to
March 2012

                                                              DOD       DHS   HHS   DOI   OPM     NSF
 Risk level reduced                                                15    26    17     4       5      0
 Risk level increased                                              18    12    25    11       1      1
 No net change                                                     3      3    22     2       1      0
 Total number of changed investments                               36    41    64    17       7      1
Source: GAO analysis of data downloaded from OMB’s IT Dashboard.

Note: Table does not include downgraded or eliminated investments. Table represents investments’
initial CIO rating compared to the CIO rating as of March 2012 and does not represent any
fluctuations that may have occurred in between that time frame.




Page 18                                                      GAO-13-98 Information Technology Dashboard
Agencies most commonly cited additional oversight or program reviews
as factors that contributed to decreased risk levels. Specifically, agencies
commented that the CIO ratings and Dashboard reporting had spurred
improved program management and risk mitigation. For example, one
agency’s officials commented that the CIO now closely monitors the
monthly performance and risk data generated by their investments, and
that the additional oversight has brought about strengthened processes
and more focused attention to issues.

In contrast, several agencies cited generally poor risk management at the
investment level, the introduction of new investment/programs risks, as
well as instances of poor project management as factors contributing to
increased risk for investments. For example, one agency responded that
internal review findings revealed new risks that caused an investment’s
risk level to increase. Another agency’s officials reported that various
technical issues caused one of their investments to fall behind schedule,
thus increasing risk.

Both OMB and several agencies suggested caution in interpreting
changing risk levels for investments. They noted that an increase in an
investment’s risk level can sometimes indicate better management by the
program or CIO because previously unidentified risks have been
assessed and included in the CIO evaluation. Conversely, a decrease in
an investment’s risk level may not indicate improved management if the
data and analysis on which the CIO rating are based is incomplete,
inconsistent, or outdated.

Further analysis of the characteristics and causes of Dashboard’s CIO
ratings, and reporting on the patterns of risk within and among agencies,
could provide Congress and the public with additional perspectives on
federal IT investment risk over time. However, for the past four budget
submissions, OMB has not summarized the extent of risk represented by
major federal IT investments in the analysis it prepares annually for the
President’s budget submission, as it did prior to the fiscal year 2010
submission. As a result, OMB is missing an opportunity to integrate such
risk assessments into its evaluation of major capital investments in
reporting to Congress.




Page 19                               GAO-13-98 Information Technology Dashboard
                      OMB has provided agencies with instructions for assigning CIO ratings for
Most Agencies         the major IT investments reported on the Dashboard. Specifically, OMB’s
Established           instructions state that agency CIOs should rate each investment based on
                      his/her best judgment and should
Approaches for CIO
Ratings That Follow      include input from stakeholders, such as Chief Acquisition Officers,
                          program managers, and others;
OMB’s Instructions
                         update the rating as soon as new information becomes available that
                          might affect the assessment of a given investment; and

                         utilize OMB’s investment rating factors, including: risk management,
                          requirements management, contractor oversight, historical
                          performance, and human capital, as well as any other factors deemed
                          relevant by the CIO.

                      Despite differences in the specific inputs and processes used, agencies
                      generally followed OMB’s instructions for assigning CIO ratings. However,
                      DOD’s ratings reflected additional considerations beyond OMB’s
                      instructions and did not reflect available information about significant risks
                      for certain investments. The sections that follow describe how each
                      agency addressed OMB’s instructions.

                      Include input from stakeholders. Each of the six agencies we reviewed
                      relied on stakeholder input, at least in part, when assigning CIO ratings.
                      Agencies also cited a variety of review boards, data from program and
                      financial systems, and other investment assessments as inputs to the
                      rating. Table 7 describes the data and processes that agencies reported
                      using when they derived their CIO ratings.




                      Page 20                               GAO-13-98 Information Technology Dashboard
Table 7: Data Sources and Derivation of CIO Ratings by Selected Agencies

Agency     Data sources used in CIO rating                                      How rating is derived
DOD        Input and recommendations from a variety of                          The DOD CIO reviews investment performance information and
           stakeholders, including functional leads, acquisition                recommendations from stakeholders and makes a subjective
           community, component CIOs, and cost and schedule                     evaluation of the investment based upon those inputs.
           experts
DHS        Data from internal investment management systems                     The DHS CIO determines the rating based upon information
           and the department’s periodic reporting system,                      and recommendation gathered by an analysis team from the
           program management reviews, CIO portfolio reviews,                   department’s Enterprise Business Management Office.
           and input from the executive steering committees
HHS        Cost and schedule variance data, exhibit 300 Quality                 The HHS CIO determines the CIO ratings using a score
           Review, number of rebaselines, and investment                        calculated from the data.
           manager self assessment
DOI        Bureau-proposed rating of investment, Electronic                     One of the processes the agency follows begins with the bureau
                                                          a
           Capital Planning Investment Control Technology data,                 responsible for the investment completing a template to
                b
           iStat performance reviews, applicable GAO or Office of               document its proposed bureau rating of the investment and
           Inspector General findings                                           submits the proposed rating to the department CIO through the
                                                                                Capital Planning and Investment Control office. The office
                                                                                reviews and analyzes the data, compares the data against
                                                                                known information, such as iStat performance reviews or
                                                                                applicable GAO or Office of Inspector General findings
                                                                                regarding the current state of the investment, and provides a
                                                                                recommendation to the CIO, who determines the rating.
                                                                                Additionally, the CIO can make the decision to change the
                                                                                rating outside of this process.
NSF        IT portfolio management tool; program manager and                    NSF staff meets monthly with the CIO to discuss investment
           stakeholder input from monthly reviews where budget,                 data and review evaluations using OMB’s guidance. The staff
           schedule, and risks are reviewed                                     assesses and ranks the investments. The CIO issues the rating
                                                                                based upon this information.
OPM        IT security rating; enterprise architecture rating; earned           The IT Investment Management group reports to the CIO on
           value management program manager rating; risk                        collected information regarding the investments. The group
           management rating; Electronic Capital Planning                       produces a chart that tracks 20 different metrics; each metric is
           Investment Control Technology; cost, schedule, and                   presented as either a red, yellow, or green rating. The CIO
           overall variance rating, IT Dashboard cost and schedule              issues the rating based upon this information.
           variance; and program managers’ self-assessments
                                            Source: GAO analysis of agency-reported data.

                                            a
                                             Electronic Capital Planning Investment Control Technology is a government-owned technology
                                            system that is designed to help federal agencies in the management and control of their initiatives,
                                            portfolios, and investment priorities. This web-based application assists managers and staff involved
                                            in IT planning in assessing IT investments in terms of their costs, risks, and expected returns.
                                            b
                                            iStat is DOI’s internal investment review process, which was modeled after OMB’s TechStat process.

                                            Update CIO ratings. All six agencies established guidelines for
                                            periodically reviewing and updating their CIO ratings. Specifically, HHS,
                                            NSF, DOI, and OPM reported that they update CIO ratings on a monthly
                                            basis. DOD has adopted a quarterly update cycle, although an official
                                            noted that the actual process of collecting information and evaluating
                                            investments for the ratings takes slightly longer than 3 months. DHS


                                            Page 21                                                 GAO-13-98 Information Technology Dashboard
officials with the Office of the CIO stated that the frequency of its updates
varies based on the risk level of an investment’s previous rating:
investments with a previous CIO rating of green are to be reviewed
semiannually; yellow investments are to be reviewed quarterly; and red
investments are to be reviewed monthly.

Utilize OMB’s investment rating factors. Most of the selected agencies
use OMB’s investment rating factors when evaluating their investments.
Only one agency (HHS) does not use all of them. Specifically, an HHS
official from the Office of the CIO told us that human capital issues are not
explicitly covered in their CIO rating criteria because investment owners
are to provide adequate IT human capital, and that these owners will
reflect any issues that arise when providing input for the CIO rating.

Among the agencies we reviewed, DOD was unique in that its ratings
reflected additional considerations beyond OMB’s instructions. For
example, briefing slides prepared for DOD’s 2011 CIO rating exercise
identified the need to “balance” CIO ratings, and advised that yellow or
red ratings could lead to an OMB review. 19 In addition, DOD officials
explained that the department rated investments green (or low risk) if the
risk of the investment not meeting its performance goals is low; yellow (or
medium risk) if the investment is facing difficulty; and red (high risk) only if
the department planned to restructure or cancel the investment, or had
already done so. DOD officials further stated that their CIO ratings
provide a measured assessment of how DOD believes an investment will
perform in the future.

Although the CIO ratings submitted by DOD to the Dashboard are
consistent with their ratings approach, they do not reflect other available
information about the risk of these investments. As we previously noted,
none of DOD’s investments that were active in March 2012 were rated as
high risk, and approximately 85 percent were rated as either low risk or
moderately low risk throughout their time on the Dashboard. However,
these ratings did not always reflect significant schedule delays, cost
increases, and other weaknesses identified for certain investments in our




19
   Office of the DOD CIO, “Summary of June 2011 Recommended Rating Changes”
(including Backup Slides), June 2011 Update.




Page 22                                 GAO-13-98 Information Technology Dashboard
recent reviews, or problems with those investments identified in a recent
report by the DOD Inspector General. 20

Based on the department’s long-standing difficulties with such programs,
we designated DOD business systems modernization as a high-risk area
in 1995 and it remains a high-risk area today. More recently, we reported
weaknesses in several of the department’s business system
investments. 21 Specifically, we reported that the department had not
effectively ensured that these systems would deliver capabilities on time
and within budget; that acquisition delays required extended funding for
duplicative legacy systems; that delays and cost overruns were likely to
erode the cost savings these systems were to provide; and that,
ultimately, DOD’s management of these investments was putting the
department’s transformation of business operations at risk.

Although the following selected examples of DOD investments
experienced significant performance problems and were included with
those considered to be high-risk business system investments in our
recent reviews of those systems, they were all rated low risk or
moderately low risk by the DOD CIO.

•    Air Force’s Defense Enterprise Accounting and Management
     System (DEAMS): DEAMS is the Air Force’s target accounting
     system designed to provide accurate, reliable, and timely financial
     information. In early 2012, GAO reported that DEAMS faced a 2-year
     deployment delay, an estimated cost increase of about $500 million
     for an original life-cycle cost estimate of $1.1 billion (an increase of
     approximately 45 percent), and that assessments by DOD users had
     identified operational problems with the system, such as data
     accuracy issues, an inability to generate auditable financial reports,




20
 Department of Defense Office of Inspector General, Enterprise Resource Planning
Systems Schedule Delays and Reengineering Weaknesses Increase Risks to DOD’s
Auditability Goals. DODIG-2012-111 (Alexandria, Va.: July 13, 2012).
21
 GAO, DOD Business Transformation: Improved Management Oversight of Business
System Modernization Efforts Needed, GAO-11-53 (Washington, D.C.: Oct. 7, 2010) and
DOD Financial Management: Reported Status of Department of Defense’s Enterprise
Resource Planning Systems, GAO-12-565R (Washington, D.C: Mar. 30, 2012).




Page 23                                   GAO-13-98 Information Technology Dashboard
      and the need for manual workarounds. 22 In July 2012, the DOD
      Inspector General reported that the DEAMS’ schedule delays were
      likely to diminish the cost savings it was to provide, and would
      jeopardize the department’s goals for attaining an auditable financial
      statement. DOD’s CIO rated DEAMS low risk or moderately low risk
      from July 2009 through March 2012.

•     Army’s General Fund Enterprise Business System (GFEBS):
      GFEBS is an Army financial management system intended to improve
      the timeliness and reliability of financial information and to support the
      department’s auditability goals. In early 2012, we reported that
      GFEBS faced a 10-month implementation delay, and that DOD users
      reported operational problems, including deficiencies in data accuracy
      and an inability to generate auditable financial reports. 23 These
      concerns were reiterated by the DOD Inspector General in July 2012.
      DOD’s CIO rated GFEBS as moderately low risk from July 2009
      through March 2012.

•     Army’s Global Combat Support System-Army (GCSS-Army):
      GCSS-Army is intended to improve the Army’s supply chain
      management capabilities and provide accurate equipment readiness
      status reports, among other things. In March 2012, we reported that
      GCSS-Army was experiencing a cost overrun of approximately $300
      million on an original life-cycle cost estimate of $3.9 billion (an
      increase of approximately 8 percent) and a deployment delay of
      approximately 2 years. 24 DOD rated GCSS-Army as low or
      moderately low risk from July 2009 through March 2012.

Explanations submitted by DOD with the CIO ratings for these
investments did not provide meaningful insight for why they were rated at




22
  GAO, DOD Financial Management: Reported Status of Department of Defense’s
Enterprise Resource Planning Systems, GAO-12-565R (Washington, D.C.: Mar. 30, 2012)
and DOD Financial Management: Implementation Weaknesses in Army and Air Force
Business Systems Could Jeopardize DOD’s Auditability Goals, GAO-12-134 (Washington,
D.C.: Feb. 28, 2012).
23
    GAO-12-565R and GAO-12-134.
24
    GAO-12-565R.




Page 24                                  GAO-13-98 Information Technology Dashboard
                       the lowest risk levels in the face of known issues. 25 DOD officials told us
                       that they rated these investments as low risk because, in their view, the
                       cost and schedule variances listed above did not constitute significant
                       risks. Officials explained that: (1) the cost variances were not that large
                       compared to DOD’s overall size and large amount of IT spending; (2) the
                       schedule variance needed to be understood in the context that the
                       average DOD large-scale IT program takes 7 years (or 84 months) to
                       implement; and (3) that each of those programs had risk mitigation plans
                       in place. However, the first two reasons are inconsistent with DOD’s own
                       risk management guidance, 26 which recommends that risks be assessed
                       against the program’s own cost and schedule estimates, not other
                       department investments. In addition, completing risk mitigation plans
                       does not necessarily lower investment risk. DOD’s guidance calls for
                       implementing the mitigation plan and then reassessing resulting changes
                       to the risk. Even if the department adopts these elements of its own
                       guidance, the CIO’s evaluation will be incomplete unless it also reflects
                       the assessments of investment performance and risks identified by us
                       and others. Until the department does so, CIO ratings for DOD’s
                       Dashboard investments may not be sufficiently accurate or useful for its
                       TechStat sessions or OMB’s management and oversight.


                       Selected agencies identified various benefits associated with performing
CIO Ratings Present    CIO ratings and Dashboard reporting in general. Almost all of the
Benefits and           agencies (five of six) reported the following three benefits.
Challenges for         •    Increased quality of investment performance data. For example, one
Agencies’ Investment        agency also reported that the Dashboard has made information about
                            investments more understandable.
Management
                       •    Greater transparency and visibility for CIOs and their staff into
                            investment- and program-level performance data. One agency
                            reported that its CIO was better able to conduct reviews with actual
                            investment numbers, as opposed to self-reported data presented by


                       25
                         Explanations included “program is being closely monitored” (Defense Enterprise
                       Accounting and Management System), “program schedule being closely monitored”
                       (General Fund Enterprise Business System), and “no outstanding issues or concerns”
                       (Global Combat Support System-Army).
                       26
                         Department of Defense, Risk Management Guide for DOD Acquisition, Sixth Edition
                       (Version 1.0), August 2006.




                       Page 25                                    GAO-13-98 Information Technology Dashboard
    the investment’s program managers. Agencies could also compare
    their investments’ ratings to those of other agencies and departments.

•   Increased focus on project management practices. Two agencies
    reported improved investment performance as a direct result of their
    Dashboard rating and reporting activities; another stated that
    Dashboard reporting supported and reinforced their existing IT
    governance, capital planning, and program management processes.

Some of these benefits were interrelated. Several agencies viewed the
improved data quality as a by-product of greater scrutiny brought about
by having to report such data to the Dashboard on a regular basis. One
agency response noted that their program managers were surprised to
see the extent to which investment data were visible to the public, and
that this visibility motivated their staff to provide accurate and timely data
(which has improved data quality). Another agency noted that the visibility
of the IT Dashboard has increased awareness among investment and
project managers about the need to improve the planning of project
activities and the definition of operational performance metrics (which
support program management).

Nevertheless, agencies also identified challenges associated with
producing and reporting CIO ratings. First, three agencies reported a
challenge associated with the time and effort required to gather, validate,
and gain internal approval for CIO ratings and other data reported to the
Dashboard. For example, one agency reported that, due to the number of
organizations involved and the number of investments being evaluated, it
generally takes 90 to 120 days to develop and update its CIO ratings. The
agency further reported that this effort was separate from (and in addition
to) time it already spends on its own internal processes for managing and
overseeing acquisition programs.

Second, four of the six agencies identified challenges with the number of
changes OMB has made to the Dashboard, as well as with the timeliness
and clarity of OMB’s communication regarding those changes. For
example, officials at one agency commented that the frequency of
changes has actually hindered their efforts to improve data quality, since
errors sometimes resulted when it adapted to changes required by OMB.
Officials at another agency stated that OMB allowed insufficient time for
agencies to test their systems’ interfaces with the Dashboard when
changes were made, which they said resulted in data errors and
challenges for staff. These officials also noted that OMB’s guidance for
agency submissions has, at times, not matched the technical data



Page 26                                GAO-13-98 Information Technology Dashboard
              schemas implemented by OMB, impeding agencies’ efforts to
              successfully upload their data. An OMB staff member commented that
              their office releases changes to the Dashboard as early in the fiscal year
              as possible to give agencies time to adjust and that OMB announces
              planned changes to agencies before they are implemented via the
              Dashboard’s interagency web portal. OMB has recently held meetings
              with agency officials to discuss these issues and determine ways to better
              communicate going forward.

              Finally, one agency responded that while monthly updates to the
              Dashboard have increased investment and project managers’ attention to
              the performance of their investments and projects, this regular scrutiny
              could encourage investment and project managers to “perform to the test”
              rather than concentrate on effective investment and project management.
              However, based on the interrelationships of the benefits of CIO ratings
              identified by some agencies, the process of generating and reporting CIO
              ratings does not have to be just a grading exercise. As previously noted,
              the benefit of improved investment performance data for the CIO’s
              investment evaluation can lead to more effective management, which
              could, in turn, improve investment performance. Executives and staff who
              can envision these results from the Dashboard’s CIO evaluations may be
              less likely to view the additional time and effort required to generate the
              CIO ratings as a challenge, but as an opportunity for more efficient and
              effective management.


              Since its inception in 2009, the Federal IT Dashboard has increased the
Conclusions   transparency of the performance of major federal IT investments. Its CIO
              ratings, in particular, have improved visibility into changes in the risk
              levels of agencies’ investments over time. Determining whether such
              changes represent improvements or deficiencies in management and
              oversight can be difficult without additional information on investment
              performance and the rating process, but analyzing and reporting the
              ratings for investments and agencies over time for the President’s budget
              submission could help OMB ensure that risk is accurately assessed and
              that patterns of risk deserving of special management attention are
              identified.

              DOD demonstrated one such pattern of interest in its CIO ratings. During
              the 34-month life of the Dashboard, none of the 87 investments that were
              active as of March 2012 were rated high risk or moderately high risk, and
              approximately 85 percent of ratings were low risk or moderately low risk.
              Although DOD implemented OMB’s broad instructions for producing CIO


              Page 27                              GAO-13-98 Information Technology Dashboard
                      ratings, it also considered how the ratings might increase the likelihood of
                      an OMB review of an investment and minimized the effects of significant
                      schedule delays and cost increases, which were identified in our reviews
                      and those of DOD’s Inspector General. As a result, DOD is masking
                      significant investment risks, has not employed its own risk management
                      guidance, and has not delivered the transparency intended by the
                      Dashboard. By incorporating the results of external reviews into its
                      evaluations, DOD can further improve the quality of the information on
                      which investment risk ratings are based.

                      Beyond the transparency they promote, CIO ratings present an
                      opportunity to improve the data and processes agencies use to assess
                      investment risk. Some agencies have already experienced collateral
                      benefits and management results from their risk evaluations. Continuing
                      focus from OMB and agencies on how to accurately portray and derive
                      value from the ratings and the associated processes could enable
                      agencies to experience such benefits.


                      To ensure that OMB’s preparation of the President’s budget submission
Recommendations for   accurately reflects the risks associated with all major IT investments, we
Executive Action      are recommending that the Federal CIO analyze agency trends reflected
                      in Dashboard CIO ratings, and present the results of this analysis with the
                      President’s annual budget submission.

                      To ensure that DOD’s CIO evaluations of investment risk for its major IT
                      Dashboard investments reflect all available performance assessments
                      and are consistent with the department’s own guidance for managing risk,
                      we are recommending that the Secretary of Defense direct the
                      department’s CIO to reassess the department’s considerations for
                      assigning CIO risk levels for Dashboard investments, including
                      assessments of investment performance and risk from outside the
                      programs, and apply the appropriate elements of the department’s risk
                      management guidance to OMB’s evaluation factors in determining CIO
                      ratings.


                      We provided a draft of our report to the six agencies selected for our
Agency Comments       review and to OMB. In oral comments, staff from OMB’s Office of E-
and Our Evaluation    Government & Information Technology stated that OMB concurred with
                      our recommendation that the Federal CIO analyze agency trends
                      reflected in Dashboard CIO ratings and present the results of this analysis
                      with the President’s annual budget submission. OMB staff also provided


                      Page 28                               GAO-13-98 Information Technology Dashboard
technical comments, which we incorporated as appropriate. In a written
response, DOD’s Deputy Chief Information Officer for Information
Enterprise agreed with our recommendation that the department’s CIO
reassess considerations for assigning CIO risk levels for Dashboard
investments, and committed to updating the department’s CIO ratings
process to better report risk and improve the timeliness and transparency
of reporting. DOD’s written response is reprinted in Appendix IV. Officials
at DOI provided technical comments, which we incorporated as
appropriate. The remaining agencies had no comment on the draft report.


As agreed with your offices, unless you publicly announce the contents of
this report earlier, we plan no further distribution until 30 days from the
report date. At that time, we will send copies to interested congressional
committees; the Secretaries of Defense, Interior, Homeland Security,
Health and Human Services, the Director of the National Science
Foundation, the Director of the Office of Personnel Management, the
Director of the Office of Management and Budget; and other interested
parties. In addition, the report will be available at no charge on the GAO
website at http://www.gao.gov.

If you or your staffs have any questions on the matters discussed in this
report, please contact David A. Powner at (202) 512-9286 or by e-mail at
pownerd@gao.gov. Contact points for our Offices of Congressional
Relations and Public Affairs may be found on the last page of this report.
GAO staff who made major contributions to this report are listed in
appendix IV.




David A. Powner
Director, Information Technology
   Management Issues




Page 29                              GAO-13-98 Information Technology Dashboard
Appendix I: Objectives, Scope, and
              Appendix I: Objectives, Scope, and
              Methodology



Methodology

              Our objectives were to (1) characterize the Chief Information Officer (CIO)
              ratings for selected federal agencies’ information technology (IT)
              investments as reported over time on the federal IT Dashboard; (2)
              determine how agencies’ approaches for assigning and updating CIO
              ratings vary; and (3) describe the benefits and challenges associated with
              agencies’ approaches to the CIO rating.

              To establish the scope of our review, we downloaded and examined data
              on total IT spending for fiscal year 2011 for the 27 agencies reported on
              the IT Dashboard. (The Office of Management and Budget (OMB)
              extracts these data based on exhibit 300 forms submitted by each
              agency.) We then selected six agencies that spanned a range of IT
              spending for fiscal year 2011, including the three highest spending
              agencies, two of the lowest, and an agency in the middle. Collectively,
              these agencies accounted for approximately $51 billion, or 65 percent, of
              2011 spending on IT investments. The six agencies are the Department
              of Defense, Department of Homeland Security, Department of Health and
              Human Services, Department of the Interior, National Science
              Foundation, and Office of Personnel Management. The results in this
              report represent only these agencies.

              To address the first objective, we downloaded and examined the
              Dashboard’s CIO ratings for all investments at the six agencies we
              selected (a total of approximately 308 investments reported by these
              agencies). 1 To characterize the numbers and percentages of major IT
              investments at each risk level at each of our subject agencies, we
              analyzed, summarized, and—where appropriate—graphically depicted
              average CIO ratings for investments by agencies over time during the
              period from June 2009 to March 2012. Specifically, we compared the CIO
              ratings in June 2009 (or whenever an individual investment was first
              rated) up through and including each investment’s rating as of March
              2012 and summarized the data by agency. To describe whether CIO
              ratings indicated higher or lower investment risk over time, we calculated
              the numbers and percentages of investments (by agency and collectively
              for all the agencies) that maintained a constant rating over the entire
              performance period, and those that experienced a change to their CIO


              1
               We did not independently evaluate the ratings reported by agencies. However, we
              determined that they were sufficiently reliable for the purposes of our objectives by
              confirming with each agency that the ratings that we downloaded from the IT Dashboard
              were complete, accurate, and reflected the data they had reported to OMB.




              Page 30                                    GAO-13-98 Information Technology Dashboard
Appendix I: Objectives, Scope, and
Methodology




rating in at least one rating period. Then we analyzed the subset of
investments that experienced at least one changed rating and compared
the first CIO rating with the latest CIO rating (no later than March 2012) to
determine the numbers and percentages of investments (by agency and
collectively for all the agencies) that experienced a net rating increase, a
net rating decrease, or no net change. We also examined the comments
provided with the ratings to determine whether such comments were
useful in understanding the ratings. We presented our results to each
agency and OMB and solicited their input, explanations for the results,
and additional corroborating documentation, where appropriate.

To address our second objective, we reviewed available documentation,
obtained written responses to questions we posed to all agencies, and
interviewed OMB and agency officials to determine their policies and
practices related to assigning and updating the CIO ratings and related
data for the Dashboard. Specifically, we gathered descriptions about the
data, participants, and processes used to generate CIO ratings for
investments; when and under what circumstances each agency updates
its ratings; the specific factors agencies used in assigning their ratings;
and the reason(s) for their approaches to assigning and reporting the
ratings. We reviewed our results with agency officials to ensure that our
presentation of their approach was accurate. In addition, we utilized our
prior work and a report by the Department of Defense’s Office of the
Inspector General related to the department’s major IT investments. We
compared the findings in these reports to the CIO ratings the department
submitted to the Dashboard for investments that had been rated
consistently low or moderately low risk, and discussed our results with
department officials.

To address our third objective, we reviewed written and oral descriptions
of the benefits and challenges that agencies and OMB have experienced
in developing, submitting, updating, and utilizing CIO ratings. We sought
specific examples, corroborating documentation, and causal factors,
where available. After obtaining this information from individual agencies,
we compared their responses to identify benefits and challenges common
to multiple agencies and applied our judgment in determining whether any
additional benefits or challenges were present.

We conducted this performance audit from January 2012 to September
2012 in accordance with generally accepted government auditing
standards. Those standards require that we plan and perform the audit to
obtain sufficient, appropriate evidence to provide a reasonable basis for
our findings and conclusions based on our audit objectives. We believe


Page 31                               GAO-13-98 Information Technology Dashboard
Appendix I: Objectives, Scope, and
Methodology




that the evidence obtained provides a reasonable basis for our findings
and conclusions based on our audit objectives.




Page 32                              GAO-13-98 Information Technology Dashboard
Appendix II: Risk Levels for Investments at
                                        Appendix II: Risk Levels for Investments at
                                        Selected Agencies, as of March 2012



Selected Agencies, as of March 2012

                                        The table below lists the total number of major information technology (IT)
                                        investments rated on the federal IT Dashboard as of March 2012 for each
                                        agency selected for this review, with the numbers of investments rated at
                                        each of the risk levels specified by the Office of Management and Budget
                                        (OMB) for the chief information officer (CIO) rating. The last line in the
                                        table reports each agency’s total budget for fiscal year 2012 for their
                                        major IT investments, as also reported on the Dashboard in March 2012.

Table 8: CIO Ratings and Budget Totals for Selected Agencies’ Major IT Investments as of March 2012

                                                                        DOD              DHS                HHS       DOI      OPM       NSF
High risk and moderately high risk                                           0                3               3         6          0        0
Medium risk                                                                13               35               19         8          3        0
Low risk and moderately low risk                                           74               45               59         34         6        5
Total number of investments rated on the Dashboard                         87               83               81         48         9        5
Budget for major IT investments (FY 2012)                            $13.3B            $4.4B               $2.8B   $865.6M   $46.3M $85.5M
                                        Source: GAO analysis of data downloaded from OMB’s IT Dashboard.


                                        Note: Table does not include downgraded or eliminated investments.




                                        Page 33                                                      GAO-13-98 Information Technology Dashboard
Appendix III: CIO Ratings and Net Changes
              Appendix III: CIO Ratings and Net Changes for
              Major IT Investments at Selected Agencies,
              June 2009 through March 2012


for Major IT Investments at Selected
Agencies, June 2009 through March 2012
              This appendix provides additional information about chief information
              officer (CIO) ratings for major IT information technology (IT) investments
              at each of the agencies selected for this review. The first figure for each
              agency depicts the number of investments at each rating level for the end
              of each month, as reported on the federal IT Dashboard. 1 The second
              figure depicts the number of investments whose risk level demonstrated a
              net increase, net decrease, no net change, or remained constant during
              the investment’s entire time on the Dashboard. 2




              1
               The Office of Management and Budget (OMB) directs agency CIOs to evaluate
              investments and assign ratings according to a five-point scale. The risk levels are: 5-low
              risk (green), 4-moderately low risk (green), 3-medium risk (yellow), 2-moderately high risk
              (red), and 1-high risk (red).
              2
               An investment’s risk level increased when it received a lower CIO rating in March 2012
              compared with its initial Dashboard rating. An investment’s risk level was reduced when it
              received a higher CIO rating in March 2012 compared with its initial Dashboard rating. An
              investment’s CIO rating exhibited no net change when it received the same rating in
              March 2012 as its initial rating, despite any interim changes that may have occurred. An
              investment’s CIO rating was constant when the rating remained unchanged from the initial
              CIO rating through March 2012.




              Page 34                                         GAO-13-98 Information Technology Dashboard
                                        Appendix III: CIO Ratings and Net Changes for
                                        Major IT Investments at Selected Agencies,
                                        June 2009 through March 2012




Department of Defense

Figure 6: CIO Ratings for Major IT Investments at the Department of Defense, as Reported on the IT Dashboard from July 2009
through March 2012




                                        Note: Figure does not include downgraded or eliminated investments.




                                        Page 35                                         GAO-13-98 Information Technology Dashboard
Appendix III: CIO Ratings and Net Changes for
Major IT Investments at Selected Agencies,
June 2009 through March 2012




Figure 7: Changes to Risk Levels for Major IT Investments at the Department of
Defense, Initial CIO Rating through March 2012




Note: Figure does not include downgraded or eliminated investments. An investment’s CIO rating is
constant when the rating remained unchanged from the initial rating through March 2012. All other
categories of risk in the chart compare the initial CIO rating to the CIO rating as of March 2012,
ignoring any interim changes that may have occurred in between those dates.




Page 36                                          GAO-13-98 Information Technology Dashboard
                                        Appendix III: CIO Ratings and Net Changes for
                                        Major IT Investments at Selected Agencies,
                                        June 2009 through March 2012




Department of Homeland
Security

Figure 8: CIO Ratings for Major IT Investments at the Department of Homeland Security, as Reported on the IT Dashboard
from July 2009 through March 2012




                                        Note: Figure does not include downgraded or eliminated investments.




                                        Page 37                                         GAO-13-98 Information Technology Dashboard
Appendix III: CIO Ratings and Net Changes for
Major IT Investments at Selected Agencies,
June 2009 through March 2012




Figure 9: Changes to Risk Levels for Major IT Investments at the Department of
Homeland Security, Initial CIO Rating through March 2012




Note: Figure does not include downgraded or eliminated investments. An investment’s CIO rating is
constant when the rating remained unchanged from the initial rating through March 2012. All other
categories of risk in the chart compare the initial CIO rating to the CIO rating as of March 2012,
ignoring any interim changes that may have occurred in between those dates.




Page 38                                          GAO-13-98 Information Technology Dashboard
                                        Appendix III: CIO Ratings and Net Changes for
                                        Major IT Investments at Selected Agencies,
                                        June 2009 through March 2012




Department of Health and
Human Services

Figure 10: CIO Ratings for Major IT Investments at the Department of Health and Human Services, as Reported on the IT
Dashboard from July 2009 through March 2012




                                        Note: Figure does not include downgraded or eliminated investments.




                                        Page 39                                         GAO-13-98 Information Technology Dashboard
Appendix III: CIO Ratings and Net Changes for
Major IT Investments at Selected Agencies,
June 2009 through March 2012




Figure 11: Changes to Risk Levels for Major IT Investments at the Department of
Health and Human Services, Initial CIO Rating through March 2012




Note: Figure does not include downgraded or eliminated investments. An investment’s CIO rating is
constant when the rating remained unchanged from the initial rating through March 2012. All other
categories of risk in the chart compare the initial CIO rating to the CIO rating as of March 2012,
ignoring any interim changes that may have occurred in between those dates.




Page 40                                          GAO-13-98 Information Technology Dashboard
                                         Appendix III: CIO Ratings and Net Changes for
                                         Major IT Investments at Selected Agencies,
                                         June 2009 through March 2012




Department of the Interior

Figure 12: CIO Ratings for Major IT Investments at the Department of the Interior, as Reported on the IT Dashboard from July
2009 through March 2012




                                         Note: Figure does not include downgraded or eliminated investments.




                                         Page 41                                         GAO-13-98 Information Technology Dashboard
Appendix III: CIO Ratings and Net Changes for
Major IT Investments at Selected Agencies,
June 2009 through March 2012




Figure 13: Changes to Risk Levels for Major IT Investments at the Department of the
Interior, Initial CIO Rating through March 2012




Note: Figure does not include downgraded or eliminated investments. An investment’s CIO rating is
constant when the rating remained unchanged from the initial rating through March 2012. All other
categories of risk in the chart compare the initial CIO rating to the CIO rating as of March 2012,
ignoring any interim changes that may have occurred in between those dates.




Page 42                                          GAO-13-98 Information Technology Dashboard
                                        Appendix III: CIO Ratings and Net Changes for
                                        Major IT Investments at Selected Agencies,
                                        June 2009 through March 2012




National Science
Foundation

Figure 14: CIO Ratings for Major IT Investments at the National Science Foundation, as Reported on the IT Dashboard from
June 2009 through March 2012




                                        Note: Figure does not include downgraded or eliminated investments.




                                        Page 43                                         GAO-13-98 Information Technology Dashboard
Appendix III: CIO Ratings and Net Changes for
Major IT Investments at Selected Agencies,
June 2009 through March 2012




Figure 15: Changes to Risk Levels for Major IT Investments at the National Science
Foundation, Initial CIO Rating through March 2012




Note: Figure does not include downgraded or eliminated investments. An investment’s CIO rating is
constant when the rating remained unchanged from the initial rating through March 2012. All other
categories of risk in the chart compare the initial CIO rating to the CIO rating as of March 2012,
ignoring any interim changes that may have occurred in between those dates.




Page 44                                          GAO-13-98 Information Technology Dashboard
                                        Appendix III: CIO Ratings and Net Changes for
                                        Major IT Investments at Selected Agencies,
                                        June 2009 through March 2012




Office of Personnel
Management

Figure 16: CIO Ratings for Major IT Investments at the Office of Personnel Management, as Reported on the IT Dashboard
from July 2009 through March 2012




                                        Note: Figure does not include downgraded or eliminated investments.




                                        Page 45                                         GAO-13-98 Information Technology Dashboard
Appendix III: CIO Ratings and Net Changes for
Major IT Investments at Selected Agencies,
June 2009 through March 2012




Figure 17: Changes to Risk Levels for Major IT Investments at the Office of
Personnel Management, Initial CIO Rating through March 2012




Note: Figure does not include downgraded or eliminated investments. An investment’s CIO rating is
constant when the rating remained unchanged from the initial rating through March 2012. All other
categories of risk in the chart compare the initial CIO rating to the CIO rating as of March 2012,
ignoring any interim changes that may have occurred in between those dates.




Page 46                                          GAO-13-98 Information Technology Dashboard
Appendix IV: Comments from the
             Appendix IV: Comments from the Department
             of Defense



Department of Defense




             Page 47                                     GAO-13-98 Information Technology Dashboard
Appendix IV: Comments from the Department
of Defense




Page 48                                     GAO-13-98 Information Technology Dashboard
Appendix V: GAO Contacts and Staff
                  Appendix V: GAO Contacts and Staff
                  Acknowledgments



Acknowledgments

                  David A. Powner, (202) 512-9286 or pownerd@gao.gov
GAO Contacts
                  In addition to the contact name above, the following staff also made key
Staff             contributions to this report: Paula Moore (Assistant Director), Neil
Acknowledgments   Doherty, Lynn Espedido, Rebecca Eyler, Kate Feild, Dan Gordon,
                  Andrew Stavisky, Sonya Vartivarian, Shawn Ward, Kevin Walsh, Jessica
                  Waselkow, and Monique Williams.




(311264)
                  Page 49                              GAO-13-98 Information Technology Dashboard
GAO’s Mission         The Government Accountability Office, the audit, evaluation, and
                      investigative arm of Congress, exists to support Congress in meeting its
                      constitutional responsibilities and to help improve the performance and
                      accountability of the federal government for the American people. GAO
                      examines the use of public funds; evaluates federal programs and
                      policies; and provides analyses, recommendations, and other assistance
                      to help Congress make informed oversight, policy, and funding decisions.
                      GAO’s commitment to good government is reflected in its core values of
                      accountability, integrity, and reliability.

                      The fastest and easiest way to obtain copies of GAO documents at no
Obtaining Copies of   cost is through GAO’s website (www.gao.gov). Each weekday afternoon,
GAO Reports and       GAO posts on its website newly released reports, testimony, and
                      correspondence. To have GAO e-mail you a list of newly posted products,
Testimony             go to www.gao.gov and select “E-mail Updates.”

Order by Phone        The price of each GAO publication reflects GAO’s actual cost of
                      production and distribution and depends on the number of pages in the
                      publication and whether the publication is printed in color or black and
                      white. Pricing and ordering information is posted on GAO’s website,
                      http://www.gao.gov/ordering.htm.
                      Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
                      TDD (202) 512-2537.
                      Orders may be paid for using American Express, Discover Card,
                      MasterCard, Visa, check, or money order. Call for additional information.
                      Connect with GAO on Facebook, Flickr, Twitter, and YouTube.
Connect with GAO      Subscribe to our RSS Feeds or E-mail Updates. Listen to our Podcasts.
                      Visit GAO on the web at www.gao.gov.
                      Contact:
To Report Fraud,
Waste, and Abuse in   Website: www.gao.gov/fraudnet/fraudnet.htm
                      E-mail: fraudnet@gao.gov
Federal Programs      Automated answering system: (800) 424-5454 or (202) 512-7470

                      Katherine Siggerud, Managing Director, siggerudk@gao.gov, (202) 512-
Congressional         4400, U.S. Government Accountability Office, 441 G Street NW, Room
Relations             7125, Washington, DC 20548

                      Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800
Public Affairs        U.S. Government Accountability Office, 441 G Street NW, Room 7149
                      Washington, DC 20548




                        Please Print on Recycled Paper.