oversight

Information Technology: Agencies Need to Strengthen Oversight of Billions of Dollars in Operations and Maintenance Investments

Published by the Government Accountability Office on 2012-10-16.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

               United States Government Accountability Office

GAO            Report to the Subcommittee on Federal
               Financial Management, Government
               Information, Federal Services, and
               International Security, Committee on
               Homeland Security and Governmental
               Affairs, U.S. Senate
October 2012
               INFORMATION
               TECHNOLOGY

               Agencies Need to
               Strengthen Oversight
               of Billions of Dollars
               in Operations and
               Maintenance
               Investments




GAO-13-87
                                                  October 2012

                                                  INFORMATION TECHNOLOGY
                                                  Agencies Need to Strengthen Oversight of Billions of
                                                  Dollars in Operations and Maintenance Investments
Highlights of GAO-13-87, a report to the
Subcommittee on Federal Financial Management,
Government Information, Federal Services, and
International Security, Committee on Homeland
Security and Governmental Affairs, U.S. Senate

Why GAO Did This Study                           What GAO Found
Of the $79 billion federal agencies              Federal agency assessments of the performance of information technology (IT)
budgeted for IT in 2011, $54 billion             investments in operations and maintenance (O&M)—commonly referred to as
(about 69 percent) was reported to               operational analyses (OAs)—vary significantly. Office of Management and Budget
have been spent on the operations and            (OMB) guidance calls for agencies to develop an OA policy and perform such
maintenance of existing legacy IT                analyses annually to ensure steady state investments continue to meet agency
systems—commonly referred to as                  needs. The guidance also includes 17 key factors (addressing areas such as cost,
steady state investments. Given the              schedule, customer satisfaction, and innovation) that are to be assessed. The five
size and magnitude of these                      agencies GAO reviewed varied in the extent to which they carried out these tasks.
investments, it is essential that
agencies effectively manage them to              The Departments of Homeland Security (DHS) and Health and Human Services
ensure they continue to meet agency              (HHS) developed a policy which included all OMB assessment factors and
needs. As such, OMB directs agencies             performed OAs. However, they did not include all investments and key factors. In
to periodically examine the                      particular, DHS analyzed 16 of its 44 steady state investments, meaning 28
performance of such investments                  investments with annual budgets totaling $1 billion were not analyzed; HHS
against, among other things,                     analyzed 7 of its 8 steady state investments. For OAs performed by DHS and
established cost, schedule, and                  HHS, both fully addressed approximately half of the key factors. With regard to the
performance goals by performing                  DHS and HHS investments that did not undergo an analysis or were not fully
annual OAs.                                      assessed against key factors, agency officials said this was due in part to program
GAO was asked to determine the                   officials inconsistently applying OMB and agency guidance in conducting OAs and
extent to which federal agencies                 that OAs were not a priority. DHS and HHS have recently begun to take action to
analyze the performance of steady                make OAs a priority and improve consistency. For example, DHS’s chief
state investments in accordance with             information officer recently issued a directive requiring all steady state IT
OMB guidance. To do so, GAO (1)                  investments to conduct analyses annually and plans to assign staff in the office of
selected five agencies, DOD, HHS,                the chief information officer to review them to ensure they are complete.
DHS, Treasury, and VA, which
                                       The Departments of Defense (DOD), the Treasury (Treasury), and Veterans
reported spending $4.6 billion annually
on major steady state investments; and Affairs (VA) did not develop a policy and did not perform analyses on their 23
(2) and compared their fiscal year 2011major steady state investments with annual budgets totaling $2.1 billion. DOD and
OAs to OMB criteria. GAO also          VA officials said that they did not have a policy or perform analyses because they
analyzed documents and interviewed     measure the performance of steady state investments via development of plans
agency officials regarding any         and business cases submitted to OMB (called exhibit 300s) as part of the budget
variances as well as their causes.     process. While these can be helpful in managing performance and do address
                                       aspects of the 17 key factors, they do not address 11 of the key factors. For
What GAO Recommends                    example, the exhibit 300 does not address reviewing strategic business results
GAO is recommending that DOD,          and making recommendations to modify or terminate an investment. Treasury
Treasury, and VA develop an OA policy  officials stated that they did not to perform OAs in 2011 and instead decided to
and conduct annual OAs; and that DHS   use  the  time to develop a policy. However, the officials stated that they did not
and HHS ensure OAs are being           anticipate the policy to be completed until the end of this calendar year.
performed for all investments and that Overall, these five agencies have steady state investments with a fiscal year 2011
all factors are fully assessed. GAO is
                                       budget of over $3 billion which have not undergone needed analyses. While OMB
also recommending that OMB revise its
                                       requires agencies to perform OAs, its existing guidance does not provide
guidance to incorporate mechanisms to
ensure OAs are completed and provide
                                       mechanisms that ensure the OAs are completed and allow public transparency
for increased transparency. In         into the results of the assessments. Until agencies address these shortcomings,
commenting on a draft of this report,  there is increased risk that these agencies will not know whether the multibillion
OMB and the five agencies GAO          dollar investments fully meet their intended objectives.
reviewed agreed with its content and
recommendations.
View GAO-13-87. For more information,
contact David A. Powner at (202) 512-9286 or
pownerd@gao.gov.                                                                          United States Government Accountability Office
Contents


Letter                                                                                      1
               Background                                                                   2
               Federal Agencies’ Assessments of Major IT Steady State
                 Investments Vary Significantly                                            7
               Conclusions                                                                17
               Recommendations for Executive Action                                       18
               Agency Comments and Our Evaluation                                         18

Appendix I     Objective, Scope, and Methodology                                          21



Appendix II    Extent to Which DHS’s and HHS’s OAs Addressed Key Factors                  23



Appendix III   Comments from the Department of Defense                                    47



Appendix IV    Comments from the Department of Homeland Security                          49



Appendix V     Comments from the Department of the Treasury                               51



Appendix VI    Comments from the Department of Veterans Affairs                           52



Appendix VII   GAO Contact and Staff Acknowledgments                                      54



Tables
               Table 1: Total Reported Federal IT Spending by the 26 Key
                        Agencies for Fiscal Year 2011 (in millions), in Descending
                        Order of O&M Spending                                               3
               Table 2: Total Steady State IT Investments, and Number of
                        Investments for Five Agencies That Had OAs and Did Not
                        Have OAs with Cost (in millions)                                    8



               Page i                                         GAO-13-87 Information Technology
Table 3: Extent to Which DHS Steady State IT Investments
         Underwent OAs in Fiscal Year 2011 (by Component and
         Cost in Millions)                                                 10
Table 4: Extent to Which HHS Steady State IT Investments
         Underwent OAs in Fiscal Year 2011 (by Component and
         Cost in Millions)                                                 12
Table 5: U.S. Customs and Border Protection: Analysis of Extent to
         Which Infrastructure’s Fiscal Year 2011 OA Addressed
         OMB Key Factors                                                   23
Table 6: Office of the Chief Information Officer: Analysis of Extent
         to Which Homeland Secure Data Network’s Fiscal Year
         2011 OA Addressed OMB Key Factors                                 24
Table 7: Federal Emergency Management Agency: Analysis of
         Extent to Which Disaster Management E-Government
         Initiative’s Fiscal Year 2011 OA Addressed OMB Key
         Factors                                                           25
Table 8: Federal Emergency Management Agency: Analysis of
         Extent to Which Integrated Financial Management
         Information System’s Fiscal Year 2011 OA Addressed OMB
         Key Factors                                                       26
Table 9: U.S. Immigration and Customs Enforcement: Analysis of
         Extent to Which Intelligence Fusion System’s Fiscal Year
         2011 OA Addressed OMB Key Factors                                 27
Table 10: National Protection and Programs Directorate: Analysis
         of Extent to Which National Security and Emergency
         Preparedness Priority Telecommunication Services’s
         Fiscal Year 2011 OA Addressed OMB Key Factors                     28
Table 11: Transportation Security Administration: Analysis of
         Extent to Which Federal Air Marshal Service Mission
         Scheduling and Notification System’s Fiscal Year 2011 OA
         Addressed OMB Key Factors                                         29
Table 12: Transportation Security Administration: Analysis of
         Extent to Which Federal Air Marshal Service Network’s
         Fiscal Year 2011 OA Addressed OMB Key Factors                     30
Table 13: Transportation Security Administration: Analysis of
         Extent to Which Hazardous Materials Endorsement Threat
         Assessment Program’s Fiscal Year 2011 OA Addressed
         OMB Key Factors                                                   31
Table 14: Transportation Security Administration: Analysis of
         Extent to Which Information Technology Infrastructure
         Program’s Fiscal Year 2011 OA Addressed OMB Key
         Factors                                                           32



Page ii                                        GAO-13-87 Information Technology
Table 15: Transportation Security Administration: Analysis of
        Extent to Which Secure Flight’s Fiscal Year 2011 OA
        Addressed OMB Key Factors                                          33
Table 16: U.S. Coast Guard: Analysis of Extent to Which Coast
        Guard Business Intelligence’s Fiscal Year 2011 OA
        Addressed OMB Key Factors                                          34
Table 17: U.S. Coast Guard: Analysis of Extent to Which Core
        Accounting System Suite’s Fiscal Year 2011 OA Addressed
        OMB Key Factors                                                    35
Table 18: U.S. Coast Guard: Analysis of Extent to Which Direct
        Access’s Fiscal Year 2011 OA Addressed OMB Key Factors             36
Table 19: U.S. Coast Guard: Analysis of Extent to Which Marine
        Information for Safety and Law Enforcement’s Fiscal Year
        2011 OA Addressed OMB Key Factors                                  37
Table 20: U.S. Citizenship and Immigration Services: Analysis of
        Extent to Which Immigration Computer Linked
        Application Information Management System’s Fiscal Year
        2011 OA Addressed OMB Key Factors                                  38
Table 21: Centers for Disease Control and Prevention: Analysis of
        Extent to Which National Select Agency Registry’s Fiscal
        Year 2011 OA Addressed OMB Key Factors                             40
Table 22: Centers for Medicare and Medicaid Services: Analysis of
        Extent to Which Beneficiary e-Services’s Fiscal Year 2011
        OA Addressed OMB Key Factors                                       41
Table 23: Centers for Medicare and Medicaid Services: Analysis of
        Extent to Which Health Care Quality Improvement and
        Evaluation System’s Fiscal Year 2011 OA Addressed OMB
        Key Factors                                                        42
Table 24: Health Resources and Services Administration: Analysis
        of Extent to Which Electronic Handbooks Program
        Management Office’s Fiscal Year 2011 OA Addressed OMB
        Key Factors                                                        43
Table 25: Health Resources and Services Administration: Analysis
        of Extent to Which National Practitioner Data Bank’s
        Fiscal Year 2011 OA Addressed OMB Key Factors                      44
Table 26: Indian Health Service: Analysis of Extent to Which
        Infrastructure, Office Automation, and
        Telecommunications’s Fiscal Year 2011 OA Addressed
        OMB Key Factors                                                    45
Table 27: National Institutes of Health: Analysis of Extent to Which
        Business Intelligence System’s Fiscal Year 2011 OA
        Addressed OMB Key Factors                                          46



Page iii                                       GAO-13-87 Information Technology
Figure
         Figure 1: Percentages of Total IT Spending for Fiscal Years 2011
                  and 2012 for the 26 Key Federal Agencies                                         3




         Abbreviations

         CIO                        chief information officer
         DOD                        Department of Defense
         DHS                        Department of Homeland Security
         HHS                        Department of Health and Human Services
         IT                         information technology
         NASA                       National Aeronautics and Space Administration
         OA                         operational analysis
         O&M                        operations and maintenance
         OMB                        Office of Management and Budget
         Treasury                   Department of the Treasury
         VA                         Department of Veterans Affairs


         This is a work of the U.S. government and is not subject to copyright protection in the
         United States. The published product may be reproduced and distributed in its entirety
         without further permission from GAO. However, because this work may contain
         copyrighted images or other material, permission from the copyright holder may be
         necessary if you wish to reproduce this material separately.




         Page iv                                                 GAO-13-87 Information Technology
United States Government Accountability Office
Washington, DC 20548




                                   October 16, 2012

                                   The Honorable Thomas R. Carper
                                   Chairman
                                   The Honorable Scott P. Brown
                                   Ranking Member
                                   Subcommittee on Federal Financial Management, Government
                                   Information, Federal Services, and International Security
                                   Committee on Homeland Security and Governmental Affairs
                                   United States Senate

                                   In fiscal year 2011, 26 key federal agencies reported spending
                                   approximately $79 billion on information technology (IT) systems to the
                                   Office of Management and Budget (OMB). 1 Of the $79 billion, $54 billion
                                   was reported by the agencies to be spent on operations and maintenance
                                   (O&M), which consists of existing legacy systems (i.e., steady state) and
                                   systems that are in both development and O&M (known as mixed life
                                   cycle). Given the size and magnitude of these investments, it is important
                                   that agencies effectively manage the operations and maintenance of
                                   existing investments to ensure they (1) continue to meet agency needs,
                                   (2) deliver value, and (3) do not unnecessarily duplicate or overlap with
                                   other investments.

                                   OMB directs agencies to periodically examine the performance of these
                                   investments against, among other things, established cost, schedule, and
                                   performance goals. Specifically, OMB calls for agencies to perform
                                   annual operational analyses (OA), which is a key method for examining
                                   the performance of such investments in O&M.

                                   As requested, our objective was to determine the extent to which federal
                                   agencies assess the performance of steady state IT investments in


                                   1
                                    The 26 federal departments and agencies that report to OMB on their IT investments are
                                   the Departments of Agriculture, Commerce, Defense, Education, Energy, Health and
                                   Human Services, Homeland Security, Housing and Urban Development, Interior, Justice,
                                   Labor, State, Transportation, the Treasury, and Veterans Affairs; Environmental Protection
                                   Agency, General Services Administration, National Aeronautics and Space Administration,
                                   National Archives and Records Administration, National Science Foundation, Nuclear
                                   Regulatory Commission, Office of Personnel Management, Small Business
                                   Administration, Smithsonian Institution, Social Security Administration, and U.S. Agency
                                   for International Development.




                                   Page 1                                                 GAO-13-87 Information Technology
             accordance with this OMB guidance. To do so, we selected five agencies,
             the Departments of Defense (DOD), Health and Human Services (HHS),
             Homeland Security (DHS), the Treasury (Treasury), and Veterans Affairs
             (VA), which have the largest budgets for major steady state IT
             investments, accounting for approximately $37 billion annually or about
             70 percent of all reported O&M spending in fiscal year 2011. In doing this
             we focused on these agencies’ 75 major IT investments valued at $4.6
             billion annually that were strictly in the O&M phase (i.e., excluded mixed
             cycle investments). We determined whether the agencies developed OA
             policies in accordance with OMB guidance. We also determined whether
             these agencies were conducting OAs to manage these investments. More
             specifically, we reviewed all of these agencies’ OAs performed during
             fiscal year 2011 and compared them to OMB and related criteria.

             We conducted this performance audit from October 2011 to September
             2012 in accordance with generally accepted government auditing
             standards. Those standards require that we plan and perform the audit to
             obtain sufficient, appropriate evidence to provide a reasonable basis for
             our findings and conclusions based on our audit objectives. We believe
             that the evidence obtained provides a reasonable basis for our findings
             and conclusions based on our audit objective. Details on our objective,
             scope, and methodology are contained in appendix I.


             In fiscal year 2011, the 26 key federal agencies that report to OMB on
Background   their IT investments reported spending approximately $79 billion on a
             wide variety of IT systems. Of this amount, agencies reported spending
             $54 billion on O&M for existing steady state investments; they plan on
             spending about $53 billion in fiscal year 2012. As shown in figure 1, these
             amounts represent a significant majority (i.e., 69 and 71 percent) of the
             overall reported IT spending in 2011 ($79 billion) and that planned for
             2012 ($75 billion), respectively.




             Page 2                                        GAO-13-87 Information Technology
                                          Figure 1: Percentages of Total IT Spending for Fiscal Years 2011 and 2012 for the 26
                                          Key Federal Agencies




                                          Although O&M spending governmentwide is about 70 percent of total IT
                                          spending, the amount spent by each agency varies from a high of 98
                                          percent to a low of 45 percent (as shown in the table below).


Table 1: Total Reported Federal IT Spending by the 26 Key Agencies for Fiscal Year 2011 (in millions), in Descending Order of
O&M Spending

Agency                                                   Total Development             O&M    O&M percentage of total spending
                        a
Department of Defense                                 $37,120        $13,896        $23,224                                 62.6
Department of Health and Human Services                 7,030          1,721          5,309                                 75.5
Department of Homeland Security                         5,987          1,631          4,356                                 72.8
Department of the Treasury                              3,419            553          2,866                                 83.8
Department of Veterans Affairs                          3,193            980          2,213                                 69.3
Department of Agriculture                               2,504            409          2,095                                 83.7
Department of Justice                                   2,991            978          2,013                                 67.3
Department of Energy                                    1,998            266          1,732                                 86.7
National Aeronautics and Space Administration           1,777            173          1,604                                 90.2
Department of Commerce                                  2,337            840          1,496                                 64.0
Department of Transportation                            2,926          1,613          1,313                                 44.9
Department of State                                     1,400             94          1,306                                 93.3
Department of the Interior                              1,028            132            896                                 87.2
Social Security Administration                          1,471            727            743                                 50.5
Department of Labor                                       607            113            495                                 81.5
General Services Administration                           599            106            493                                 82.4
Department of Education                                   580             97            483                                 83.3
Environmental Protection Agency                           467             76            391                                 83.7




                                          Page 3                                              GAO-13-87 Information Technology
Agency                                                           Total Development                O&M     O&M percentage of total spending
Department of Housing and Urban Development                         546                  210       336                                      61.6
Nuclear Regulatory Commission                                       153                   18       136                                      88.5
U.S. Agency for International Development                           145                   31       115                                      78.9
Small Business Administration                                       122                   19       103                                      84.2
National Science Foundation                                          95                   12        82                                      86.9
National Archives and Records Administration                        143                   70        73                                      51.1
Office of Personnel Management                                       79                   10        69                                      86.8
Smithsonian Institution                                              66                    4        62                                      94.1
Total                                                         $78,784                 $24,779   $54,005                                   68.5%
                                            Source: GAO analysis based on OMB data.
                                            a
                                             While the Army Corps of Engineers submits information on its IT investments to OMB separate from
                                            the Department of Defense’s submission, we have included it here as part of the Department of
                                            Defense.


                                            The National Aeronautics and Space Administration (NASA) reported
                                            spending approximately 90 percent of its total IT spending on O&M with
                                            the remaining 10 percent going to new investments. The reason for this
                                            mix of spending, according to NASA officials, is due to NASA’s mission
                                            (i.e., the space shuttle mission) which relies heavily on legacy systems.
                                            By contrast, the Department of Transportation reported spending
                                            approximately 45 percent of its total IT costs on O&M with the other 55
                                            percent going to new investments. According to department officials, this
                                            mix of spending is largely due to the fact that the department has a
                                            number of IT development investments underway that involve large
                                            financial commitments relative to O&M investments.

OMB’s Roles and                             To assist agencies in managing their investments, Congress enacted the
Responsibilities                            Clinger-Cohen Act of 1996, which requires OMB to establish processes to
                                            analyze, track, and evaluate the risks and results of major capital
for Overseeing IT                           investments in information systems made by federal agencies and report
Investments,                                to Congress on the net program performance benefits achieved as a
Including Operations and                    result of these investments. 2 Further, the act places responsibility for
                                            managing investments with the heads of agencies and establishes chief
Maintenance                                 information officers to advise and assist agency heads in carrying out this
                                            responsibility.




                                            2
                                                40 U.S.C. § 11302(c).




                                            Page 4                                                        GAO-13-87 Information Technology
    In carrying out its responsibilities, OMB uses several data collection
    mechanisms to oversee federal IT spending during the annual budget
    formulation process. Specifically, OMB requires 26 key federal
    departments and agencies to provide information to it related to their IT
    investments (called exhibit 53s) and capital asset plans and business
    cases (called exhibit 300s). 3

•   Exhibit 53. The purpose of the exhibit 53 is to identify all IT investments—
    both major and nonmajor 4—and their associated costs within a federal
    organization. Information included on agency exhibit 53s is designed, in
    part, to help OMB better understand what agencies are spending on IT
    investments. The information also supports cost analyses prescribed by
    the Clinger-Cohen Act. As part of the annual budget, OMB publishes a
    report on IT spending for the federal government representing a
    compilation of exhibit 53 data submitted by the 26 agencies.
•   Exhibit 300. The purpose of the exhibit 300 is to provide a business case
    for each major IT investment and to allow OMB to monitor IT investments
    once they are funded. Agencies are required to provide information on
    each major investment’s cost, schedule, and performance.

    In addition, in June 2009, to further improve the transparency into and
    oversight of agencies’ IT investments, OMB publicly deployed a website,
    known as the IT Dashboard (Dashboard), which replaced its Management
    Watch List and High-Risk List. As part of this effort, OMB issued guidance
    directing federal agencies to report, via the Dashboard, the performance
    of their IT investments. Currently, the Dashboard publicly displays
    information on the cost, schedule, and performance of over 700 major
    federal IT investments at 26 federal agencies. In addition, the Dashboard
    allows users to download exhibit 53 data, which include information on
    both major and nonmajor investments. According to OMB, these data are
    intended to provide a near real-time perspective of the performance of
    these investments, as well as a historical perspective. Further, the public


    3
      OMB Circular No. A-11, Preparation, Submission, and Execution of the Budget (August
    2012).
    4
     According to OMB guidance, a major investment is a system or acquisition requiring
    special management attention because of its importance to the mission or function of the
    agency, a component of the agency, or another organization; is for financial management
    and obligates more than $500,000 annually; has significant program or policy implications;
    has high executive visibility; has high development, operating, or maintenance costs; is
    funded through other than direct appropriations; or is defined as major by the agency’s
    capital planning and investment control process.




    Page 5                                                 GAO-13-87 Information Technology
display of these data is intended to allow OMB, other oversight bodies,
and the general public to hold the government agencies accountable for
results and progress. Since the Dashboard has been implemented, we
have reported and made recommendations to improve the data accuracy
and reliability. In 2010 and 2011, we reported on the progress of the
Dashboard and made recommendations to further improve how it rates
investments relative to current performance. 5

Further, OMB has developed guidance that calls for agencies to develop
an OA policy for examining the ongoing performance of existing IT
investments to measure, among other things, that the investment is
continuing to meet business and customer needs and is contributing to
meeting the agency’s strategic goals. 6 This guidance calls for the policy to
provide for an annual OA of each investment that addresses the
following: cost, schedule, customer satisfaction, strategic and business
results, financial goals, and innovation. To address these areas, the
guidance specifies the following 17 key factors that are to be addressed:

    •    assessment of current costs against life-cycle costs;
    •    a structured schedule assessment (i.e., measuring the
         performance of the investment against its established schedule);
    •    a structured assessment of performance goals (i.e., measuring the
         performance of the investment against established goals);
    •    identification of whether the investment supports customer
         processes as designed and is delivering goods and services it
         was designed to deliver;
    •    a measure of the effect the investment has on the performing
         organization itself;
    •    a measure of how well the investment contributes to achieving the
         organization’s business needs and strategic goals;



5
 GAO, IT Dashboard: Accuracy Has Improved and Additional Efforts Are Under Way to
Better Inform Decision Making, GAO-12-210 (Washington, D.C.: Nov. 7, 2011);
Information Technology: OMB Has Made Improvements to Its Dashboard, but Further
Work Is Needed by Agencies and OMB to Ensure Data Accuracy, GAO-11-262
(Washington, D.C.: Mar. 15, 2011); and Information Technology: OMB’s Dashboard Has
Increased Transparency and Oversight, but Improvements Needed, GAO-10-701
(Washington, D.C.: July 16, 2010).
6
 Capital Programming Guide, Supplement to OMB Circular A-11, Part 7 (July 2012); OMB
Memorandum M-10-27 (June 2010), requires agencies to establish a policy for performing
OAs on steady state investments as a part of managing and monitoring investment
baselines.




Page 6                                               GAO-13-87 Information Technology
                           •    a comparison of current performance with a pre-established cost
                                baseline and estimates;
                           •    areas for innovation in the areas of customer satisfaction, strategic
                                and business results, and financial performance;
                           •    indication if the agency revisited alternative methods for achieving
                                the same mission needs and strategic goals;
                           •    consideration of issues, such as greater utilization of technology
                                or consolidation of investments to better meet organizational
                                goals;
                           •    an ongoing review of the status of the risks identified in the
                                investment’s planning and acquisition phases;
                           •    identification of whether there is a need to redesign, modify, or
                                terminate the investment;
                           •    an analysis on the need for improved methodology (i.e., better
                                ways for the investment to meet cost and performance goals);
                           •    lessons learned;
                           •    cost or schedule variances;
                           •    recommendations to redesign or modify an asset in advance of
                                potential problems; and
                           •    overlap with other investments.

                       With regard to overseeing the agencies’ development of policies and
                       annual performance, OMB officials responsible for governmentwide OA
                       policy stated that they expect agencies to perform all the steps specified
                       in the guidance and to be prepared to show documentation as evidence
                       of compliance with the guidance should OMB decide to check.


                       Although OMB guidance calls for agencies to develop an OA policy and
Federal Agencies’      perform such analyses annually, the extent to which the five federal
Assessments of Major   agencies we reviewed carried out these tasks varied significantly.
                       Specifically, DHS and HHS developed a policy and conducted OAs, but in
IT Steady State        doing so, they excluded key investments and assessment factors. DOD,
Investments Vary       Treasury, and VA did not develop a policy or conduct OAs. The following
Significantly          table shows the total number of steady state investments for each
                       agency, and provides the number and budgeted amount for those
                       investments that underwent an assessment and those that did not.




                       Page 7                                          GAO-13-87 Information Technology
                         Table 2: Total Steady State IT Investments, and Number of Investments for Five
                         Agencies That Had OAs and Did Not Have OAs with Cost (in millions)

                                                                       Total  Fiscal            Total  Fiscal
                          Agency (total investments            investments year 2011    investments year 2011
                          in steady state)                       with an OA     cost   without an OA     cost
                          DOD (4)                                       0         --               4       $381
                          DHS (44)                                     16      1,175              28       1,011
                          HHS (8)                                       7       207                1          77
                          Treasury (16)                                 0         --              16        152
                          VA (3)                                        0         --               3       1,600
                          Total (75)                                   23     $1,400              52     $3,200
                         Source: GAO analysis based on OMB data.


                         Until agencies more completely address their policy and performance
                         shortcomings, there is increased risk that existing multibillion dollar
                         investments will continue to be funded although it is not fully known
                         whether they meet their intended objectives.


DHS and HHS Developed    DHS and HHS had developed policies, which contained all performance
an OA Policy and         factors identified in OMB’s guidance. Specifically,
Performed OAs, but Did
                         •    In 2008, DHS issued its policy called “Operational Analysis
Not Address All               Guidance.” 7 The guidance states that OAs should be performed on an
Investments and Key           annual basis to evaluate the operational results of agency steady
Factors                       state investments. In addition, the guidance provides a report
                              template which includes sections that should be contained and
                              reported on in it. DHS’s policy addressed all of the key factors in the
                              OMB guidance, including, for example, assessing current costs
                              against life-cycle costs and a detailed schedule assessment.
                         •    In 2008, HHS issued its policy called “Practices Guide: Annual
                              Operational Analysis.” 8 The guide states OAs are required to be
                              performed on an annual basis. Further, the guide includes a template
                              and a checklist for conducting them. In addition, agencies within the
                              department have issued their own policy. For example, in 2011, The


                         7
                          DHS, Operational Analysis Guidance, Version 1.1, May 2008.
                         8
                          HHS, Enterprise Performance Life Cycle Framework, Practices Guide, Annual
                         Operational Analysis, September 2008.




                         Page 8                                                  GAO-13-87 Information Technology
    Centers for Disease Control and Prevention issued its “Operational
    Analysis Guide” and in 2010, the National Institutes of Health issued
    its framework, “A How-to Guide.” 9 These policies contained all of the
    key factors identified in the OMB policy, such as measuring the effect
    the investment has on the performing organization itself and
    identifying any areas for innovation.

Further, DHS and HHS performed OAs on some of their steady state
investments, but not for all. Specifically, of their 52 total steady state
investments, DHS and HHS conducted analyses on 23 with total budgets
of $1.4 billion and did not conduct analyses on 29 investments with total
budgets of $1.1 billion. More specifically,

•   Of DHS’s 44 steady state investments, the department conducted
    OAs on16 of them, which have an annual budget of $1.2 billion; it did
    not perform analyses on the other 28, which have an annual budget of
    almost $1 billion.
•   Of HHS’s 8 steady state investments, the department conducted
    analyses on 7 of them, which have an annual budget of $207 million;
    it did not perform an OA on the remaining investment, which has an
    annual budget of $77 million.
Tables 3 and 4 show DHS’s and HHS’s steady state investments by
component agency and whether OAs were performed on these
investments in fiscal year 2011. (Details of our analysis of all the analyses
and a brief description of the investments are included in app. II.)




9
 HHS, Centers for Disease Control and Prevention: Operational Analysis Guide, Version
9.0, January 2011 and HHS, National Institutes of Health: NIG Operational Analysis
Management Framework: A How-to Guide, Version 1.2, August 2010.




Page 9                                               GAO-13-87 Information Technology
Table 3: Extent to Which DHS Steady State IT Investments Underwent OAs in Fiscal Year 2011 (by Component and Cost in
Millions)

                                                                                                                            Cost for
                                                                                                                               fiscal
                                                                                                         Yes       No      year 2011
Component              Investment
U.S. Customs and       1. Advance Passenger Information System                                                     X           $4.78
Border Protection      2. Automated Targeting System Maintenance                                                   X           33.89
                       3. SBInet Block 1                                                                           X           27.65
                       4. Infrastructure                                                                 X                    556.69
                       5. Land Border Integration                                                                  X           75.78
                       6. Non-Intrusive Inspection Systems Program                                                 X          110.49
                       7. Systems, Applications, and Products                                                      X           18.21
DHS, Office of the     8. Homeland Secure Data Network                                                   X                     47.66
Chief Information
Officer
Federal Emergency      9. Disaster Management E-Government Initiative                                    X                     12.27
Management Agency      10. Infrastructure                                                                          X           56.14
                       11. Integrated Financial Management Information System                            X                      4.30
                       12. National Flood Insurance Program Technology Systems and Services                        X           10.83
U.S. Immigration and 13. Federal Financial Management System                                                       X           24.69
Customs Enforcement 14. Intelligence Fusion System                                                       X                      7.36
National Protection    15. Information Systems Security Line of Business                                           X            3.02
and Programs           16. National Security and Emergency Preparedness Priority Telecommunication X                           46.05
Directorate            Services
                       17. Visitor and Immigrant Status Indicator Technology Arrival and Departure                 X           20.65
                       Information System
                       18. Visitor and Immigrant Status Indicator Technology Automated Biometric                   X          104.36
                       Identification System
Office of Health Affairs 19. BioSurveillance Common Operating Network                                              X            1.49
Transportation         20. Crew Vetting                                                                            X           11.60
Security
                       21. Federal Air Marshal Service Mission Scheduling and Notification System        X                     14.13
Administration
                       22. Federal Air Marshal Service Network                                           X                     41.79
                       23. Hazardous Materials Endorsement Threat Assessment Program                     X                     12.00
                       24. Information Technology Infrastructure Program                                 X                    284.20
                       25. Performance Management Information System                                               X           10.64
                       26. Secure Flight                                                                 X                     84.19
                       27. Transportation Worker Identification Credential                                         X            9.20
                       28. Transportation Security Administration Operating Platform                               X          121.80




                                            Page 10                                                  GAO-13-87 Information Technology
                                                                                                                            Cost for
                                                                                                                               fiscal
                                                                                                             Yes    No     year 2011
Component              Investment
U.S. Coast Guard       29. Asset Logistics Management Information System                                            X           8.15
                       30. Coast Guard Business Intelligence                                                 X                  6.29
                       31. Core Accounting System Suite                                                      X                 32.13
                       32. Direct Access                                                                     X                  9.64
                       33. Infrastructure Standard Workstation Infrastructure Recapitalization and                  X          62.93
                       Sustainment
                       34. Infrastructure Coast Guard One                                                           X          25.00
                       35. Marine Information for Safety and Law Enforcement                                 X                 10.39
                       36. Ports and Waterways Safety System IT                                                     X          38.40
                       37. Vessel Logistics System                                                                  X           4.85
U.S. Citizenship and   38. Customer Service Web Portal                                                              X          20.89
Immigration Services   39. Immigration Computer Linked Application Information Management System 3           X                  5.99
                       40. Infrastructure (End User Support)                                                        X         111.42
                       41. Integrated Document Production                                                           X          48.14
                       42. Naturalization Computer Linked Application Information Management System 4               X           2.42
U.S. Secret Service    43. Enterprise Financial Management System                                                   X           5.67
                       44. Information Technology Infrastructure                                                    X          38.03
Total                  44                                                                                    16     28     $2,186.20
                                           Source: GAO analysis based on OMB data.




                                           Page 11                                                   GAO-13-87 Information Technology
Table 4: Extent to Which HHS Steady State IT Investments Underwent OAs in Fiscal Year 2011 (by Component and Cost in
Millions)

                                                                                                       Yes    No     Cost for
                                                                                                                       fiscal
                                                                                                                         year
                                                                                                                        2011
Component                Investment
Centers for Disease      1. Information Technology Infrastructure                                             X        $77.09
Control and              2. National Select Agency Registry                                            X                 5.05
Prevention
Centers for Medicare 3. Beneficiary e-Services                                                         X                60.82
and Medicaid Services 4. Health Care Quality Improvement and Evaluation System                         X                21.20
Health Resources and 5. Electronic Handbooks Program Management Office                                 X                23.69
Services
                     6. National Practitioner Data Bank                                                X                23.51
Administration
Indian Health Service    7. Infrastructure, Office Automation, and Telecommunications                  X                55.31
National Institutes of   8. Business Intelligence System                                               X                17.60
Health
Total                    8                                                                             7      1       $284.27
                                             Source: GAO analysis based on OMB data.




                                             In addition, although DHS and HHS performed analyses, the agencies did
                                             not address all key factors in conducting them. Specifically,

                                         •   of DHS’s 16 OAs, which were to include a total 272 key factors, DHS:
                                                  •    addressed 145 (or 53 percent),
                                                  •    partially addressed 20 (or 7 percent), and
                                                  •    did not address 107 (or 39 percent); and
                                         •   of HHS’s 7 OAs, which were to include a total of 119 key factors, HHS:
                                                  •    addressed 66 (or 55 percent),
                                                  •    partially addressed 6 (or 5 percent), and
                                                  •    did not address 47 (or 39 percent) factors.
                                             The following provides key examples by component agency to illustrate
                                             how factors were fully addressed, partially addressed, or not addressed at
                                             all.

                                         •   In its operational analysis of its U.S. Coast Guard Business Intelligence
                                             investment, the U.S. Coast Guard fully addressed five key factors (see
                                             table 16 in app. II). For example, on the factor regarding whether the
                                             investment supports customer processes as designed and is delivering



                                             Page 12                                          GAO-13-87 Information Technology
    the goods and services it was designed to deliver, the component
    measured (via surveys) customer satisfaction, usage trends, system
    trends, and feedback, and used this information to implement system
    improvements. U.S. Coast Guard partially addressed three factors. For
    example, in assessing performance goals, the component identified two
    major goals of the investment, but did not include how or when these
    goals were to be achieved. U.S. Coast Guard did not address nine key
    factors, including those on identifying lessons learned and reviewing the
    status of risk versus cost, schedule, and performance. These factors are
    important because they provide management with key information on
    why problems occurred and how they can be avoided in the future, as
    well as whether the investment is worth pursing given anticipated costs,
    benefits, and associated risks.
•   In assessing the Information Technology Infrastructure Program,
    Transportation Security Administration addressed eight key factors (see
    table 14 in app. II). For example, on the factor calling for performance of
    a structured schedule assessment, the component analyzed a detailed
    list of task descriptions, start and end dates, and planned versus actual
    costs to ensure the investment is performing against an established
    schedule which can minimize costs over the life cycle of an investment.
    The component partially addressed one key factor; specifically, the factor
    calling for identifying whether the investment supports customer
    processes and is delivering the goods and services intended. In
    assessing this factor, Transportation Security Administration conducted
    surveys to measure customer satisfaction, but in doing so did not include
    measures to assess whether the investment was delivering the goods
    and services it was designed to deliver. The component did not address
    eight key factors. For example, it did not identify any areas for innovation
    or whether the investment overlapped with other systems. These latter
    steps are essential to identifying investment improvements, increasing
    value and reducing costs, and eliminating duplicate systems and the
    costs associated with them.
•   For the U.S. Immigration and Customs Enforcement’s Intelligence Fusion
    System, the component fully addressed nine key factors (see table 9 in
    app. II). These factors included analyzing current costs against life-cycle
    costs and whether the investment supports customer processes as
    designed and is delivering the goods and services it was designed to
    deliver, through measures such as customer surveys and help desk
    metrics. The component partially addressed the factor on identifying
    areas (e.g., business results and customer satisfaction, financial
    performance) for innovation. Specifically, it identified two areas for
    innovation, namely strategic and business results and customer
    satisfaction, but did not address financial performance. U.S. Immigration
    and Customs Enforcement did not address seven key factors; for


    Page 13                                        GAO-13-87 Information Technology
    example, it did not identify lessons learned or assess whether to modify
    or terminate the investment. Fully addressing these factors is crucial to
    agencies in determining whether to continue an investment that is not
    performing as required.
•   For its Infrastructure, Office Automation, and Telecommunications
    investment, Indian Health Service fully addressed 14 key factors (see
    table 26 in app. II). For example, in addressing the factor on assessing
    performance goals, it analyzed the investment’s performance goals
    against the results to date for each goal. The component partially
    addressed the factor on the status of risks versus cost, schedule, and
    performance. Specifically, it analyzed cost and schedule progress, but
    did not include an assessment of risks. Indian Health Service did not
    address two key factors. For example, it did not identify lessons learned
    and whether the investment overlapped with other systems. Addressing
    these factors is important because they help agencies to, among other
    things, identify where cost-effective improvements can be made.
•   HHS’s Health Resources and Services Administration fully addressed 15
    key factors in its operational analysis of its Electronic Handbooks
    Program Management Office (see table 24 in app. II). For example, it
    conducted a structured assessment of performance goals, including a
    detailed list of goals, and how and when they were addressed. Health
    Resources and Services Administration only partially addressed the key
    factor on providing a structured schedule assessment. Specifically, the
    component identified certain parts of the investment schedule, such as
    standard and unscheduled maintenance efforts, but other schedule
    elements, such as completion dates and goals, were not identified.
    Health Resources and Services Administration did not address one
    factor. For example, it did not assess current costs against life-cycle
    costs. This factor is important because it can, among other things,
    provide information to agency decision makers with answers to whether
    annual operating and maintenance costs are comparable to the
    estimated costs developed during the development phase.
•   In its analysis of the Business Intelligence System, National Institutes of
    Health fully assessed six key factors (see table 27 in app. II). For
    example, on the factor calling for identifying whether the investment
    supports customer processes as designed and is delivering the goods
    and services it was designed to deliver, the component analyzed user
    and customer assessments that showed improvement in this area. The
    component partially addressed the key factor on measuring the effect an
    investment has on the performing organization itself. For example,
    National Institutes of Health identified that the investment was in line with
    the appropriate component enterprise architecture, but did not identify the
    effect the investment had on other aspects of the department such as its
    mission and business processes. National Institutes of Health did not


    Page 14                                        GAO-13-87 Information Technology
    address 10 factors, including lessons learned and determining whether to
    modify or terminate the investment. These factors are critical to whether
    to continue an investment that is not performing as required.
    With regard to why analyses were not performed on all investments and
    why those that were conducted did not fully address all factors, DHS and
    HHS attributed these shortfalls to the following:

•   Officials from DHS’s Office of the CIO who are responsible for overseeing
    the performance of OAs department-wide told us the components only
    performed 16 of the 44 analyses and did not address all key factors (in
    the 16 OAs that were performed) because they were not consistently
    implementing department and OMB policy as they should have because
    it was not a priority. To illustrate their point, the officials told us that while
    most components strive to perform annual analyses, other components
    do not require them to be conducted on an annual basis citing other tasks
    as taking precedence. To address these shortfalls, the department
    recently took steps to make OAs a priority and to ensure consistent
    application of department and OMB policy. Specifically, in May 2012,
    DHS’s CIO issued a memorandum stating that all steady state IT
    investments are required to have an annual OA completed no later than
    June of each fiscal year and that component CIOs are to work with
    program managers to implement and ensure compliance with DHS OA
    requirements. Further, as part of this initiative, DHS’s CIO plans to assign
    resources and responsibility to CIO office staff to review and ensure
    compliance with DHS’s policy. These are steps in the right direction;
    however, the DHS CIO officials told us that these initiatives will not be
    fully implemented until sometime in fiscal year 2013.
•   HHS officials from the Office of the CIO said their shortfalls (i.e., one
    component did not perform an OA and those that were performed did not
    address all factors) were due, in part, to inconsistent implementation of
    department and OMB policy across the components due to analyses not
    being a priority. As an example of this, officials from the office of the HHS
    CIO who are responsible for overseeing the department OA program
    cited how they had planned to implement an initiative to annually review
    all analyses performed by the components to ensure consistency and
    quality but have not been able to do so due to limited CIO staff being
    assigned to other initiatives.
    Although DHS and HHS had 23 investments—with collective annual
    budgets of $1.4 billion—that underwent OAs, these investments were not
    thoroughly assessed against all key factors. Until these agencies assess
    all steady state investments and ensure that they are fully assessed
    against factors, there is increased risk that these agencies will not know



    Page 15                                           GAO-13-87 Information Technology
                            whether the multibillion dollar investments fully meet their intended
                            objectives.


DOD, Treasury, and VA Did   DOD, Treasury, and VA had not developed a policy for performing OAs
Not Develop Policies or     and did not conduct OAs for their 23 steady state investments that have
Perform OAs                 combined annual budgets of $2.1 billion. Specifically,

                        •   DOD did not conduct analyses for its 4 major investments that have
                            annual budgets totaling $381 million,
                        •   Treasury did not conduct such analyses for its16 major investments that
                            have annual budgets totaling $152 million, and
                        •   VA did not conduct OAs for its 3 major investments that have annual
                            budgets totaling $1.6 billion.
                            Regarding why DOD and VA had not developed policies and are not
                            performing analyses, officials from those agencies stated that in lieu of
                            conducting OAs, they assess the performance of steady state
                            investments as part of developing their annual exhibit 300 submissions to
                            OMB. While we have previously reported that using the exhibit 300
                            process can be a tool to manage investment performance, 10 our analysis
                            shows that the process does not fully address 11 of the 17 factors. For
                            example, the exhibit 300 process does not fully provide for addressing the
                            following factors:

                        •   identifying alternative methods for achieving the same mission needs and
                            strategic goals. Doing this is important because it helps agencies assess
                            whether they are using the most cost effective solution to achieving
                            agency goals;
                        •   addressing greater utilization of technology or consolidation of
                            investments will better meet organizational goals. It is also critical to
                            helping agencies ensure that their investments are meeting performance
                            goals in the most cost-effective manner;
                        •   identifying lessons learned, why problems occurred, or how savings were
                            realized, which is essential to avoid repeating the same mistakes, which
                            helps saving resources; and
                        •   identifying where the agency needs to redesign, modify, or terminate the
                            investment, which is a means to achieving solutions that return the
                            greatest benefit for funds invested.


                            10
                              GAO, Information Technology: Agencies Need to Improve the Accuracy and Reliability
                            of Investment Information, GAO-06-250 (Washington, D.C: Jan.12, 2006).




                            Page 16                                              GAO-13-87 Information Technology
              Further, OMB officials told us that the exhibit 300 process is not a
              substitute for conducting OAs. Although OMB requires OAs for all steady
              state systems, its guidance does not provide a mechanism for ensuring
              they are completed and submitted to OMB for review. In particular, it does
              not have a reporting mechanism that provides for public transparency into
              the results of these assessments, which the IT Dashboard could provide.
              Having such a mechanism for the performance of steady state systems is
              consistent with Clinger-Cohen Act requirements that call for OMB to
              analyze and report on the performance of IT capital investments.
              Moreover, such public disclosure promotes increased transparency which
              is one of OMB’s goals in establishing the IT Dashboard.

              Treasury officials from the department’s office of the CIO told us they
              decided not to perform OAs in 2011 and instead decided to use the time
              to develop a policy. However, the officials stated that they did not
              anticipate the policy to be completed until the end of this calendar year.

              Until these agencies establish policies and begin performing OA
              assessments for their steady state investments, there is increased risk
              that these agencies will not know whether the multibillion dollar
              investments fully meet their intended objectives, therefore increasing the
              potential for waste and duplication.


              The federal government has made a multibillion dollar commitment to
Conclusions   operating and maintaining its IT investments. OMB has established
              guidance for federal agencies to use to evaluate the performance of such
              investments, including whether a sound basis exists for agencies to
              continue funding them. DHS and HHS had developed policies in
              accordance with OMB guidance and performed analyses, but did not do
              so for all of their investments and their analyses did not address all key
              factors. During the course of this review, DHS reiterated the importance
              of performing OAs and issued a memorandum with initiatives to address
              the department’s shortcomings. Further, DOD, Treasury, and VA had not
              developed a policy nor had they performed OAs. Taken together, these
              five agencies continue to invest billions of dollars each year on IT steady
              state investments without ensuring that they are continuing to meet
              agency needs and are delivering value. These shortcomings are due in
              part to a number of factors, including agencies relying on budget
              submission processes through their annual exhibit 300 submissions,
              which are not intended as a substitute for OAs, and not viewing
              performance of these assessments as a priority. Although OMB’s current
              guidance does not require agencies to report on OAs to OMB, using


              Page 17                                        GAO-13-87 Information Technology
                      existing oversight and transparency tools like the IT Dashboard could
                      help ensure that these important performance assessments are
                      completed and available for public viewing. Nonetheless, until the
                      agencies address these shortcomings and ensure all their steady state
                      investments are fully assessed, there is increased potential for these
                      multibillion dollar investments to result in unnecessary waste and
                      duplication.


                      To ensure that major steady state IT investments are being adequately
Recommendations for   analyzed, we recommend that the Secretaries of Defense, Veterans
Executive Action      Affairs, and the Treasury direct appropriate officials to develop an OA
                      policy, annually perform OAs on all investments, and ensure the
                      assessments include all key factors.

                      In addition, we recommend that the Secretaries of Homeland Security
                      and Health and Human Services direct their Chief Information Officers to
                      ensure OAs are performed annually on all major steady state investments
                      and the assessments include all key factors.

                      Further, to ensure that OA policies are developed and that annual
                      analyses are conducted and to promote transparency into the results of
                      these analyses, we recommend that the Director of OMB revise existing
                      guidance to include directing agencies to report on the IT Dashboard the
                      results from the OAs of their steady state investments.


                      In commenting on a draft of this report, OMB and the five agencies
Agency Comments       agreed with our findings and recommendations. Their comments are
and Our Evaluation    discussed in more detail below.

                      •   In oral comments, staff from OMB’s Office of E-Government and
                          Information Technology concurred with our recommendations and
                          stated that OMB had recently initiated an effort to address the specific
                          recommendation directed to it. Specifically, the staff stated that OMB’s
                          fiscal year 2014 budget guidance (dated August 3, 2012) directs
                          agencies to include OAs as part of their exhibit 300 submissions to
                          OMB.
                      •   In written comments—signed by DOD’s Deputy Chief Information
                          Officer for Information Enterprise and reprinted in appendix III—DOD
                          concurred with our recommendation and said it plans to establish an
                          OA policy in coordination with OMB.




                      Page 18                                        GAO-13-87 Information Technology
•   In written comments—signed by the Director of the Departmental
    GAO-OIG Liaison Office and reprinted in appendix IV—DHS
    concurred with our recommendation. The department, after receiving
    our draft report, identified and provided to us OAs that it had
    performed for 3 additional investments in fiscal year 2011. DHS also
    provided technical comments which we incorporated in the report as
    appropriate.
•   In comments provided via e-mail from its GAO Intake Coordinator
    within the Office of the Assistant Secretary for Legislation, HHS stated
    that it did not have any general comments on the report. The
    department did provide technical comments which we incorporated in
    the report as appropriate.
•   In written comments—signed by the Deputy Assistant Secretary for
    Information Systems and Chief Information Officer and reprinted in
    appendix V—Treasury agreed with the report’s recommendations. In
    addition, after receiving our draft report, the department identified and
    provided to us OAs that it had performed on 9 of its 17 investments in
    fiscal year 2011.
•   In written comments—signed by its Chief of Staff and reprinted in
    appendix VI—VA generally agreed with our conclusions and
    concurred with the recommendation to it.

As agreed with your offices, unless you publicly announce the contents of
this report earlier, we plan no further distribution until 30 days from the
report date. At that time, we will send copies to interested congressional
committees; the Secretaries of Defense, Homeland Security, Health and
Human Services, the Treasury, Veterans Affairs; and other interested
parties. In addition, the report will be available at no charge on the GAO
website at http://www.gao.gov.




Page 19                                        GAO-13-87 Information Technology
If you or your staff have any questions on the matters discussed in this
report, please contact me at (202) 512-9286 or pownerd@gao.gov.
Contact points for our Offices of Congressional Relations and Public
Affairs may be found on the last page of this report. GAO staff who made
major contributions to this report are listed in appendix VII.




David A. Powner
Director, Information Technology
Management Issues




Page 20                                      GAO-13-87 Information Technology
Appendix I: Objective, Scope, and
                 Appendix I: Objective, Scope, and
                 Methodology



Methodology

                 Our objective was to determine the extent to which selected federal
                 agencies assess the performance of steady state information technology
                 (IT) investments in accordance with Office of Management and Budget
                 (OMB) guidance. To accomplish our objective, we selected the five
                 agencies (Departments of Defense (DOD), Health and Human Services
                 (HHS), Homeland Security (DHS), the Treasury (Treasury), and Veterans
                 Affairs (VA)) that have the largest budgets for major steady state IT
                 investments; collectively, these investments accounted for approximately
                 $37 billion annually or about 70 percent of all reported IT operations and
                 maintenance (O&M) spending. In particular, we focused on these
                 agencies’ 75 major IT investments valued at $4.6 billion annually that
                 were strictly in the steady state phase as opposed to the agencies’ other
                 O&M investments—called mixed life-cycle investments by the agencies
                 and OMB—which are not solely in O&M; these mixed life-cycle
                 investments have projects under development as well as projects being
                 placed into O&M.

                 We analyzed OMB’s guidance and identified a key practice called
                 operational analysis (OA) that agencies are to use to assess the
                 performance of existing O&M investments. We also interviewed OMB
                 officials to corroborate our understanding of the key practice.

                 We then determined whether the five agencies developed OA policies as
                 called for by the OMB guidance. Specifically, we compared each
                 agency’s policy, if they had one, to the OMB criteria to determine the
                 extent of compliance and where there were variances. We further
                 determined whether the agencies were conducting analyses. Specifically,
                 for the 75 major investments, we determined whether the agencies had
                 performed an OA on each of them. In those cases where one had been
                 performed, we analyzed the agencies’ efforts to address the OMB criteria
                 in the analysis and categorized the extent to which the OMB key factors
                 had been addressed using the following criteria:

             •   Yes: if all aspects of the key factor specified in the OMB criteria were fully
                 addressed.
             •   No: if none of the aspects of the key factor were addressed.
             •   Partial: if some, but not the entire key factor was addressed.
                 In cases where agencies did not fully address factors (i.e., partially or not
                 all), we analyzed documentation and interviewed agency officials,
                 including staff from the offices of the chief information officers responsible
                 for overseeing these investments, to assist in identifying causes for
                 shortfalls and any actions planned or underway to address the causes.



                 Page 21                                         GAO-13-87 Information Technology
Appendix I: Objective, Scope, and
Methodology




We conducted this performance audit from October 2011 to September
2012 in accordance with generally accepted government auditing
standards. Those standards require that we plan and perform the audit to
obtain sufficient, appropriate evidence to provide a reasonable basis for
our findings and conclusions based on our audit objectives. We believe
that the evidence obtained provides a reasonable basis for our findings
and conclusions based on our audit objective.




Page 22                                      GAO-13-87 Information Technology
Appendix II: Extent to Which DHS’s and
                                              Appendix II: Extent to Which DHS’s and HHS’s
                                              OAs Addressed Key Factors



HHS’s OAs Addressed Key Factors

                                              Tables 5-20 show our analysis for DHS’s investments with OAs in fiscal
                                              year 2011.

Table 5: U.S. Customs and Border Protection: Analysis of Extent to Which Infrastructure’s Fiscal Year 2011 OA Addressed
OMB Key Factors

                                                                          Yes,
                                                                          no, or
Key factor                                                                partial Summary of partial rating
1. Assesses current costs against life-cycle cost                         N
2. Includes a structured schedule assessment                              N
3. Includes a structured assessment of performance goals                  Y
4. Identifies whether the investment supports customer processes as       Y
designed and is delivering the goods and services it was designed to
deliver
5. Measures the effect the investment has on the performing               Y
organization itself
6. Includes a measure of how well the investment contributes to           Y
achieving the organization’s business needs and strategic goals
7. Compares current performance with a pre-established cost baseline P           U.S. Customs and Border Protection included an
and estimates                                                                    analysis of performance measures against a
                                                                                 performance baseline for, among other things,
                                                                                 software and hardware maintenance and network
                                                                                 availability; however, a measure of cost against its
                                                                                 baseline was not included.
8. Identifies any areas for innovation in the areas of customer           Y
satisfaction, strategic and business results, and financial performance
9. Identifies if the agency revisited alternative methods for achieving   N
the same mission needs and strategic goals
10. Addresses issues such as greater utilization of technology or         Y
consolidation of investments to better meet organizational goals
11. Includes an ongoing review of the status of the risks identified in   Y
the investment’s planning and acquisition phases
12. Identifies a need to redesign, modify, or terminate the investment    P      U.S. Customs and Border Protection found that the
                                                                                 investment was currently meeting established
                                                                                 performance goals. The assessment did not include
                                                                                 an analysis on whether opportunities to improve the
                                                                                 system’s efficiency had been identified.
13. Includes an analysis on the need for improved methodology (i.e.,      N
better ways for the investment to meet cost and performance goals)
14. Identifies any lessons learned                                        N
15. Identifies if the investment had a cost or schedule variance          N
16. Identifies recommendations to redesign or modify an asset before      Y
it becomes a problem
17. Includes information on the overlap of the investment with other      N
systems




                                              Page 23                                                GAO-13-87 Information Technology
                                              Appendix II: Extent to Which DHS’s and HHS’s
                                              OAs Addressed Key Factors




                                              Key:        Y= fully addressed the key factor
                                                          P= partially addressed the key factor
                                                          N= did not address the key factor
                                              Source: GAO analysis of DHS data.



Table 6: Office of the Chief Information Officer: Analysis of Extent to Which Homeland Secure Data Network’s Fiscal Year
2011 OA Addressed OMB Key Factors

                                                                                  Yes,
                                                                                  no, or
Key factor                                                                        partial Summary of partial rating
1. Assesses current costs against life-cycle cost                                 N
2. Includes a structured schedule assessment                                      N
3. Includes a structured assessment of performance goals                          Y
4. Identifies whether the investment supports customer processes as               Y
designed and is delivering the goods and services it was designed to
deliver
5. Measures the effect the investment has on the performing                       Y
organization itself
6. Includes a measure of how well the investment contributes to                   P      The Office of the Chief Information Officer identified
achieving the organization’s business needs and strategic goals                          which strategic goal the investment supports, namely
                                                                                         to “Strengthen and Unify DHS Operations and
                                                                                         Management,” but it did not include measures of how
                                                                                         well the investment contributes to achieving the goal.
7. Compares current performance with a pre-established cost baseline P                   The Office of the Chief Information Officer assessed
and estimates                                                                            current performance with pre-established
                                                                                         performance measures, but it did not analyze current
                                                                                         cost performance against a cost baseline.
8. Identifies any areas for innovation in the areas of customer                   Y
satisfaction, strategic and business results, and financial performance
9. Identifies if the agency revisited alternative methods for achieving           N
the same mission needs and strategic goals
10. Addresses issues such as greater utilization of technology or                 Y
consolidation of investments to better meet organizational goals
11. Includes an ongoing review of the status of the risks identified in           P      The Office of the Chief Information Officer identified
the investment’s planning and acquisition phases                                         the investment’s risk management plan, as
                                                                                         containing current risks, their status, and associated
                                                                                         mitigation efforts. It also identified that current risks
                                                                                         are reported at monthly meetings; however, it did not
                                                                                         address specific risks in its assessment and the
                                                                                         impact of these risks.
12. Identifies a need to redesign, modify, or terminate the investment            Y
13. Includes an analysis on the need for improved methodology (i.e.,              N
better ways for the investment to meet cost and performance goals)
14. Identifies any lessons learned                                                N
15. Identifies if the investment had a cost or schedule variance                  N




                                              Page 24                                                         GAO-13-87 Information Technology
                                             Appendix II: Extent to Which DHS’s and HHS’s
                                             OAs Addressed Key Factors




16. Identifies recommendations to redesign or modify an asset before                 N
it becomes a problem
17. Includes information on the overlap of the investment with other                 N
systems

                                             Key:        Y= fully addressed the key factor
                                                         P= partially addressed the key factor
                                                         N= did not address the key factor
                                             Source: GAO analysis of DHS data.




Table 7: Federal Emergency Management Agency: Analysis of Extent to Which Disaster Management E-Government
Initiative’s Fiscal Year 2011 OA Addressed OMB Key Factors

                                                                                 Yes,
                                                                                 no, or
Key factor                                                                       partial Summary of partial rating
1. Assesses current costs against life-cycle cost                                N
2. Includes a structured schedule assessment                                     N
3. Includes a structured assessment of performance goals                         N
4. Identifies whether the investment supports customer processes                 N
as designed and is delivering the goods and services it was
designed to deliver
5. Measures the effect the investment has on the performing                      P       Federal Emergency Management Agency identified
organization itself                                                                      the strategic mission, goals, and objectives that this
                                                                                         investment supports, but it did not measure the effect
                                                                                         the investment is having on the department and its
                                                                                         mission.
6. Includes a measure of how well the investment contributes to                  P       Federal Emergency Management Agency identified
achieving the organization’s business needs and strategic goals                          the strategic mission, goals, and objectives that this
                                                                                         investment supports, but it did not provide any analysis
                                                                                         on metrics showing the extent to which the investment
                                                                                         is contributing to achieving the department’s business
                                                                                         needs and strategic goals.
7. Compares current performance with a pre-established cost                      N
baseline and estimates
8. Identifies any areas for innovation in the areas of customer                  Y
satisfaction, strategic and business results, and financial
performance
9. Identifies if the agency revisited alternative methods for achieving N
the same mission needs and strategic goals
10. Addresses issues such as greater utilization of technology or                Y
consolidation of investments to better meet organizational goals




                                             Page 25                                                          GAO-13-87 Information Technology
                                              Appendix II: Extent to Which DHS’s and HHS’s
                                              OAs Addressed Key Factors




11. Includes an ongoing review of the status of the risks identified in           P             Federal Emergency Management Agency identified
the investment’s planning and acquisition phases                                                how the investment’s project management office
                                                                                                monitored risks through a risk management plan and
                                                                                                risk register, which is to be updated biweekly, but it did
                                                                                                not assess risks against investment cost, schedule,
                                                                                                and performance.
12. Identifies a need to redesign, modify, or terminate the                       N
investment
13. Includes an analysis on the need for improved methodology (i.e., N
better ways for the investment to meet cost and performance goals)
14. Identifies any lessons learned                                                N
15. Identifies if the investment had a cost or schedule variance                  N
16. Identifies recommendations to redesign or modify an asset                     Y
before it becomes a problem
17. Includes information on the overlap of the investment with other              N
systems

                                              Key:        Y= fully addressed the key factor
                                                          P= partially addressed the key factor
                                                          N= did not address the key factor
                                              Source: GAO analysis of DHS data.




Table 8: Federal Emergency Management Agency: Analysis of Extent to Which Integrated Financial Management Information
System’s Fiscal Year 2011 OA Addressed OMB Key Factors

                                                                                      Yes,
                                                                                      no, or
Key factor                                                                            partial     Summary of partial rating
1. Assesses current costs against life cycle cost                                     N
2. Includes a structured schedule assessment                                          N
3. Includes a structured assessment of performance goals                              Y
4. Identifies whether the investment supports customer processes as                   Y
designed and is delivering the goods and services it was designed to
deliver
5. Measures the effect the investment has on the performing                           Y
organization itself
6. Includes a measure of how well the investment contributes to                       Y
achieving the organization’s business needs and strategic goals
7. Compares current performance with a pre-established cost baseline N
and estimates
8. Identifies any areas for innovation in the areas of customer                       Y
satisfaction, strategic and business results, and financial performance
9. Identifies if the agency revisited alternative methods for achieving               Y
the same mission needs and strategic goals




                                              Page 26                                                                 GAO-13-87 Information Technology
                                              Appendix II: Extent to Which DHS’s and HHS’s
                                              OAs Addressed Key Factors




10. Addresses issues such as greater utilization of technology or                 Y
consolidation of investments to better meet organizational goals
11. Includes an ongoing review of the status of the risks identified in           P       Federal Emergency Management Agency used the
the investment’s planning and acquisition phases                                          investment’s risk management plan to identify risks
                                                                                          and potential impact on schedule milestones, but
                                                                                          this assessment did not identify the risk and
                                                                                          potential impact on cost.
12. Identifies a need to redesign, modify, or terminate the investment            Y
13. Includes an analysis on the need for improved methodology (i.e.,              Y
better ways for the investment to meet cost and performance goals)
14. Identifies any lessons learned                                                N
15. Identifies if the investment had a cost or schedule variance                  N
16. Identifies recommendations to redesign or modify an asset before              Y
it becomes a problem
17. Includes information on the overlap of the investment with other              N
systems

                                              Key:        Y= fully addressed the key factor
                                                          P= partially addressed the key factor
                                                          N= did not address the key factor
                                              Source: GAO analysis of DHS data.




Table 9: U.S. Immigration and Customs Enforcement: Analysis of Extent to Which Intelligence Fusion System’s Fiscal Year
2011 OA Addressed OMB Key Factors

                                                                                  Yes,
                                                                                  no, or
Key factor                                                                        partial Summary of partial rating
1. Assesses current costs against life-cycle cost                                 Y
2. Includes a structured schedule assessment                                      Y
3. Includes a structured assessment of performance goals                          Y
4. Identifies whether the investment supports customer processes as               Y
designed and is delivering the goods and services it was designed to
deliver
5. Measures the effect the investment has on the performing                       Y
organization itself
6. Includes a measure of how well the investment contributes to                   Y
achieving the organization’s business needs and strategic goals
7. Compares current performance with a pre-established cost baseline Y
and estimates




                                              Page 27                                                        GAO-13-87 Information Technology
                                              Appendix II: Extent to Which DHS’s and HHS’s
                                              OAs Addressed Key Factors




8. Identifies any areas for innovation in the areas of customer                   P     U.S. Immigration and Customs Enforcement
satisfaction, strategic and business results, and financial performance                 identified opportunities for innovation in the areas of
                                                                                        strategic and business results and customer
                                                                                        satisfaction; it did not address whether there were
                                                                                        any innovation opportunities in the area of financial
                                                                                        performance.
9. Identifies if the agency revisited alternative methods for achieving           N
the same mission needs and strategic goals
10. Addresses issues such as greater utilization of technology or                 Y
consolidation of investments to better meet organizational goals
11. Includes an ongoing review of the status of the risks identified in           Y
the investment’s planning and acquisition phases
12. Identifies a need to redesign, modify, or terminate the investment            N
13. Includes an analysis on the need for improved methodology (i.e.,              N
better ways for the investment to meet cost and performance goals)
14. Identifies any lessons learned                                                N
15. Identifies if the investment had a cost or schedule variance                  N
16. Identifies recommendations to redesign or modify an asset before              N
it becomes a problem
17. Includes information on the overlap of the investment with other              N
systems

                                              Key:        Y= fully addressed the key factor
                                                          P= partially addressed the key factor
                                                          N= did not address the key factor
                                              Source: GAO analysis of DHS data.




Table 10: National Protection and Programs Directorate: Analysis of Extent to Which National Security and Emergency
Preparedness Priority Telecommunication Services’s Fiscal Year 2011 OA Addressed OMB Key Factors

                                                                                  Yes,
                                                                                  no, or
Key factor                                                                        partial Summary of partial rating
1. Assesses current costs against life-cycle cost                                 N
2. Includes a structured schedule assessment                                      N
3. Includes a structured assessment of performance goals                          Y
4. Identifies whether the investment supports customer processes as               Y
designed and is delivering the goods and services it was designed to
deliver
5. Measures the effect the investment has on the performing                       Y
organization itself
6. Includes a measure of how well the investment contributes to                   Y
achieving the organization’s business needs and strategic goals
7. Compares current performance with a pre-established cost baseline              N
and estimates




                                              Page 28                                                        GAO-13-87 Information Technology
                                             Appendix II: Extent to Which DHS’s and HHS’s
                                             OAs Addressed Key Factors




8. Identifies any areas for innovation in the areas of customer                   Y
satisfaction, strategic and business results, and financial performance
9. Identifies if the agency revisited alternative methods for achieving the Y
same mission needs and strategic goals
10. Addresses issues such as greater utilization of technology or                 Y
consolidation of investments to better meet organizational goals
11. Includes an ongoing review of the status of the risks identified in the N
investment’s planning and acquisition phases
12. Identifies a need to redesign, modify, or terminate the investment            Y
13. Includes an analysis on the need for improved methodology (i.e.,              N
better ways for the investment to meet cost and performance goals)
14. Identifies any lessons learned                                                N
15. Identifies if the investment had a cost or schedule variance                  Y
16. Identifies recommendations to redesign or modify an asset before it           Y
becomes a problem
17. Includes information on the overlap of the investment with other              N
systems

                                             Key:        Y= fully addressed the key factor
                                                         P= partially addressed the key factor
                                                         N= did not address the key factor
                                             Source: GAO analysis of DHS data.




Table 11: Transportation Security Administration: Analysis of Extent to Which Federal Air Marshal Service Mission
Scheduling and Notification System’s Fiscal Year 2011 OA Addressed OMB Key Factors

                                                                                 Yes,
                                                                                 no, or
Key factor                                                                       partial Summary of partial rating
1. Assesses current costs against life-cycle cost                                Y
2. Includes a structured schedule assessment                                     P      Transportation Security Administration developed a
                                                                                        schedule performance table and a framework to
                                                                                        analyze results, but the schedule assessment
                                                                                        omitted analysis of key schedule aspects, including
                                                                                        actual start dates and percent completed.
3. Includes a structured assessment of performance goals                         Y
4. Identifies whether the investment supports customer processes as              Y
designed and is delivering the goods and services it was designed to
deliver
5. Measures the effect the investment has on the performing                      Y
organization itself
6. Includes a measure of how well the investment contributes to                  Y
achieving the organization’s business needs and strategic goals
7. Compares current performance with a pre-established cost baseline Y
and estimates




                                             Page 29                                                       GAO-13-87 Information Technology
                                              Appendix II: Extent to Which DHS’s and HHS’s
                                              OAs Addressed Key Factors




8. Identifies any areas for innovation in the areas of customer                   Y
satisfaction, strategic and business results, and financial performance
9. Identifies if the agency revisited alternative methods for achieving           N
the same mission needs and strategic goals
10. Addresses issues such as greater utilization of technology or                 Y
consolidation of investments to better meet organizational goals
11. Includes an ongoing review of the status of the risks identified in           Y
the investment’s planning and acquisition phases
12. Identifies a need to redesign, modify, or terminate the investment            Y
13. Includes an analysis on the need for improved methodology (i.e.,              N
better ways for the investment to meet cost and performance goals)
14. Identifies any lessons learned                                                N
15. Identifies if the investment had a cost or schedule variance                  N
16. Identifies recommendations to redesign or modify an asset before              Y
it becomes a problem
17. Includes information on the overlap of the investment with other              N
systems

                                              Key:        Y= fully addressed the key factor
                                                          P= partially addressed the key factor
                                                          N= did not address the key factor
                                              Source: GAO analysis of DHS data.




Table 12: Transportation Security Administration: Analysis of Extent to Which Federal Air Marshal Service Network’s Fiscal
Year 2011 OA Addressed OMB Key Factors

                                                                                  Yes,
                                                                                  no, or
Key factor                                                                        partial Summary of partial rating
1. Assesses current costs against life-cycle cost                                 Y
2. Includes a structured schedule assessment                                      Y
3. Includes a structured assessment of performance goals                          Y
4. Identifies whether the investment supports customer processes as               Y
designed and is delivering the goods and services it was designed to
deliver
5. Measures the effect the investment has on the performing                       Y
organization itself
6. Includes a measure of how well the investment contributes to                   Y
achieving the organization’s business needs and strategic goals
7. Compares current performance with a pre-established cost baseline Y
and estimates
8. Identifies any areas for innovation in the areas of customer                   Y
satisfaction, strategic and business results, and financial performance




                                              Page 30                                                      GAO-13-87 Information Technology
                                              Appendix II: Extent to Which DHS’s and HHS’s
                                              OAs Addressed Key Factors




9. Identifies if the agency revisited alternative methods for achieving           Y
the same mission needs and strategic goals
10. Addresses issues such as greater utilization of technology or                 Y
consolidation of investments to better meet organizational goals
11. Includes an ongoing review of the status of the risks identified in           N
the investment’s planning and acquisition phases
12. Identifies a need to redesign, modify, or terminate the investment            N
13. Includes an analysis on the need for improved methodology (i.e.,              N
better ways for the investment to meet cost and performance goals)
14. Identifies any lessons learned                                                N
15. Identifies if the investment had a cost or schedule variance                  N
16. Identifies recommendations to redesign or modify an asset before              N
it becomes a problem
17. Includes information on the overlap of the investment with other              N
systems

                                              Key:        Y= fully addressed the key factor
                                                          P= partially addressed the key factor
                                                          N= did not address the key factor
                                              Source: GAO analysis of DHS data.




Table 13: Transportation Security Administration: Analysis of Extent to Which Hazardous Materials Endorsement Threat
Assessment Program’s Fiscal Year 2011 OA Addressed OMB Key Factors

                                                                                  Yes,
                                                                                  no, or
Key factor                                                                        partial Summary of partial rating
1. Assesses current costs against life-cycle cost                                 Y
2. Includes a structured schedule assessment                                      Y
3. Includes a structured assessment of performance goals                          Y
4. Identifies whether the investment supports customer processes as               Y
designed and is delivering the goods and services it was designed to
deliver
5. Measures the effect the investment has on the performing                       Y
organization itself
6. Includes a measure of how well the investment contributes to                   Y
achieving the organization’s business needs and strategic goals
7. Compares current performance with a pre-established cost baseline Y
and estimates
8. Identifies any areas for innovation in the areas of customer                   Y
satisfaction, strategic and business results, and financial performance
9. Identifies if the agency revisited alternative methods for achieving           Y
the same mission needs and strategic goals
10. Addresses issues such as greater utilization of technology or                 Y
consolidation of investments to better meet organizational goals




                                              Page 31                                                      GAO-13-87 Information Technology
                                              Appendix II: Extent to Which DHS’s and HHS’s
                                              OAs Addressed Key Factors




11. Includes an ongoing review of the status of the risks identified in           Y
the investment’s planning and acquisition phases
12. Identifies a need to redesign, modify, or terminate the investment            Y
13. Includes an analysis on the need for improved methodology (i.e.,              Y
better ways for the investment to meet cost and performance goals)
14. Identifies any lessons learned                                                Y
15. Identifies if the investment had a cost or schedule variance                  N
16. Identifies recommendations to redesign or modify an asset before              N
it becomes a problem
17. Includes information on the overlap of the investment with other              N
systems

                                              Key:        Y= fully addressed the key factor
                                                          P= partially addressed the key factor
                                                          N= did not address the key factor
                                              Source: GAO analysis of DHS data.




Table 14: Transportation Security Administration: Analysis of Extent to Which Information Technology Infrastructure
Program’s Fiscal Year 2011 OA Addressed OMB Key Factors

                                                                                  Yes,
                                                                                  no, or
Key factor                                                                        partial Summary of partial rating
1. Assesses current costs against life-cycle cost                                 Y
2. Includes a structured schedule assessment                                      Y
3. Includes a structured assessment of performance goals                          Y
4. Identifies whether the investment supports customer processes as               P       Transportation Security Administration stated that
designed and is delivering the goods and services it was designed to                      customer satisfaction is to be measured using
deliver                                                                                   surveys, service legal agreements, and key
                                                                                          performance measures. However, the assessment
                                                                                          did not include results of these efforts, and thus did
                                                                                          not identify if the investment was meeting customer
                                                                                          needs.
5. Measures the effect the investment has on the performing                       Y
organization itself
6. Includes a measure of how well the investment contributes to                   Y
achieving the organization’s business needs and strategic goals
7. Compares current performance with a pre-established cost baseline Y
and estimates
8. Identifies any areas for innovation in the areas of customer                   N
satisfaction, strategic and business results, and financial performance
9. Identifies if the agency revisited alternative methods for achieving           N
the same mission needs and strategic goals
10. Addresses issues such as greater utilization of technology or                 Y
consolidation of investments to better meet organizational goals




                                              Page 32                                                         GAO-13-87 Information Technology
                                              Appendix II: Extent to Which DHS’s and HHS’s
                                              OAs Addressed Key Factors




11. Includes an ongoing review of the status of the risks identified in           N
the investment’s planning and acquisition phases
12. Identifies a need to redesign, modify, or terminate the investment            Y
13. Includes an analysis on the need for improved methodology (i.e.,              N
better ways for the investment to meet cost and performance goals)
14. Identifies any lessons learned                                                N
15. Identifies if the investment had a cost or schedule variance                  N
16. Identifies recommendations to redesign or modify an asset before              N
it becomes a problem
17. Includes information on the overlap of the investment with other              N
systems

                                              Key:        Y= fully addressed the key factor
                                                          P= partially addressed the key factor
                                                          N= did not address the key factor
                                              Source: GAO analysis of DHS data.




Table 15: Transportation Security Administration: Analysis of Extent to Which Secure Flight’s Fiscal Year 2011 OA Addressed
OMB Key Factors

                                                                                  Yes,
                                                                                  no, or
Key factor                                                                        partial Summary of partial rating
1. Assesses current costs against life cycle cost                                 Y
2. Includes a structured schedule assessment                                      Y
3. Includes a structured assessment of performance goals                          Y
4. Identifies whether the investment supports customer processes as               Y
designed and is delivering the goods and services it was designed to
deliver
5. Measures the effect the investment has on the performing                       Y
organization itself
6. Includes a measure of how well the investment contributes to                   Y
achieving the organization’s business needs and strategic goals
7. Compares current performance with a pre-established cost baseline Y
and estimates
8. Identifies any areas for innovation in the areas of customer                   Y
satisfaction, strategic and business results, and financial performance
9. Identifies if the agency revisited alternative methods for achieving           N
the same mission needs and strategic goals
10. Addresses issues such as greater utilization of technology or                 Y
consolidation of investments to better meet organizational goals
11. Includes an ongoing review of the status of the risks identified in           Y
the investment’s planning and acquisition phases
12. Identifies a need to redesign, modify, or terminate the investment            Y




                                              Page 33                                                      GAO-13-87 Information Technology
                                              Appendix II: Extent to Which DHS’s and HHS’s
                                              OAs Addressed Key Factors




13. Includes an analysis on the need for improved methodology (i.e.,              N
better ways for the investment to meet cost and performance goals
14. Identifies any lessons learned                                                N
15. Identifies if the investment had a cost or schedule variance                  N
16. Identifies recommendations to redesign or modify an asset before              Y
it becomes a problem
17. Includes information on the overlap of the investment with other              N
systems

                                              Key:        Y= fully addressed the key factor
                                                          P= partially addressed the key factor
                                                          N= did not address the key factor
                                              Source: GAO analysis of DHS data.




Table 16: U.S. Coast Guard: Analysis of Extent to Which Coast Guard Business Intelligence’s Fiscal Year 2011 OA Addressed
OMB Key Factors

                                                                                   Yes,
                                                                                  no, or
Key factor                                                                        partial Summary of partial rating
1. Assesses current costs against life-cycle cost                                     N
2. Includes a structured schedule assessment                                          N
3. Includes a structured assessment of performance goals                              P   In its assessment, U.S. Coast Guard described how
                                                                                          it has established two major goals for the
                                                                                          investment, namely, to provide products and
                                                                                          services which leverage standardized Enterprise
                                                                                          Measures while ensuring repeatable answers, and
                                                                                          to provide Enterprise Solutions for lower level
                                                                                          reporting requirements; but the department did not
                                                                                          analyze the performance to date against these
                                                                                          established investment goals.
4. Identifies whether the investment supports customer processes as                   Y
designed and is delivering the goods and services it was designed to
deliver
5. Measures the effect the investment has on the performing                           P   U.S. Coast Guard described the investment as
organization itself                                                                       organizational knowledge that is directly relevant to
                                                                                          decision making towards organizational goals, but it
                                                                                          did not measure the effect the investment had on the
                                                                                          component in performing its mission.
6. Includes a measure of how well the investment contributes to                       Y
achieving the organization’s business needs and strategic goals
7. Compares current performance with a pre-established cost baseline                  N
and estimates
8. Identifies any areas for innovation in the areas of customer                       Y
satisfaction, strategic and business results, and financial performance
9. Identifies if the agency revisited alternative methods for achieving               N
the same mission needs and strategic goals




                                              Page 34                                                        GAO-13-87 Information Technology
                                              Appendix II: Extent to Which DHS’s and HHS’s
                                              OAs Addressed Key Factors




10. Addresses issues such as greater utilization of technology or                     Y
consolidation of investments to better meet organizational goals
11. Includes an ongoing review of the status of the risks identified in               N
the investment’s planning and acquisition phases
12. Identifies a need to redesign, modify, or terminate the investment                Y
13. Includes an analysis on the need for improved methodology (i.e.,                  N
better ways for the investment to meet cost and performance goals)
14. Identifies any lessons learned                                                    N
15. Identifies if the investment had a cost or schedule variance                      N
16. Identifies recommendations to redesign or modify an asset before                  N
it becomes a problem
17. Includes information on the overlap of the investment with other              P           In its assessment, U.S. Coast Guard identified an
systems                                                                                       area of focus for fiscal year 2011 that was to
                                                                                              determine how the investment aligns
                                                                                              strategically/tactically with other business
                                                                                              intelligence investments. U.S. Coast Guard did
                                                                                              not identify whether the Business Intelligence
                                                                                              investment overlapped or duplicated functions
                                                                                              performed by other business intelligence
                                                                                              investments.

                                              Key:        Y= fully addressed the key factor
                                                          P= partially addressed the key factor
                                                          N= did not address the key factor
                                              Source: GAO analysis of DHS data.




Table 17: U.S. Coast Guard: Analysis of Extent to Which Core Accounting System Suite’s Fiscal Year 2011 OA Addressed
OMB Key Factors

                                                                                  Yes,
                                                                                  no, or
Key factor                                                                        partial Summary of partial rating
1. Assesses current costs against life-cycle cost                                 N
2. Includes a structured schedule assessment                                      N
3. Includes a structured assessment of performance goals                          Y
4. Identifies whether the investment supports customer processes as               Y
designed and is delivering the goods and services it was designed to
deliver
5. Measures the effect the investment has on the performing                       Y
organization itself
6. Includes a measure of how well the investment contributes to                   Y
achieving the organization’s business needs and strategic goals
7. Compares current performance with a pre-established cost baseline N
and estimates




                                              Page 35                                                         GAO-13-87 Information Technology
                                              Appendix II: Extent to Which DHS’s and HHS’s
                                              OAs Addressed Key Factors




8. Identifies any areas for innovation in the areas of customer                   Y
satisfaction, strategic and business results, and financial performance
9. Identifies if the agency revisited alternative methods for achieving           N
the same mission needs and strategic goals
10. Addresses issues such as greater utilization of technology or                 Y
consolidation of investments to better meet organizational goals
11. Includes an ongoing review of the status of the risks identified in           N
the investment’s planning and acquisition phases
12. Identifies a need to redesign, modify, or terminate the investment            Y
13. Includes an analysis on the need for improved methodology (i.e.,              N
better ways for the investment to meet cost and performance goals)
14. Identifies any lessons learned                                                N
15. Identifies if the investment had a cost or schedule variance                  Y
16. Identifies recommendations to redesign or modify an asset before              N
it becomes a problem
17. Includes information on the overlap of the investment with other              Y
systems

                                              Key:        Y= fully addressed the key factor
                                                          P= partially addressed the key factor
                                                          N= did not address the key factor
                                              Source: GAO analysis of DHS data.




Table 18: U.S. Coast Guard: Analysis of Extent to Which Direct Access’s Fiscal Year 2011 OA Addressed OMB Key Factors

                                                                                  Yes,
                                                                                  no, or
Key factor                                                                        partial Summary of partial rating
1. Assesses current costs against life-cycle cost                                 Y
2. Includes a structured schedule assessment                                      Y
3. Includes a structured assessment of performance goals                          Y
4. Identifies whether the investment supports customer processes as               Y
designed and is delivering the goods and services it was designed to
deliver
5. Measures the effect the investment has on the performing                       Y
organization itself
6. Includes a measure of how well the investment contributes to                   Y
achieving the organization’s business needs and strategic goals
7. Compares current performance with a pre-established cost baseline N
and estimates
8. Identifies any areas for innovation in the areas of customer                   Y
satisfaction, strategic and business results, and financial performance
9. Identifies if the agency revisited alternative methods for achieving           Y
the same mission needs and strategic goals




                                              Page 36                                                      GAO-13-87 Information Technology
                                              Appendix II: Extent to Which DHS’s and HHS’s
                                              OAs Addressed Key Factors




10. Addresses issues such as greater utilization of technology or                 Y
consolidation of investments to better meet organizational goals
11. Includes an ongoing review of the status of the risks identified in           N
the investment’s planning and acquisition phases
12. Identifies a need to redesign, modify, or terminate the investment            Y
13. Includes an analysis on the need for improved methodology (i.e.,              Y
better ways for the investment to meet cost and performance goals)
14. Identifies any lessons learned                                                N
15. Identifies if the investment had a cost or schedule variance                  N
16. Identifies recommendations to redesign or modify an asset before              Y
it becomes a problem
17. Includes information on the overlap of the investment with other              Y
systems

                                              Key:        Y= fully addressed the key factor
                                                          P= partially addressed the key factor
                                                          N= did not address the key factor
                                              Source: GAO analysis of DHS data.




Table 19: U.S. Coast Guard: Analysis of Extent to Which Marine Information for Safety and Law Enforcement’s Fiscal Year
2011 OA Addressed OMB Key Factors

                                                                                  Yes,
                                                                                  no, or
Key factor                                                                        partial Summary of partial rating
1. Assesses current costs against life-cycle cost                                 N
2. Includes a structured schedule assessment                                      N
3. Includes a structured assessment of performance goals                          P      U.S. Coast Guard provided a summary of the
                                                                                         investments’ performance goals for the past year,
                                                                                         including significant deliverables, but it did not
                                                                                         include how or when these goals were to be
                                                                                         achieved nor did it include if any other goals
                                                                                         remained outstanding.
4. Identifies whether the investment supports customer processes as               Y
designed and is delivering the goods and services it was designed to
deliver
5. Measures the effect the investment has on the performing                       Y
organization itself
6. Includes a measure of how well the investment contributes to                   Y
achieving the organization’s business needs and strategic goals
7. Compares current performance with a pre-established cost baseline N
and estimates
8. Identifies any areas for innovation in the areas of customer                   Y
satisfaction, strategic and business results, and financial performance




                                              Page 37                                                       GAO-13-87 Information Technology
                                              Appendix II: Extent to Which DHS’s and HHS’s
                                              OAs Addressed Key Factors




9. Identifies if the agency revisited alternative methods for achieving           Y
the same mission needs and strategic goals
10. Addresses issues such as greater utilization of technology or                 Y
consolidation of investments to better meet organizational goals
11. Includes an ongoing review of the status of the risks identified in           N
the investment’s planning and acquisition phases
12. Identifies a need to redesign, modify, or terminate the investment            Y
13. Includes an analysis on the need for improved methodology (i.e.,              N
better ways for the investment to meet cost and performance goals)
14. Identifies any lessons learned                                                N
15. Identifies if the investment had a cost or schedule variance                  N
16. Identifies recommendations to redesign or modify an asset before              N
it becomes a problem
17. Includes information on the overlap of the investment with other              N
systems

                                              Key:        Y= fully addressed the key factor
                                                          P= partially addressed the key factor
                                                          N= did not address the key factor
                                              Source: GAO analysis of DHS data.




Table 20: U.S. Citizenship and Immigration Services: Analysis of Extent to Which Immigration Computer Linked Application
Information Management System’s Fiscal Year 2011 OA Addressed OMB Key Factors

                                                                                  Yes,
                                                                                  no, or
Key factor                                                                        partial Summary of partial rating
                                                                                  P      U.S. Citizenship and Immigration Services identified
                                                                                         cost centers and said that they managed costs of the
                                                                                         program well, but did not assess current costs
1. Assesses current costs against life-cycle cost                                        against life-cycle costs.
2. Includes a structured schedule assessment                                      P      U.S. Citizenship and Immigration Services identified
                                                                                         schedule milestones, the management of scheduling,
                                                                                         and identified assurance of the project moving
                                                                                         forward on schedule, but it did not measure current
                                                                                         progress on milestones against the established
                                                                                         schedule.
3. Includes a structured assessment of performance goals                          Y
4. Identifies whether the investment supports customer processes as               Y
designed and is delivering the goods and services it was designed to
deliver
5. Measures the effect the investment has on the performing                       Y
organization itself
6. Includes a measure of how well the investment contributes to                   Y
achieving the organization’s business needs and strategic goals




                                              Page 38                                                       GAO-13-87 Information Technology
                                              Appendix II: Extent to Which DHS’s and HHS’s
                                              OAs Addressed Key Factors




7. Compares current performance with a pre-established cost baseline P                  The assessment identified the financial management
and estimates                                                                           background of the system, its scope, and major
                                                                                        costs. It also identified the financial performance
                                                                                        results, and identified planned actions to improve
                                                                                        financial performance, but it did not provide a
                                                                                        comparison between the pre-established cost
                                                                                        baseline and current performance.
8. Identifies any areas for innovation in the areas of customer                   Y
satisfaction, strategic and business results, and financial performance
9. Identifies if the agency revisited alternative methods for achieving           Y
the same mission needs and strategic goals
10. Addresses issues such as greater utilization of technology or                 Y
consolidation of investments to better meet organizational goals
11. Includes an ongoing review of the status of the risks identified in           P     U.S. Citizenship and Immigration Services identified
the investment’s planning and acquisition phases                                        risks associated with the system, including risks that
                                                                                        could affect schedule, among other things; however,
                                                                                        it did not detail potential costs associated with the
                                                                                        risks.
12. Identifies a need to redesign, modify, or terminate the investment            Y
13. Includes an analysis on the need for improved methodology (i.e.,              N
better ways for the investment to meet cost and performance goals)
14. Identifies any lessons learned                                                Y
15. Identifies if the investment had a cost or schedule variance                  N
16. Identifies recommendations to redesign or modify an asset before              N
it becomes a problem
17. Includes information on the overlap of the investment with other              Y
systems

                                              Key:        Y= fully addressed the key factor
                                                          P= partially addressed the key factor
                                                          N= did not address the key factor
                                              Source: GAO analysis of DHS data.




                                              Tables 21-27 show our analysis for HHS’s investments with OAs in fiscal
                                              year 2011.




                                              Page 39                                                       GAO-13-87 Information Technology
                                              Appendix II: Extent to Which DHS’s and HHS’s
                                              OAs Addressed Key Factors




Table 21: Centers for Disease Control and Prevention: Analysis of Extent to Which National Select Agency Registry’s Fiscal
Year 2011 OA Addressed OMB Key Factors

                                                                                  Yes,
                                                                                  no, or
Key factor                                                                        partial Summary of partial rating
1. Assesses current costs against life-cycle cost                                 Y
2. Includes a structured schedule assessment                                      N
3. Includes a structured assessment of performance goals                          Y
4. Identifies whether the investment supports customer processes as               N
designed and is delivering the goods and services it was designed to
deliver
5. Measures the effect the investment has on the performing                       Y
organization itself
6. Includes a measure of how well the investment contributes to                   Y
achieving the organization’s business needs and strategic goals
7. Compares current performance with a pre-established cost baseline N
and estimates
8. Identifies any areas for innovation in the areas of customer                   Y
satisfaction, strategic and business results, and financial performance
9. Identifies if the agency revisited alternative methods for achieving           Y
the same mission needs and strategic goals
10. Addresses issues such as greater utilization of technology or                 Y
consolidation of investments to better meet organizational goals
11. Includes an ongoing review of the status of the risks identified in           N
the investment’s planning and acquisition phases
12. Identifies a need to redesign, modify, or terminate the investment            Y
13. Includes an analysis on the need for improved methodology (i.e.,              N
better ways for the investment to meet cost and performance goals)
14. Identifies any lessons learned                                                N
15. Identifies if the investment had a cost or schedule variance                  N
16. Identifies recommendations to redesign or modify an asset before              Y
it becomes a problem
17. Includes information on the overlap of the investment with other              N
systems

                                              Key:        Y= fully addressed the key factor
                                                          P= partially addressed the key factor
                                                          N= did not address the key factor
                                              Source: GAO analysis of HHS data.




                                              Page 40                                                      GAO-13-87 Information Technology
                                              Appendix II: Extent to Which DHS’s and HHS’s
                                              OAs Addressed Key Factors




Table 22: Centers for Medicare and Medicaid Services: Analysis of Extent to Which Beneficiary e-Services’s Fiscal Year 2011
OA Addressed OMB Key Factors

                                                                                  Yes,
                                                                                  no, or
Key factor                                                                        partial Summary of partial rating
1. Assesses current costs against life-cycle cost                                 N
2. Includes a structured schedule assessment                                      N
3. Includes a structured assessment of performance goals                          Y
4. Identifies whether the investment supports customer processes as               Y
designed and is delivering the goods and services it was designed to
deliver
5. Measures the effect the investment has on the performing                       P      Centers for Medicare and Medicaid Services
organization itself                                                                      determined that it improved population health by
                                                                                         providing Medicare information via public websites
                                                                                         and providing customer service channels for
                                                                                         beneficiaries to manage their health; it did not
                                                                                         analyze or identify the effect the investment had on
                                                                                         the department and its mission.
6. Includes a measure of how well the investment contributes to                   N
achieving the organization’s business needs and strategic goals
7. Compares current performance with a pre-established cost baseline N
and estimates
8. Identifies any areas for innovation in the areas of customer                   Y
satisfaction, strategic and business results, and financial performance
9. Identifies if the agency revisited alternative methods for achieving           N
the same mission needs and strategic goals
10. Addresses issues such as greater utilization of technology or                 Y
consolidation of investments to better meet organizational goals
11. Includes an ongoing review of the status of the risks identified in           N
the investment’s planning and acquisition phases
12. Identifies a need to redesign, modify, or terminate the investment            N
13. Includes an analysis on the need for improved methodology (i.e.,              N
better ways for the investment to meet cost and performance goals)
14. Identifies any lessons learned                                                N
15. Identifies if the investment had a cost or schedule variance                  N
16. Identifies recommendations to redesign or modify an asset before              N
it becomes a problem
17. Includes information on the overlap of the investment with other              N
systems

                                              Key:        Y= fully addressed the key factor
                                                          P= partially addressed the key factor
                                                          N= did not address the key factor
                                              Source: GAO analysis of HHS data.




                                              Page 41                                                        GAO-13-87 Information Technology
                                              Appendix II: Extent to Which DHS’s and HHS’s
                                              OAs Addressed Key Factors




Table 23: Centers for Medicare and Medicaid Services: Analysis of Extent to Which Health Care Quality Improvement and
Evaluation System’s Fiscal Year 2011 OA Addressed OMB Key Factors

                                                                                  Yes,
                                                                                  no, or
Key factor                                                                        partial Summary of partial rating
1. Assesses current costs against life-cycle cost                                 N
2. Includes a structured schedule assessment                                      N
3. Includes a structured assessment of performance goals                          N
4. Identifies whether the investment supports customer processes as               Y
designed and is delivering the goods and services it was designed to
deliver
5. Measures the effect the investment has on the performing                       N
organization itself
6. Includes a measure of how well the investment contributes to                   N
achieving the organization’s business needs and strategic goals
7. Compares current performance with a pre-established cost baseline N
and estimates
8. Identifies any areas for innovation in the areas of customer                   Y
satisfaction, strategic and business results, and financial performance
9. Identifies if the agency revisited alternative methods for achieving           N
the same mission needs and strategic goals
10. Addresses issues such as greater utilization of technology or                 Y
consolidation of investments to better meet organizational goals
11. Includes an ongoing review of the status of the risks identified in           N
the investment’s planning and acquisition phases
12. Identifies a need to redesign, modify, or terminate the investment            Y
13. Includes an analysis on the need for improved methodology (i.e.,              N
better ways for the investment to meet cost and performance goals)
14. Identifies any lessons learned                                                N
15. Identifies if the investment had a cost or schedule variance                  N
16. Identifies recommendations to redesign or modify an asset before              N
it becomes a problem
17. Includes information on the overlap of the investment with other              N
systems

                                              Key:        Y= fully addressed the key factor
                                                          P= partially addressed the key factor
                                                          N= did not address the key factor
                                              Source: GAO analysis of HHS data.




                                              Page 42                                                      GAO-13-87 Information Technology
                                              Appendix II: Extent to Which DHS’s and HHS’s
                                              OAs Addressed Key Factors




Table 24: Health Resources and Services Administration: Analysis of Extent to Which Electronic Handbooks Program
Management Office’s Fiscal Year 2011 OA Addressed OMB Key Factors

                                                                                  Yes,
                                                                                  no, or
Key factor                                                                        partial Summary of partial rating
1. Assesses current costs against life-cycle cost                                 N
2. Includes a structured schedule assessment                                      P      Health Resources and Services Administration
                                                                                         identified standard and unscheduled maintenance
                                                                                         efforts performed as scheduled, but it did not
                                                                                         comparatively analyze these efforts against
                                                                                         completion dates and goals.
3. Includes a structured assessment of performance goals                          Y
4. Identifies whether the investment supports customer processes as               Y
designed and is delivering the goods and services it was designed to
deliver
5. Measures the effect the investment has on the performing                       Y
organization itself
6. Includes a measure of how well the investment contributes to                   Y
achieving the organization’s business needs and strategic goals
7. Compares current performance with a pre-established cost baseline Y
and estimates
8. Identifies any areas for innovation in the areas of customer                   Y
satisfaction, strategic and business results, and financial performance
9. Identifies if the agency revisited alternative methods for achieving           Y
the same mission needs and strategic goals
10. Addresses issues such as greater utilization of technology or                 Y
consolidation of investments to better meet organizational goals
11. Includes an ongoing review of the status of the risks identified in           Y
the investment’s planning and acquisition phases
12. Identifies a need to redesign, modify, or terminate the investment            Y
13. Includes an analysis on the need for improved methodology (i.e.,              Y
better ways for the investment to meet cost and performance goals)
14. Identifies any lessons learned                                                Y
15. Identifies if the investment had a cost or schedule variance                  Y
16. Identifies recommendations to redesign or modify an asset before              Y
it becomes a problem
17. Includes information on the overlap of the investment with other              Y
systems

                                              Key:        Y= fully addressed the key factor
                                                          P= partially addressed the key factor
                                                          N= did not address the key factor
                                              Source: GAO analysis of HHS data.




                                              Page 43                                                      GAO-13-87 Information Technology
                                              Appendix II: Extent to Which DHS’s and HHS’s
                                              OAs Addressed Key Factors




Table 25: Health Resources and Services Administration: Analysis of Extent to Which National Practitioner Data Bank’s Fiscal
Year 2011 OA Addressed OMB Key Factors

                                                                             Yes,
                                                                             no, or
Key factor                                                                   partial Summary of partial rating
1. Assesses current costs against life-cycle cost                            Y
2. Includes a structured schedule assessment                                 P        Health Resources and Services Administration
                                                                                      identified schedule assessment results that reflect a
                                                                                      “green” score for the last four quarters. It also
                                                                                      identifies that earned value management is reporting
                                                                                      no discrepancies in cost and schedule. However, the
                                                                                      department did not perform a comparative analysis of
                                                                                      the planned and actual schedule milestones.
3. Includes a structured assessment of performance goals                     Y
4. Identifies whether the investment supports customer processes as          Y
designed and is delivering the goods and services it was designed to
deliver
5. Measures the effect the investment has on the performing                  Y
organization itself
6. Includes a measure of how well the investment contributes to              Y
achieving the organization’s business needs and strategic goals
7. Compares current performance with a pre-established cost baseline Y
and estimates
8. Identifies any areas for innovation in the areas of customer              Y
satisfaction, strategic and business results, and financial performance
9. Identifies if the agency revisited alternative methods for achieving      Y
the same mission needs and strategic goals
10. Addresses issues such as greater utilization of technology or            Y
consolidation of investments to better meet organizational goals
11. Includes an ongoing review of the status of the risks identified in      P        Health Resources and Services Administration
the investment’s planning and acquisition phases                                      identified the investment’s cost, schedule, and
                                                                                      performance outcomes; but did not address the
                                                                                      status of risks, and their potential impacts were not
                                                                                      identified.
12. Identifies a need to redesign, modify, or terminate the investment       Y
13. Includes an analysis on the need for improved methodology (i.e.,         Y
better ways for the investment to meet cost and performance goals)
14. Identifies any lessons learned                                           Y
15. Identifies if the investment had a cost or schedule variance             Y
16. Identifies recommendations to redesign or modify an asset before         Y
it becomes a problem
17. Includes information on the overlap of the investment with other         N
systems

                                              Key:      Y= fully addressed the key factor
                                                        P= partially addressed the key factor




                                              Page 44                                                     GAO-13-87 Information Technology
                                              Appendix II: Extent to Which DHS’s and HHS’s
                                              OAs Addressed Key Factors




                                                          N= did not address the key factor
                                              Source: GAO analysis of HHS data.




Table 26: Indian Health Service: Analysis of Extent to Which Infrastructure, Office Automation, and Telecommunications’s
Fiscal Year 2011 OA Addressed OMB Key Factors

                                                                                  Yes,
                                                                                  no, or
Key factor                                                                        partial Summary of partial rating
1. Assesses current costs against life-cycle cost                                 Y
2. Includes a structured schedule assessment                                      Y
3. Includes a structured assessment of performance goals                          Y
4. Identifies whether the investment supports customer processes as               Y
designed and is delivering the goods and services it was designed to
deliver
5. Measures the effect the investment has on the performing                       Y
organization itself
6. Includes a measure of how well the investment contributes to                   Y
achieving the organization’s business needs and strategic goals
7. Compares current performance with a pre-established cost baseline Y
and estimates
8. Identifies any areas for innovation in the areas of customer                   Y
satisfaction, strategic and business results, and financial performance
9. Identifies if the agency revisited alternative methods for achieving           Y
the same mission needs and strategic goals
10. Addresses issues such as greater utilization of technology or                 Y
consolidation of investments to better meet organizational goals
11. Includes an ongoing review of the status of the risks identified in           P      Indian Health Service analyzed cost and schedule
the investment’s planning and acquisition phases                                         progress, but it did not include an assessment of the
                                                                                         risks and potential impacts.
12. Identifies a need to redesign, modify, or terminate the investment            Y
13. Includes an analysis on the need for improved methodology (i.e.,              Y
better ways for the investment to meet cost and performance goals)
14. Identifies any lessons learned                                                N
15. Identifies if the investment had a cost or schedule variance                  Y
16. Identifies recommendations to redesign or modify an asset before              Y
it becomes a problem
17. Includes information on the overlap of the investment with other              N
systems

                                              Key:        Y= fully addressed the key factor
                                                          P= partially addressed the key factor
                                                          N= did not address the key factor
                                              Source: GAO analysis of HHS data.




                                              Page 45                                                        GAO-13-87 Information Technology
                                              Appendix II: Extent to Which DHS’s and HHS’s
                                              OAs Addressed Key Factors




Table 27: National Institutes of Health: Analysis of Extent to Which Business Intelligence System’s Fiscal Year 2011 OA
Addressed OMB Key Factors

                                                                                  Yes,
                                                                                  no, or
Key factor                                                                        partial Summary of partial rating
1. Assesses current costs against life-cycle cost                                 N
2. Includes a structured schedule assessment                                      N
3. Includes a structured assessment of performance goals                          Y
4. Identifies whether the investment supports customer processes as               Y
designed and is delivering the goods and services it was designed to
deliver
5. Measures the effect the investment has on the performing                       P       National Institutes of Health identified that the
organization itself                                                                       investment was in line with the appropriate
                                                                                          component’s enterprise architecture, but did not
                                                                                          identify the effect the investment had on other
                                                                                          aspects of the component, such as its mission and
                                                                                          business processes.
6. Includes a measure of how well the investment contributes to                   N
achieving the organization’s business needs and strategic goals
7. Compares current performance with a pre-established cost baseline N
and estimates
8. Identifies any areas for innovation in the areas of customer                   N
satisfaction, strategic and business results, and financial performance
9. Identifies if the agency revisited alternative methods for achieving           Y
the same mission needs and strategic goals
10. Addresses issues such as greater utilization of technology or                 N
consolidation of investments to better meet organizational goals
11. Includes an ongoing review of the status of the risks identified in           Y
the investment’s planning and acquisition phases
12. Identifies a need to redesign, modify, or terminate the investment            N
13. Includes an analysis on the need for improved methodology (i.e.,              Y
better ways for the investment to meet cost and performance goals)
14. Identifies any lessons learned                                                N
15. Identifies if the investment had a cost or schedule variance                  Y
16. Identifies recommendations to redesign or modify an asset before              N
it becomes a problem
17. Includes information on the overlap of the investment with other              N
systems

                                              Key:        Y= fully addressed the key factor
                                                          P= partially addressed the key factor
                                                          N= did not address the key factor
                                              Source: GAO analysis of HHS data.




                                              Page 46                                                       GAO-13-87 Information Technology
Appendix III: Comments from the
              Appendix III: Comments from the Department
              of Defense



Department of Defense




              Page 47                                      GAO-13-87 Information Technology
Appendix III: Comments from the Department
of Defense




Page 48                                      GAO-13-87 Information Technology
Appendix IV: Comments from the
             Appendix IV: Comments from the Department
             of Homeland Security



Department of Homeland Security




             Page 49                                     GAO-13-87 Information Technology
Appendix IV: Comments from the Department
of Homeland Security




Page 50                                     GAO-13-87 Information Technology
Appendix V: Comments from the Department
             Appendix V: Comments from the Department
             of the Treasury



of the Treasury




             Page 51                                    GAO-13-87 Information Technology
Appendix VI: Comments from the
             Appendix VI: Comments from the Department
             of Veterans Affairs



Department of Veterans Affairs




             Page 52                                     GAO-13-87 Information Technology
Appendix VI: Comments from the Department
of Veterans Affairs




Page 53                                     GAO-13-87 Information Technology
Appendix VII: GAO Contact and Staff
                  Appendix VII: GAO Contact and Staff
                  Acknowledgments



Acknowledgments

                  David A. Powner, (202) 512-9286 or pownerd@gao.gov
GAO Contact
                  In addition to the contact name above, individuals making contributions to
Staff             this report included Gary Mountjoy (Assistant Director), Gerard Aflague,
Acknowledgments   Rebecca Eyler, Lori Martinez, and Teresa Smith.




(311261)
                  Page 54                                       GAO-13-87 Information Technology
GAO’s Mission         The Government Accountability Office, the audit, evaluation, and
                      investigative arm of Congress, exists to support Congress in meeting its
                      constitutional responsibilities and to help improve the performance and
                      accountability of the federal government for the American people. GAO
                      examines the use of public funds; evaluates federal programs and
                      policies; and provides analyses, recommendations, and other assistance
                      to help Congress make informed oversight, policy, and funding decisions.
                      GAO’s commitment to good government is reflected in its core values of
                      accountability, integrity, and reliability.

                      The fastest and easiest way to obtain copies of GAO documents at no
Obtaining Copies of   cost is through GAO’s website (http://www.gao.gov). Each weekday
GAO Reports and       afternoon, GAO posts on its website newly released reports, testimony,
                      and correspondence. To have GAO e-mail you a list of newly posted
Testimony             products, go to http://www.gao.gov and select “E-mail Updates.”

Order by Phone        The price of each GAO publication reflects GAO’s actual cost of
                      production and distribution and depends on the number of pages in the
                      publication and whether the publication is printed in color or black and
                      white. Pricing and ordering information is posted on GAO’s website,
                      http://www.gao.gov/ordering.htm.
                      Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
                      TDD (202) 512-2537.
                      Orders may be paid for using American Express, Discover Card,
                      MasterCard, Visa, check, or money order. Call for additional information.
                      Connect with GAO on Facebook, Flickr, Twitter, and YouTube.
Connect with GAO      Subscribe to our RSS Feeds or E-mail Updates. Listen to our Podcasts.
                      Visit GAO on the web at www.gao.gov.
                      Contact:
To Report Fraud,
Waste, and Abuse in   Website: http://www.gao.gov/fraudnet/fraudnet.htm
                      E-mail: fraudnet@gao.gov
Federal Programs      Automated answering system: (800) 424-5454 or (202) 512-7470

                      Katherine Siggerud, Managing Director, siggerudk@gao.gov, (202) 512-
Congressional         4400, U.S. Government Accountability Office, 441 G Street NW, Room
Relations             7125, Washington, DC 20548

                      Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800
Public Affairs        U.S. Government Accountability Office, 441 G Street NW, Room 7149
                      Washington, DC 20548




                        Please Print on Recycled Paper.