oversight

IRS Management: Cost Estimate for New Information Reporting System Needs to be Made More Reliable

Published by the Government Accountability Office on 2012-01-31.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

               United States Government Accountability Office

GAO            Report to Congressional Committees




January 2012
               IRS MANAGEMENT

               Cost Estimate for New
               Information Reporting
               System Needs to be
               Made More Reliable




GAO-12-59
                                               January 2012

                                               IRS MANAGEMENT
                                               Cost Estimate for New Information Reporting
                                               System Needs to be Made More Reliable
Highlights of GAO-12-59, a report to
congressional committees




Why GAO Did This Study                         What GAO Found
The Internal Revenue Service
(IRS) began developing the                     The 2011 Information Reporting and Document Matching (IRDM) cost estimate,
Information Reporting and Document             used to justify the program’s projected budgets of $115 million for fiscal years
Matching (IRDM) program in fiscal year         2012 through 2016, generally does not meet best practices for reliability. As
2009 to enhance IRS’s ability to               shown in the table below, the cost estimate did not fully meet any of the four best
automatically compare different                practices for a reliable cost estimate.
sources of tax information and thus
                                               Best Practices for a Reliable Cost Estimate and IRDM Assessment
improve its capacity to identify and
address taxpayer noncompliance.                 Best practices                                           IRDM rating
                                                Comprehensive: the estimate should cover the
                                                entire program over its full life-cycle.                 Partially meets.
GAO’s May 2011 report recommended               Well documented: the estimate should be
that IRS follow best practices from the         supported by detailed documentation.                     Minimally meets.
GAO’s Cost Estimating and                       Accurate: the estimate should provide unbiased
Assessment Guide if IRS updated the             results that are not overly conservative or
cost estimate for building IRDM                 optimistic.                                              Minimally meets.
                                                Credible: the estimate should check for and
systems. IRS provided a new cost                discuss any limitations.                                 Does not meet.
estimate for IRDM in August 2011. In
                                               Source: GAO analysis of IRS’s 2011 IRDM cost estimate.
this report, GAO assessed the extent
to which (1) the IRDM funding request          For example, the cost estimate minimally meets best practices for a well
is supported by a reliable cost estimate       documented estimate because the Internal Revenue Service (IRS) did not
and, if not reliably supported, why not;       provide detailed support for staff resources, and the cost estimate documentation
and (2) IRS’s practices for capturing          only justified about 6 out of the 86 requested full time equivalent staff for IRDM,
data on IRDM’s actual costs and                among other things. If documentation does not provide source data or cannot
comparing them to estimated costs—             explain the calculations underlying the cost elements, the estimate’s credibility
known as earned value management               may suffer. Although IRS has an independent office of cost estimators that can
(EVM)—generate reliable performance            develop and update cost estimates using cost modeling software that generally
data. GAO compared IRS’s 2011                  follows GAO’s best practices, this office did not develop the 2011 IRDM cost
IRDM cost estimate to criteria in GAO’s        estimate. IRS policy does not require project teams to work with the office to
cost guide and analyzed IRDM’s                 update cost estimates. Additionally, IRS’s cost estimation guidance for project
earned value management data.                  managers is inconsistent regarding how cost estimates should be related to a
What GAO Recommends                            budget, an inconsistency that could lead to budget requests that do not
                                               accurately estimate program funding needs.
GAO recommends that IRS ensure
that IRDM has a reliable cost estimate,        The IRDM program’s earned value management (EVM) data did not meet data
require certain project teams to work          reliability criteria in the areas GAO reviewed. For example, the IRDM project
with its Estimation Program Office,            schedule was not properly sequenced—meaning activities were not properly
improve cost estimation guidance, and          linked in the order in which they are to be carried out. In addition, surveillance
improve the reliability of IRDM’s EVM          was not conducted on IRDM’s EVM system, as required by the Office of
data. IRS agreed with one, partially           Management and Budget and the Department of the Treasury. Surveillance
agreed with one, and disagreed with            involves having qualified staff review an EVM system. Because IRDM’s 2011
two of GAO’s recommendations. GAO              cost estimate is based on unreliable EVM data, it does not provide adequate
generally disagrees with IRS’s                 support for IRDM’s budget requests. Until IRS addresses deficiencies in the EVM
concerns, and still believes the               data, it cannot provide a reliable cost estimate for IRDM.
recommendations have merit.



View GAO-12-59. For more information,
contact Michael Brostek at (202) 512-9110 or
brostekm@gao.gov.

                                                                                                        United States Government Accountability Office
Contents


Letter                                                                                 1
               Background                                                              3
               IRDM’s 2011 Cost Estimate Does Not Meet Best Practices for
                 Reliability and Does Not Fully Support the Program’s Budget           9
               Unreliable EVM Data Raise Additional Concerns about IRDM Cost
                 Estimate                                                             23
               Conclusions                                                            27
               Recommendations for Executive Action                                   28
               Agency Comments and Our Evaluation                                     29

Appendix I     Scope and Methodology                                                  32



Appendix II    Assessment of MITS’s Current IRDM Cost Estimate                        35



Appendix III   High Level Assessment of the Reliability of IRDM’s EVM System          41



Appendix IV    Comments from the Internal Revenue Service                             42



Appendix V     GAO Contact and Staff Acknowledgments                                  44




Tables
               Table 1: Comparison of IRDM 2007 Preliminary Estimate and the
                        2009 SCBE (dollars, in millions)                               7
               Table 2: IRDM Program Budget and Actual Amount Spent, Fiscal
                        Years 2009 through 2011 (dollars, in millions)                 8
               Table 3: MITS IRDM Cost Estimate Alignment with Best Practices         35
               Table 4: IRDM EVM Data Reliability                                     41




               Page i                                            GAO-12-59 IRS Management
Figures
          Figure 1: 2011 IRDM Cost Estimate Alignment with Best Practices
                   for Comprehensiveness                                                            11
          Figure 2: 2011 IRDM Cost Estimate Alignment with Best Practices
                   for Being Well Documented                                                        13
          Figure 3: 2011 IRDM Cost Estimate Alignment with Best Practices
                   for Accuracy                                                                     15
          Figure 4: 2011 IRDM Cost Estimate Alignment with Best Practices
                   for Credibility                                                                  17




          Abbreviations

          ANSI              American National Standards Institute
          DME               Development/modernization/enhancement
          EPO               Estimation Program Office
          EVM               Earned Value Management
          FTE               Full-time equivalent
          IRDM              Information Reporting and Document Matching
          IRS               Internal Revenue Service
          IT                Information technology
          MITS              Modernization and Information Technology Services
          SCBE              Solution concept-based estimate
          WBS               Work Breakdown Structure




          This is a work of the U.S. government and is not subject to copyright protection in the
          United States. The published product may be reproduced and distributed in its entirety
          without further permission from GAO. However, because this work may contain
          copyrighted images or other material, permission from the copyright holder may be
          necessary if you wish to reproduce this material separately.




          Page ii                                                        GAO-12-59 IRS Management
United States Government Accountability Office
Washington, DC 20548




                                   January 31, 2012

                                   The Honorable Susan M. Collins
                                   Ranking Member
                                   Committee on Homeland Security and Government Affairs
                                   United States Senate

                                   The Honorable Darrell E. Issa
                                   Chairman
                                   The Honorable Elijah E. Cummings
                                   Ranking Member
                                   Committee on Oversight and Government Reform
                                   United States House of Representatives

                                   The financing of the federal government depends largely on the Internal
                                   Revenue Service’s (IRS) efforts to collect taxes. To help carry out this
                                   work, IRS initiated the Information Reporting and Document Matching
                                   (IRDM) program in fiscal year 2009. IRS plans for the IRDM program to
                                   build information technology (IT) systems that automatically compare—
                                   that is, match—different sources of tax information to improve tax
                                   compliance. 1 In May 2011, we issued a report that assessed IRS’s 2009
                                   cost estimate for building IRDM IT systems, and found that it did not fully
                                   meet best practices. 2 We recommended that for any future updates to the
                                   IRDM cost estimate, IRS ensure that the revised estimate be developed
                                   in a manner that reflects the four characteristics of a reliable cost estimate




                                   1
                                    IRDM systems will automatically match income or expense information reported by third
                                   parties to income or expenses reported on a taxpayer’s tax returns. For example, for
                                   certain securities sales, brokers must report to the IRS cost basis information (generally,
                                   the difference between the gross proceeds from the securities sale and the original
                                   purchase price, net any fees or commissions) on the information return, Form 1099-B,
                                   “Proceeds from Broker and Barter Exchange Transaction.” IRDM systems will then
                                   compare this information to an individual’s tax return data on securities sales, such as
                                   found on the Form 1040 Schedule D, “Capital Gains and Losses,” enabling it to identify
                                   potential compliance problems.
                                   2
                                    GAO, Information Reporting: IRS Could Improve Cost Basis and Transaction Settlement
                                   Reporting Implementation, GAO-11-557 (Washington, D.C.: May 19, 2011).




                                   Page 1                                                          GAO-12-59 IRS Management
described in our report. 3 Since then, IRS provided a new cost estimate for
IRDM in August 2011.

Having a reliable cost estimate—a summation of individual cost elements
using established methods and valid data—is vital for making informed
budgetary decisions and ensuring that a project is implemented as
planned. In an environment where federal funds are scarce, it is
imperative that IT projects, such as IRDM, are implemented as planned,
not only because of their value to the government, 4 but also because
every dollar spent on one program will mean one less to fund other
efforts. Our objectives were to determine the extent to which: (1) the
IRDM funding request is supported by a reliable cost estimate and, if not
reliably supported, why not; and (2) IRS’s practices for capturing IRDM’s
actual costs and comparing them to estimated costs—known as “earned
value management” (EVM)—generate reliable performance data. We are
making four recommendations that warrant management’s consideration.

This report builds on our May 2011 report 5 and further analysis conducted
from June 2011 to August 2011, which was used to provide technical
assistance to Congress. For this report, we compared the current 2011
IRDM cost estimate to the best practices in our Cost Estimating and
Assessment Guide. 6 We also compared IRS guidance on cost estimation
to criteria from our cost guide. We shared our cost guide as well as our
preliminary analysis of the IRDM 2011 cost estimate with program
officials. When warranted, we updated our analyses based on the
agency’s response and additional documentation provided to us. To
assess IRDM’s practices for capturing actual costs and comparing them
to estimated costs, we used high level criteria on EVM data reliability from
our cost guide; EVM is a project management approach that uses actual


3
 A reliable cost estimate is (1) comprehensive, (2) well documented, (3) accurate, and (4)
credible. See GAO, GAO Cost Estimating and Assessment Guide: Best Practices for
Developing and Managing Capital Program Costs, GAO-09-3SP (Washington, D.C.:
March 2009), for a full description of each of the four characteristics.
4
 The IRDM system will increase information that is available for compliance purposes and
potentially raise significant revenue. Two components of the IRDM program, cost basis
and transaction settlement reporting, are expected to generate more than $16 billion over
the next 10 years, according to the Joint Committee on Taxation in 2008.
5
    GAO-11-557.
6
See GAO, Cost Estimating and Assessment Guide: Best Practices for Developing and
Managing Capital Program Costs, GAO-09-3SP (Washington, D.C.: Mar. 9, 2009).




Page 2                                                         GAO-12-59 IRS Management
               project data to provide reports on project status. We also compared
               IRDM’s EVM system with guidance in the Department of the Treasury’s
               Earned Value Management Guide. As part of that analysis, we assessed
               IRDM’s implementation of three guidelines from the American National
               Standards Institute (ANSI), which Treasury’s guidance states that IRDM
               should be following. We selected the three guidelines to represent some
               of the fundamental steps for maintaining a reliable EVM system, as
               identified in our cost guide, and because these guidelines are also
               Treasury Department requirements. For both research objectives, we
               interviewed officials with IRS’s Modernization and Information Technology
               Services (MITS) division, including those from the IRDM Program
               Management Office, and the Investment Planning and Management
               Office, which includes the Estimation Program Office (EPO). 7

               We conducted this performance audit from August 2011 through January
               2012 in accordance with generally accepted government auditing
               standards. Those standards require that we plan and perform the audit to
               obtain sufficient, appropriate evidence to provide a reasonable basis for
               our findings and conclusions based on our audit objectives. We believe
               that the evidence obtained provides a reasonable basis for our findings
               and conclusions based on our audit objectives. We determined that the
               IRS data we used were sufficiently reliable for our purposes. We also
               made appropriate attribution indicating the sources of the data. We are
               making recommendations to IRS to improve data reliability in the future.
               See appendix I for more information on our scope and methodology.



Background
IRDM Program   IRS initiated the IRDM program, in part, to implement new information
               reporting requirements, but more generally to increase voluntary
               compliance with tax laws by expanding and maximizing IRS’s ability to
               match existing and future information returns with tax return data and
               establishing a new business information matching program. Previously,
               IRS had only matched information returns to individuals’ and sole
               proprietors’ tax returns. 8 Under IRDM, IRS plans to build several new IT


               7
                   EPO is IRS’s cost estimation organization.
               8
                A sole proprietor is an individual who owns an unincorporated business by himself or
               herself.




               Page 3                                                        GAO-12-59 IRS Management
                       systems and enhance some existing systems as well as implement
                       numerous organizational and process changes. IRS plans for IRDM to
                       use information returns to identify individual and business tax returns that
                       are likely sources of revenue, which the current individual tax return
                       matching system is not designed to identify. IRDM implementation is led
                       by IRS’s Small Business/Self Employed division and MITS, which is
                       leading the IRDM IT system development. 9


IRS Cost Estimation    Cost estimates are a vital factor for sound management decision making
Process and Guidance   and they aid in the formation of a project’s budget. IRS uses cost
                       estimates, in part, to justify budget requests and prioritize the selection of
                       IT projects for possible funding. After an IT project is approved, the cost
                       estimate is later used as a starting point for developing the performance
                       measurement baseline for EVM, a project management approach that, if
                       implemented appropriately, provides management important tools such
                       as objective reports of project status and early warning signs of
                       impending schedule delays and cost overruns. Data from a reliable
                       performance management system, such as EVM, are necessary inputs
                       for an updated cost estimate, among other things.

                       OMB issued guidance on managing IT projects, which discusses cost
                       estimation and refers to our cost guide for how to meet cost estimating
                       requirements. 10 OMB guidelines state that cost estimates should be
                       continuously updated based on the latest information available to ensure
                       that they are current, accurate, and valid. According to our cost guide,
                       effective program and cost control requires ongoing revisions to the cost
                       estimate, budget, and projected estimates at completion. Specifically, our
                       guide states that estimates should be continuously updated with actual
                       costs incurred to that point so that significant cost, schedule, or
                       performance variances can be examined. In addition, it says that cost
                       estimates should be updated to reflect significant changes to a project’s




                       9
                       For more information on the IRDM program, including IT system development plans, see
                       GAO-11-557.
                       10
                        Office of Management and Budget, Capital Programming Guide: Supplement to Circular
                       A-11, Planning, Budgeting, and Acquisition of Capital Assets, (Executive Office of the
                       President, Washington, D.C.: August 2011).




                       Page 4                                                      GAO-12-59 IRS Management
scope or specifications and when certain projects approach key
milestones. 11

Within MITS, project managers and EPO are involved in estimating
program costs. EPO is an independent group of cost estimation experts
that assists project teams by developing and updating cost estimates
using a standard documented process. Project managers are responsible
for maintaining a program’s cost estimate. EPO only becomes involved in
updating a program’s cost estimate at the request of project managers,
according to EPO officials.

IRS procedures for developing, using, and updating cost estimates and
EVM are described in several guidance documents, specifically:

•      EPO’s Estimator’s Reference Guide, which is used by EPO staff, is
       the general resource on the processes and procedures for developing
       and delivering IT cost estimates. It discusses the technical aspects of
       updating cost estimates, such as what documents are used in cost
       modeling once a project has begun.
•      IRS’s Information Technology Investment Planning and Management
       Guide (Investment Guide) outlines the framework for selecting,
       managing, and evaluating IRS IT projects. The Investment Guide
       includes discussions of how IT projects are selected using cost
       information, and how managers should use cost information to
       monitor a project. Project managers are responsible for managing
       cost, schedule, and performance for a project.
•      MITS’s Estimation Procedures document describes IRS’s
       organizational approach to cost estimation, applicable to all IRS
       projects. The document is directed at project managers and it includes
       discussions of the steps and staff roles necessary to develop an
       estimate, and the circumstances when EPO typically becomes
       involved with updating a cost estimate. The document states that
       project managers are responsible for monitoring project progress and
       suggests initiating assistance from EPO if a project meets certain
       thresholds.
•      The Department of the Treasury’s Earned Value Management Guide
       provides guidance for implementing EVM on major IRS projects.
       Program or project managers have the ultimate responsibility for



11
     GAO-09-3SP.




Page 5                                                 GAO-12-59 IRS Management
                           implementing and monitoring the EVM system for their program or
                           project.

IRDM Program Cost   IRS developed two cost estimates for IRDM early in the program. In 2007,
Estimates           IRS developed a preliminary cost estimate for budgetary purposes when
                    very little program information was available (referred to in this report as
                    the 2007 preliminary cost estimate). 12 As shown in table 1, in 2007, IRDM
                    system development was estimated to cost about $5 million in fiscal year
                    2009 and about $23 million per year thereafter. 13 In 2009, EPO developed
                    a solution concept-based estimate (SCBE, referred to in this report as the
                    2009 SCBE), 14 which was more rigorous than the 2007 preliminary
                    estimate. The 2009 SCBE was developed before program implementation
                    began, when MITS had more information than it did in 2007, but system
                    design plans were still under development. 15 The 2009 SCBE was about
                    $36 million less through the first 4 years of the project than the 2007
                    preliminary estimate.




                    12
                      In our May report, GAO-11-557, based on information from MITS officials, we noted an
                    initial estimate developed in 2008 for IRDM’s initial budget request. We obtained
                    additional information on IRDM cost estimates indicating that the initial estimate was
                    provided for the budget request in 2007, and subsequently confirmed this information with
                    MITS officials.
                    13
                         IRS plans for IRDM system development funding to continue through FY 2016.
                    14
                       A solution concept based estimate relies on a document, referred to as a solution
                    concept, which explains a project proposal’s functional scope and technical solution.
                    15
                      The 2009 IRDM SCBE did not follow MITS’s standard cost estimation process because
                    of time constraints. For this estimate, MITS heavily modified its estimation techniques and
                    used broad assumptions.




                    Page 6                                                         GAO-12-59 IRS Management
Table 1: Comparison of IRDM 2007 Preliminary Estimate and the 2009 SCBE (dollars, in millions)

                                                                 Fiscal year                                                        Total, FY
                                                                                                                                        2009
                                                                                                                                  through FY
IRDM Estimate       2009       2010      2011           2012               2013                 2014        2015          2016          2012
2007
preliminary
         a
estimate             $5.1     $23.0     $22.9          $23.2              $23.0                $23.0       $23.0         $23.0         $74.2
              b
2009 SCBE                                                                  Not                 Not            Not           Not
                     $2.6     $13.4     $13.3            $9.3        estimated           estimated      estimated     estimated        $38.6
Difference
between the
2007
preliminary
estimate and
SCBE                 $2.5      $9.6         $9.6       $13.9                 N/A                 N/A         N/A           N/A         $35.6
                                        Source: GAO analysis of IRDM’s Exhibit 300 and the 2009 SCBE.
                                        a
                                         Annual costs based on the preliminary estimate are reported in the Exhibit 300. The Exhibit 300 is a
                                        document required by OMB to support IT projects. It includes the project’s desired outcome and
                                        budget justification. The 2007 preliminary estimate provided information from fiscal year 2009 through
                                        fiscal year 2016.
                                        b
                                         The 2009 SCBE only went through fiscal year 2012.


                                        IRS used the 2007 preliminary estimate to justify the initial IRDM budget.
                                        According to a MITS official, the 2009 SCBE was not used for budgetary
                                        purposes because program specifications were undergoing modification
                                        in fiscal year 2010, requiring the full $23-million per-year funding. For
                                        example, in 2010, IRS made several changes to the complexity of the
                                        IRDM program, which included dividing it into four projects. One of the
                                        projects required restructuring as a separate development effort, using
                                        different resources and technologies. Changes also necessitated using
                                        new software development methods not previously used within IRS, and
                                        additional contracting support. According to MITS officials, such changes
                                        increased IRDM funding needs above the amounts supported by the
                                        2009 SCBE. As a result, IRS did not revise its initial funding request for
                                        future fiscal years using the SCBE and instead relied on the 2007
                                        preliminary estimate.

                                        From fiscal years 2009 through 2011, IRS received about $52 million in
                                        total funding for IRDM, of which the IRS had spent $46 million (when
                                        accounting for $2.6 million carried over from fiscal year 2009 funds and
                                        $5.8 million carried over from fiscal year 2010), through fiscal year 2011,
                                        as shown in table 2.




                                        Page 7                                                                      GAO-12-59 IRS Management
Table 2: IRDM Program Budget and Actual Amount Spent, Fiscal Years 2009
through 2011 (dollars, in millions)

                                                                      Fiscal year                           Total FY 2009
                                                         2009                  2010              2011       through 2011
    Budget                                                $5.6                $22.9             $23.0              $51.5
    Actual amount spent from budget                       $3.0                $15.8             $18.8              $37.6
    Difference between budget and
                 a
    amount spent                                          $2.6                  $7.1              $4.2             $13.9
Source: IRS officials, IRDM budget documents, and actual spending data from IRS’s financial information system.
a
 The actual amount spent does not include money from the previous fiscal year that was carried over
and spent. The amounts are not cumulative. In FY 2010, IRDM spent $2.6 million in funds carried
over from FY 2009, and in FY 2011 IRDM spent $5.8 million of funds carried over from FY 2010.

In our May 2011 report, 16 we assessed the 2009 SCBE because it was
the most rigorous IRDM cost estimate available at the time and the 2007
preliminary estimate lacked documentation for a complete review. We
found that the 2009 SCBE did not fully follow best practices. We
recommended that if IRS updated the cost estimate, it should follow best
practices from our cost guide. In response to our report, IRS said it would
update the 2009 SCBE. IRS subsequently decided not to revise the
estimate because, according to officials, they already have a plan,
schedule and funding, the program is not over-budget, and the risks
associated with IRDM, and the program’s size, do not warrant an update.

Over the summer of 2011, MITS provided us with additional cost
information. Officials referred to these documents as IRDM’s new cost
estimate, and they were used in supporting IRS’s fiscal year 2012 IRDM
budget. Consequently, in this report, we refer to the materials provided to
us as the 2011 cost estimate. This estimate was not a SCBE or
developed by EPO. The 2011 cost estimate was based on several data
sources, including IRDM’s Exhibit 300, EVM data, spend plans, and
schedule with work breakdown structures (WBS). 17 MITS projects IRDM


16
    GAO-11-557.
17
   The Exhibit 300 is a document required by OMB to support IT project budgets. It
includes the project’s desired outcome and budget justification. A WBS shows the
requirements that must be accomplished to develop a program, and provides the basis for
identifying resources and tasks for developing a program cost estimate. It provides a basic
framework for estimating costs, developing schedules, identifying resources, determining
where risks may occur, and providing the means for measuring program status using
EVM.




Page 8                                                                                      GAO-12-59 IRS Management
                          to cost $115 million for fiscal years 2012 through 2016, or about $23
                          million per year.



IRDM’s 2011 Cost
Estimate Does Not
Meet Best Practices
for Reliability and
Does Not Fully
Support the Program’s
Budget

IRDM’s Fiscal Year 2012   According to best practices established by our cost guide, a cost estimate
and Projected Budget      should be comprehensive, well documented, accurate, and credible. 18 We
Requests Are Not          assessed the 2011 cost estimate against cost estimation best practices
                          because IRS told us the estimate was used to support its budget requests
Supported by a Reliable
                          for fiscal year 2012 and beyond. While the 2011 IRDM cost estimate
Cost Estimate             shows some characteristics of a reliable cost estimate, it does not fully
                          meet best practices.

                          The estimate partially meets best practices for a comprehensive cost
                          estimate, as shown in figure 1. 19 It reflects the current program schedule
                          and contains information about the program’s technical characteristics.
                          The estimate provided some details about costs for IRDM’s fiscal year
                          2012 budget request, but the cost estimate does not cover the program’s
                          entire life-cycle. 20 Without fully accounting for life-cycle costs,


                          18
                               GAO-09-3SP.
                          19
                            We rated the extent to which IRS met each best practice on the following scale: “Meets,”
                          IRS provided complete evidence that satisfies the entire criterion; “Substantially meets,”
                          IRS provided evidence that satisfies a large portion of the criterion; “Partially meets,” IRS
                          provided evidence that satisfies about half of the criterion; “Minimally meets,” IRS provided
                          evidence that satisfies a small portion of the criterion; and “Does not meet,” IRS provided
                          no evidence that satisfies any of the criterion. See app. I for our full scope and
                          methodology.
                          20
                               IRS plans for IRDM development to continue through fiscal year 2016.




                          Page 9                                                          GAO-12-59 IRS Management
management may have difficulty successfully planning program resource
requirements and making informed resource-planning decisions. IRS
defined assumptions used to estimate some IRDM costs, but did not
provide the assumptions used to estimate labor or program operations
costs. Furthermore, IRS did not include ground rules 21 used to develop
the estimate. Unless ground rules and assumptions are clearly defined,
the cost estimate will not have a basis to identify and mitigate areas of
potential risk.




21
 Ground rules are a set of estimating standards that provide guidance and minimize
conflicts in definitions, while assumptions are judgments about past, present, or future
conditions that may affect the estimate.




Page 10                                                         GAO-12-59 IRS Management
Interactive graphic          Figure 1: 2011 IRDM Cost Estimate Alignment with Best Practices for Comprehensiveness




                             Directions:
                             Rollover each              below to see further information on cost estimation best practices.


                                                                                                              Overall assessment:

                                                                                                              Partially meets best practices for a
A comprehensive cost estimate:                                                                                comprehensive cost estimate.


                                              Assessment of whether best
Best practices characteristics                practices are met                                      Effect

Includes all life-cycle costs.                The estimate includes costs through FY                 A life-cycle cost estimate should encompass all past,
                                              2014, but IRS plans for the project to con-            present, and future costs for every aspect of the program,
                                              tinue through FY 2016, and detailed costs              regardless of funding source, including all government and
                                              are only provided for FY 2012.                         contractor costs. Life-cycle cost estimates can enhance
                                              Minimally meets                                        program managers’ decision making by allowing them to
                                                                                                     evaluate design trade off studies on a total cost basis as
                                                                                                     well as on a technical and performance basis.

Completely defines the                        The estimate reflects the current project              Understanding the program— including the acquisition
program, reflects the                         schedule and contains high-level informa-              strategy, technical definition, characteristics, system
current schedule, and is                      tion about technical specifications but lacks          design features, and technologies to be included—is key
technically reasonable.                       details that would completely define the               to developing a credible cost estimate. Without these data,
                                              program.                                               the cost estimator will not be able to identify the technical
                                              Partially meets                                        and program parameters that will bind the cost estimate.

Has a product-oriented work                   Each of the four IRDM projects has a                   A WBS provides a necessary framework for the program to
breakdown structure (WBS),                    product-oriented WBS, but the WBSs are                 develop a schedule and cost plan that can easily track
traceable to the program’s                    not consistent and they are not traceable to           technical accomplishments. A standard, product-oriented
technical scope, at an                        the cost estimate.                                     WBS facilitates the tracking of resource allocations and
appropriate level of detail.                  Partially meets                                        expenditures, which can give the agency insight to reliably
                                                                                                     estimate the cost of future similar programs.

Documents all cost-influ-                     No ground rules were documented. Several               Unless ground rules and assumptions are clearly
encing ground rules and                       documents included assumptions, but none               documented, the cost estimate will not have a basis for
assumptions.                                  discussed associated risks.                            assessing potential risks. Furthermore, the estimate cannot
                                              Minimally meets                                        be reconstructed when the original estimators are no longer
                                                                                                     available.

                             Source: GAO analysis of IRS’s 2011 IRDM cost estimate and GAO-09-3SP.




    Print instructions       A full text version of this graphic is available in appendix II.

                                                      Page 11                                                                      GAO-12-59 IRS Management
The estimate minimally meets best practices for a well documented cost
estimate, as shown in figure 2. IRS provided supporting information for
some staff resources, but detailed data for the staffing level requested for
fiscal year 2012 was missing. The cost estimate documentation says that
the labor cost justification was captured in the resource loaded project
schedules, 22 but we found that these schedules only justified about 6 out
of the 86 requested full-time equivalent (FTE) staff for IRDM.
Furthermore, although IRS officials cited the WBS as the basis for cost
projections, we found no evidence linking the WBS to cost. Multiple
documents linked software and hardware specifications to cost, but they
did not provide consistent cost information. As a best practice,
documentation should describe the source data used, the estimating
methodology, and show step-by-step how the estimate was developed.
Without a well documented cost estimate, the program’s credibility may
suffer because the documentation cannot explain the rationale of the
methodology or the calculations underlying the cost elements.




22
  A resource loaded schedule is a schedule with resources of staff, facilities, and
materials needed to complete the activities that use them; it should be based on the
project’s WBS.




Page 12                                                        GAO-12-59 IRS Management
Interactive graphic         Figure 2: 2011 IRDM Cost Estimate Alignment with Best Practices for Being Well Documented


                            Directions:
                            Rollover each              below to see further information on cost estimation best practices.

                                                                                                             Overall assessment:

                                                                                                             Minimally meets best practices for a
A well documented cost estimate should:                                                                      well documented cost estimate.

                                            Assessment of whether best
Best practices characteristics              practices are met                                       Effect

Capture the source data                     Data sources are listed, but the estimate is            Data are the foundation of every cost estimate. Depend-
used, the reliability of the                not consistent with the source data that it             ing on data quality, an estimate can range anywhere
data, and how the data were                 cites bringing into question the reliability of         from a mere guess to a highly defensible cost position.
made compatible with other                  the data.                                               Data are often in many different forms and need to be
data in the estimate.                       Minimally meets                                         adjusted before being used. The cost estimator needs
                                                                                                    information about the source and reliability of the data
                                                                                                    in order to know whether the data collected can be used
                                                                                                    directly or need to be modified.

Describe the calculations                   Documentation does not fully explain how                Poorly documented cost estimates can cause a pro-
and the methodology used                    IRS derived estimated costs.                            gram’s credibility to suffer because the documentation
to derive each element’s                    Partially meets                                         cannot explain the rationale of the methodology or the
cost.                                                                                               calculations. Estimates that lack sufficient documentation
                                                                                                    are not useful for updates or information sharing and can
                                                                                                    hinder understanding and proper use.

Describe how the estimate                   Labor cost calculations are described at a              Without good documentation, management and oversight
was developed.                              high level, but the staffing level is not trace-        organizations will not be convinced that the estimate is
                                            able to the schedule. Hardware and software             credible; supporting data, lessons learned, and reasons
                                            calculations are not described, and costs are           why costs changed will not be available for future use;
                                            not consistent across documents.                        questions about the approach or data used to create
                                            Minimally meets                                         the estimate cannot be answered; and the scope of the
                                                                                                    analysis cannot be thoroughly defined.

Discuss the technical                       Documentation links hardware and software               Because the technical baseline is intended to serve as the
baseline description.                       specifications to costs, but project-specific           basis for developing a cost estimate, it should be discussed
                                            costs are only provided for FY 2012.                    in the cost estimate documentation. Without a technical
                                            Minimally meets                                         baseline, the cost estimate will not be based on a comprehensive
                                                                                                    program description and will lack specific information regarding
                                                                                                    technical and program risks.


Provide evidence of                         The estimate was developed by IRDM man-                 A cost estimate is not considered valid until management
management review and                       agers, not cost estimators, and while it was            has approved it. It is imperative that management under-
acceptance.                                 reviewed within the IRDM program, there                 stand how the estimate was developed, including the risks
                                            is no evidence of review by the top IRDM                associated with the underlying data and methods.
                                            managers.
                                            Minimally meets

                            Source: GAO analysis of IRS’s 2011 IRDM cost estimate and GAO-09-3SP.




    Print instructions      A full text version of this graphic is available in appendix II.

                                                     Page 13                                                                       GAO-12-59 IRS Management
The estimate minimally meets best practices for an accurate cost
estimate, as shown in figure 3. Calculations in the estimate are
mathematically correct. However, documentation that IRS provided to
support estimated costs for IRDM hardware and software’s estimated
costs did not match estimates in IRDM’s spend plans. IRS officials said
the discrepancies occurred because the spend plans were developed
using more recent cost information for software purchases that were not
included in the supporting documentation. Additionally, the estimate does
not list any confidence levels or provide a range of possible costs.
According to best practices, unless an estimate is based on an
assessment of the most likely costs and reflects the degree of uncertainty
given all of the risks considered, management will not be able to make
informed decisions. IRDM uses EVM to identify variances between
planned and actual costs, but as discussed below, we found IRDM’s EVM
data to be unreliable, and there was no evidence that IRS uses actual
cost data to evaluate whether cost projections are realistic.




Page 14                                            GAO-12-59 IRS Management
Interactive graphic        Figure 3: 2011 IRDM Cost Estimate Alignment with Best Practices for Accuracy


                           Directions:
                           Rollover each              below to see further information on cost estimation best practices.

                                                                                                               Overall assessment:

An accurate cost estimate:                                                                                     Minimally meets best practices for an
                                                                                                               accurate cost estimate.
                                         Assessment of whether best
Best practices characteristics           practices are met                                         Effect

Produces unbiased                        No confidence levels are listed and docu-                 A cost estimate is biased if the estimated work is overly con-
results.                                 ments do not provide a range of possible                  servative or too optimistic. Unless the estimate is based on an
                                         costs. Neither risk nor uncertainty is                    assessment of the most likely costs and reflects the degree of
                                         mentioned.                                                uncertainty given all of the risks considered, management will
                                         Does not meet                                             not be able to make informed decisions.


Is properly adjusted for                 Documentation contains no evidence that                   Adjusting for inflation is important because in the development
inflation.                               the cost estimate is adjusted for inflation.              of an estimate, cost data must be expressed in like terms. If
                                         Does not meet                                             a mistake is made or the inflation amount is not correct, cost
                                                                                                   overruns can result.


Contains few mistakes.                   Documentation does not specify                            Without access to estimate details, one cannot be certain that
                                         whether the cost estimate went through                    calculations are accurate or expressed consistently.
                                         a quality control process, and there are
                                         inconsistencies among documents, but
                                         calculations are accurate.
                                         Partially meets

Is regularly updated to                  IRS does not maintain an updated IRDM                     If a cost estimate is not updated, it can become more difficult to
reflect significant                      cost estimate in a single document.                       analyze changes in program costs and collect cost and schedule
program changes.                         However, IRDM costs are tracked through                   data to support future cost estimates. The cost estimate should
                                         the program’s WBS and EVM system, and                     be updated when the technical baseline changes; otherwise, it
                                         evaluates them to develop spend plans, but                will lack credibility. A properly updated cost estimate can provide
                                         we found that the EVM data are not reliable.              decision makers with accurate information for assessing alterna-
                                         Minimally meets                                           tive decisions.

Documents and explains                   Variances between planned and actual                      Without a documented comparison between the current estimate
variances between                        costs, and explanations for the variance                  (updated with actual costs) and the old estimate, cost estima-
planned and actual                       are documented in EVM data, but the                       tors cannot determine the level of variance between the two
costs.                                   EVM data are unreliable.                                  estimates. That is, the estimators cannot see how well they are
                                         Minimally meets                                           estimating and how the program is changing over time.

Reflects cost estimating                 Documentation contains no evidence                        Historical data provides the cost estimator with insight into
experiences from                         that the estimate is based on a model                     actual costs on similar programs, including any cost growth that
comparable programs.                     that uses historical records of cost                      occurred after the original estimate. As a result, historical data
                                         estimating and actual experiences from                    can be used to challenge optimistic assumptions and bring more
                                         comparable programs.                                      realism to a cost estimate.
                                         Does not meet
                           Source: GAO analysis of IRS’s 2011 IRDM cost estimate and GAO-09-3SP.



    Print instructions     A full text version of this graphic is available in appendix II.

                                                   Page 15                                                                            GAO-12-59 IRS Management
The estimate does not meet best practices for a credible cost estimate, as
shown in figure 4. For example, the estimate was not crosschecked 23 or
assessed for risk and uncertainty. According to best practices, an
estimate without risk and uncertainty analysis 24 can be unrealistic
because it does not assess how the cost estimate would be affected if, for
example, the schedule slipped, the mission changed, or a proposed
solution did not meet users’ needs. In addition, IRS did not perform a
sensitivity analysis. 25 Further, there is no evidence that another office
performed a separate cost estimate—referred to as an independent cost
estimate—to validate the 2011 cost estimate. In previous work we found
that because of limited resources, IRS generally only does an additional
independent cost estimate for its largest programs, and according to
officials, IRDM is not considered a large enough program in terms of its
funding level. 26 While some of MITS’s cost estimates are done by EPO—
which is independent of program management offices—the 2011 cost
estimate was done by the IRDM program office.




 Cross-checks use alternative estimation methods to see if they produce similar results,
23

and could enhance the estimate’s reliability.
 Risk and uncertainty refer to the fact that because a cost estimate is a forecast, there is
24

always a chance that the actual cost will differ from the estimate.
 A sensitivity analysis examines the effects on changing assumptions and estimating
25

procedures to highlight elements that are cost-sensitive.
26
  Although IRDM is not among IRS’s largest programs by funding level, it is one of IRS’s
top seven investments, as determined by funding level and mission impact. According to
our cost guide, for an estimate to be credible, it must be compared to an independent cost
estimate. However, program managers and the cost estimating team should define the
scope of the estimate based on its intended purpose, including the appropriate level of
detail for an independent cost estimate.




Page 16                                                          GAO-12-59 IRS Management
Interactive graphic         Figure 4: 2011 IRDM Cost Estimate Alignment with Best Practices for Credibility




                            Directions:
                            Rollover each              below to see further information on cost estimation best practices.


                                                                                                             Overall assessment:

                                                                                                             Does not meet best practices for a
A credible cost estimate includes:                                                                           credible cost estimate.


                                             Assessment of whether best
Best practices characteristics               practices are met                                      Effect

A sensitivity analysis that                  Documentation contains no evidence that a              Because uncertainty cannot be avoided, it is necessary
identifies a range of possible               sensitivity analysis was performed.                    to identify the cost elements that represent the most risk.
costs based on varying                       Does not meet                                          A sensitivity analysis reveals how the cost estimate is
inputs.                                                                                             affected by a change in a single assumption, which helps
                                                                                                    the cost estimator to understand which variable(s) most
                                                                                                    affects the cost estimate. Any sources of variation should
                                                                                                    be well documented and traceable.

A risk and uncertainty                       Documentation contains no evidence that a              For management to make good decisions, the program
analysis.                                    risk analysis or an uncertainty analysis were          estimate must reflect the degree of uncertainty, so that a
                                             done.                                                  level of confidence can be given about the estimate. An
                                             Does not meet                                          estimate without risk and uncertainty analysis is unre-
                                                                                                    alistic because it does not assess the variability in the
                                                                                                    cost estimate from such effects as schedules slipping,
                                                                                                    missions changing, and proposed solutions not meeting
                                                                                                    users’ needs.

Cross-checking of major                      Documentation contains no evidence that                If a cross-check demonstrates that alternative methods
cost elements.                               cross-checks were performed.                           can produce similar results, then confidence in the esti-
                                             Does not meet                                          mate increases, leading to greater credibility.


A comparison to an                           Documentation contains no evidence that an             An independent cost estimate is considered one of the best
independent cost estimate                    independent cost estimate was performed.               and most reliable methods for validating an estimate. It
conducted by another                         Does not meet                                          provides an independent view of expected program costs
organization.                                                                                       that tests the program office’s estimate for reasonableness.
                                                                                                    Without an independent cost estimate, decision makers will
                                                                                                    lack insight into a program’s potential costs because inde-
                                                                                                    pendent cost estimates frequently use different methods
                                                                                                    and are less burdened with organizational bias.

                            Source: GAO analysis of IRS’s 2011 IRDM cost estimate and GAO-09-3SP.




    Print instructions      A full text version of this graphic is available in appendix II.

                                                    Page 17                                                                        GAO-12-59 IRS Management
                          Because the 2011 cost estimate does not meet best practices, it does not
                          provide reliable support for IRDM’s fiscal year 2012 budget request, or
                          any of the projected budget requests. IRS officials said current IRS policy
                          does not require projects to routinely re-estimate project cost. The IRDM
                          program office—which does not use the same software or modeling
                          techniques as EPO—relied on spend plans, EVM, and other documents
                          to estimate costs. In July 2011, MITS officials said that it would take 90
                          days for IRDM and EPO staff to complete a new cost estimate for IRDM.
                          When considering FTEs and time, a new cost estimate developed by
                          IRDM and EPO staff would require a total of about eight staff months. 27


IRDM Could Benefit From   EPO has specialized cost estimation tools, such as software that
EPO Assistance on         incorporates many best practices from our cost guide, and expertise that
Updating Cost Estimates   project teams can leverage to update cost estimates. If used correctly,
                          EPO estimation procedures could help IRDM management to maintain
                          reliable cost information for use in budget requests. EPO officials said
                          they did not work with the IRDM team to maintain an updated SCBE
                          because the team did not seek their assistance. IRDM was not required
                          to do so because MITS guidance, as of September 2011, does not
                          require project teams to consult with EPO when updating a cost estimate.
                          Without EPO involvement, IRS has less assurance that cost estimate
                          updates will follow best practices. Our cost guide states that cost
                          estimates should be (1) updated to reflect actual costs and changes (i.e.,
                          significant modifications to a project’s scope or specifications) in order to
                          keep the estimate current as the program passes through new phases
                          and milestones and (2) updated if there are significant cost, schedule or
                          performance variances. The continual updating of the cost estimate as
                          the program matures not only results in a higher-quality estimate, but also
                          gives cost estimators the opportunity to collect data for use in future
                          estimates as well as incorporate lessons learned.

                          Our cost guide also states that cost estimation work should be done by a
                          central independent estimating organization, and estimators should



                          27
                            IRS officials said it would take 1.5 FTEs for estimators working for 90 days (about 4.5
                          staff months), 1.5 FTEs from IRDM staff working for 60 days to assist with the estimate
                          (about 3 staff months), and 0.6 additional IRDM FTEs working for 30 days (about 0.6 staff
                          months). According to our calculations using a top government salary rate and general
                          benefits rate, the direct staff cost associated with updating the cost estimate would likely
                          be less than $200,000.




                          Page 18                                                         GAO-12-59 IRS Management
monitor programs to determine whether preliminary information and
assumptions remain relevant and accurate. EPO has the following
characteristics—unlike the IRDM project team—that could help provide
more reliable cost estimate updates:

•    EPO is able to use robust cost estimation techniques, including the
     SEER-SEM software cost estimation model. 28 SEER-SEM analyzes
     project histories and cost relationships to produce cost estimates and
     can estimate costs consistent with best practices—such as adjusting
     for risk and incorporating the results of a sensitivity analysis. When
     EPO estimators validate or update a project’s funding requirements,
     they tailor SEER-SEM to the project and use it to consider actual cost
     data from the project team, according to EPO’s Estimator’s Reference
     Guide. Estimators calibrate the model to include the schedule for
     remaining work and evaluate and revise key cost drivers, according to
     the guidance. As mentioned previously, the 2011 IRDM cost estimate
     did not use a cost estimation model.
•    EPO, as a whole, has more cost estimation experience than the IRDM
     project team. EPO’s six cost estimators have 43 years of combined
     cost estimation experience. They all have received training in SEER-
     SEM and other cost estimation models. In addition, EPO officials said
     that project teams generally would not have the technical skills to
     update a SCBE using cost estimation models. Although the IRDM
     project team has some cost estimation experience and relevant
     training, IRDM officials do not have SEER-SEM training or
     experience. Further, our analysis of IRDM’s 2011 cost estimate
     illustrates that estimate updates done by project teams may not result
     in reliable cost information. According to EPO officials, an updated
     estimate developed by EPO would also be independent, more holistic,
     and would include elements that project teams may miss.
•    EPO-produced updates can help build a historical record of IRS cost
     estimate data. According to our cost guide, historical data are crucial
     to developing high-quality cost estimates because estimators usually
     develop estimates for new programs by relying on data from programs
     that already exist and adjusting for any differences. EPO officials told
     us that they are working to build a historical database that compares
     estimated costs to actual costs. As IRS’s central estimating
     organization, EPO is uniquely qualified to use cost estimate updates



28
  SEER mathematical models are commercially available cost estimation models, derived
from extensive software project histories, behavioral models, and metrics.




Page 19                                                    GAO-12-59 IRS Management
     as an opportunity to obtain data that are consistent with other
     estimates and to use the data to build a historical cost database,
     which can ensure that future cost estimates are credible.
According to MITS officials, it is up to project teams to seek EPO
assistance. However, MITS’s Estimation Procedures document, which
provides cost estimation guidance to project teams, suggests some cost
and schedule variance and project size thresholds that, if exceeded,
should cause project managers to contact EPO for an updated estimate.
It recommends EPO involvement in updating estimates when:

•    cost or schedule variance are 10 percent or greater for major projects;
•    cost or schedule variance are 25 percent or greater for non-major
     projects; 29 or
•    a project with development/modernization/enhancement (DME) 30
     costs greater than $5 million reaches milestone 3. 31

IRDM meets the first threshold. Specifically, IRDM meets the IRS’s
criteria for a “major” project because IRDM’s projected life-cycle costs are
about $166 million, based on funding projections submitted to OMB. Also,
according to IRDM EVM data, as of September 30, 2011, the program
had a greater than 18 percent cost variance and an almost 13 percent
schedule variance. IRDM officials said they did not work with EPO to
update the 2009 SCBE because it was not required by current MITS
guidance.

MITS officials said is it not feasible to require EPO to update all cost
estimates for IT projects. EPO is a relatively new office that began



29
  IRS defines major investments as those that, among other things, have an overall life-
cycle cost of greater than $50 million or an annual budget of greater than $5 million.
Investments that do not meet these criteria are considered non-major.
30
  DME is a term used by OMB to describe the program cost for new investments, changes
or modifications to existing systems to improve capability or performance, changes
mandated by the Congress or agency leadership, personnel costs for investment
management, and direct support. For major IT investments, this amount should equal the
sum of amounts reported for planning and acquisition plus the associated FTE costs
reported in the Exhibit 300.
31
  A milestone is a point in time when management reviews updated cost, progress, and
risk information. According to a MITS document, IRDM and eight other programs have
DME costs exceeding $5 million in calendar year 2011. IRDM’s DME costs exceed $5
million, but the program is not scheduled to exit milestone 3 until the summer of 2012.




Page 20                                                        GAO-12-59 IRS Management
                            developing cost estimates in 2006. Most of its work has focused on
                            developing estimates for proposed projects, rather than updating
                            estimates for existing projects. The Estimation Procedures document was
                            developed in September 2011, and as of October 2011, officials said
                            EPO has six cost estimators on staff. As a result, EPO officials said that
                            project teams generally update IT project cost estimates without EPO
                            assistance and that, as of October 2011, EPO has been involved in few
                            cost estimate updates. However, senior MITS and EPO officials said they
                            would like for EPO to have a greater role in cost estimate updates. If IRS
                            does not have reliable cost estimate updates, projects may face risks and
                            their budget requests may not be adequately justified to inform decision
                            making; these outcomes could be even more significant for projects with
                            cost or schedule variances or high DME costs, such as IRDM.


MITS Guidance Has           MITS guidance documents used by project managers do not clearly
Inconsistencies and Could   discuss the appropriate uses of different types of cost estimates.
Better Link Cost            According to current guidance used by EPO estimators, non-SCBE cost
                            estimates are less rigorous and are not for use in budgets, but as stated
Estimation Procedures       above, neither the IRDM initial budget request nor the current and
with Budget Requests        projected budgets were developed using information from the 2009
                            SCBE.

                            Three IRS guidance documents describe the relationship between cost
                            estimates and budgets. However, the documents are directed at different
                            audiences, do not present consistent information, and contain different
                            levels of detail. Specifically:

                            •   EPO’s Estimator’s Reference Guide, used by EPO estimators, states
                                that budgets for IT projects should be established using SCBEs. IRS’s
                                SCBEs rely on cost estimation methods that incorporate best
                                practices from our cost guide, including considerations of risk, and
                                provide a level of confidence associated with the estimate. According
                                to our cost guide, for management to make good decisions, the
                                program estimate must reflect the degree of uncertainty, so that a
                                level of confidence can be given about the estimate. The Estimator’s
                                Reference Guide also discusses techniques that estimators can use
                                to update SCBEs, and aligns that process with annual budget
                                submissions. Although this guidance contains many best practices, it
                                is directed at cost estimators; therefore, project managers do not
                                typically have access to it or use it.
                            •   The MITS Investment Guide, directed at MITS project managers,
                                discusses the role of EPO in developing initial rough estimates and



                            Page 21                                             GAO-12-59 IRS Management
    budget-ready SCBEs and states that, if a project does not yet have an
    SCBE, a rough estimate may be used as a placeholder in a budget
    request. The guide requires that MITS staff should work to ensure that
    if an SCBE exceeds an initial rough estimate, the project’s scope and
    SCBE fit within the appropriated budget. However, the guide does not
    discuss how, if at all, a budget request should be adjusted if an SCBE
    provides an estimated cost that is lower than the budget, or how any
    future cost information should be incorporated into budgets.
•   The Estimation Procedures document, directed at MITS project
    managers, does not define types of cost estimates or discuss whether
    they are appropriate for budget decisions. EPO officials said they did
    not believe it is necessary to characterize the different types of cost
    estimates in the Estimation Procedures document because they are
    not necessary for defining the organizational approach to estimation,
    which is the intent of the document. The document states that if
    updated cost estimates indicate that a project’s budget needs to
    change, the changes must be approved. However, it does not specify
    who should approve the estimates.

Without consistent guidance about what types of cost estimates are
appropriate for budget requests, project teams may not use the best
information available. Our cost guide states that, as a best practice, an
estimate intended to support budgetary decisions should cover the
project’s entire life-cycle and should be supported by a description of the
program’s technical characteristics, which would be found in an SCBE.
Using a cost estimate that lacks sufficient rigor—such as a preliminary
cost estimate, instead of an SCBE—could lead to budget requests that do
not accurately reflect program funding needs. For example, the 2007
preliminary IRDM estimate lacks an uncertainty analysis, which would
provide a basis for adjusting the estimate to reflect unknown facts and
circumstances that could affect costs, and as a result, IRDM managers do
not have assurance that the program’s funding level remains appropriate.
Further, not providing project managers with guidance on how to
incorporate new cost information—either from an SCBE that has replaced
a preliminary estimate or from an updated cost estimate—into budget
requests could result in requests that do not reflect current or accurate
funding needs for a project.




Page 22                                             GAO-12-59 IRS Management
                      IRS provided EVM data in the 2011 IRDM cost estimate to justify its
Unreliable EVM Data   budget requests, but we found that the program’s EVM data are not
Raise Additional      reliable in any of the areas we reviewed. Reliable data on actual
                      performance, obtained from an EVM system, are a necessary input if an
Concerns about IRDM   updated cost estimate is to be considered accurate and credible.
Cost Estimate         Because IRDM’s 2011 cost estimate is based on unreliable EVM data, it
                      does not provide adequate support for IRDM’s budget requests. Until IRS
                      addresses deficiencies in its EVM data, it cannot provide reliable cost
                      estimate updates for IRDM. EVM data reliability deficiencies, such as
                      those we observed for IRDM, are common in federal agencies, and we
                      have also previously reported on inconsistencies in implementation of
                      EVM for IT projects at Treasury bureaus. 32

                      Our cost guide identifies top-level EVM data reliability tasks for IT
                      projects, which are also included in OMB guidance. 33 We assessed IRDM
                      EVM data and IRS’s processes against three data reliability tasks:

                      •    Maintain an EVM system that is compliant with the agency’s scaling of
                           American National Standards Institute (ANSI) guidelines. ANSI has a
                           national EVM standard, comprised of 32 guidelines, which define
                           acceptable methods for agencies to evaluate an EVM system to
                           determine if cost, schedule, and technical performance data can be
                           relied on for program management.
                      •    Conduct an integrated baseline review for the program. An integrated
                           baseline review is an evaluation of a program’s baseline plan to


                      32
                        In a review of 16 federal programs using EVM, we found that many programs did not
                      fully implement practices to ensure data reliability. For example, 13 of the programs had
                      deficiencies in program schedules, which undermined the quality of EVM data.
                      Additionally, some programs did not conduct an integrated baseline review, or conduct
                      ongoing EVM surveillance. The inconsistent application of EVM across the programs
                      exists in part because of the weaknesses in some of the agencies’ policies, as well as a
                      lack of enforcement of the EVM policy. See GAO, Information Technology: Agencies Need
                      to Improve the Implementation and Use of Earned Value Techniques to Help Manage
                      Major System Acquisitions, GAO-10-2 (Washington, D.C.: Oct. 8, 2009). In a review of the
                      use of EVM at 6 selected Treasury projects, we found that none of the projects fully
                      implemented practices to ensure that the data from their EVM system was reliable. The
                      review included projects at the Financial Management Service, IRS, the Bureau of Public
                      Debt, and the Treasury departmental offices. See GAO, Information Technology: Treasury
                      Needs to Better Define and Implement Its Earned Value Management Policy, GAO-08-951
                      (Washington, D.C.: Sept. 22, 2008).
                       See GAO-09-3SP and Office of Management and Budget, Capital Programming Guide:
                      33

                      Supplement to Circular A-11, Planning, Budgeting, and Acquisition of Capital Assets
                      (Executive Office of the President, Washington, D.C.: August 2011).




                      Page 23                                                      GAO-12-59 IRS Management
                                determine whether all program requirements have been addressed,
                                risks have been identified, mitigation plans are in place, and available
                                and planned resources are sufficient to complete the work.
                           •    Using qualified staff, conduct surveillance on the EVM system.
                                Surveillance is reviewing a contractor’s EVM system to observe ANSI
                                compliance and how well a contractor is using its EVM system to
                                manage cost, schedule, and technical performance.
                           Treasury’s EVM guidance requires a project of IRDM’s size to follow an
                           abbreviated set of 10 ANSI guidelines, and to conduct surveillance on the
                           EVM system. Other departments also scale ANSI guidelines according to
                           the size of projects, which could result in some agencies not fully
                           following certain best practices. Further, projects like IRDM, according to
                           Treasury guidance, only need to complete an independent baseline
                           validation—which although not defined in the guidance, appears to be a
                           less rigorous version of an integrated baseline review. Following ANSI
                           guidelines and conducting an integrated baseline review and surveillance
                           can help ensure that EVM data can indicate how well a program is
                           performing in terms of cost, schedule, and technical matters. This
                           performance information is necessary for proactive program management
                           and risk mitigation, and to maintain a reliable cost estimate. Where
                           applicable, we assessed IRDM’s EVM data against the standards cited in
                           the Treasury guidance.


IRDM’s EVM System Is Not   We assessed IRDM on three ANSI guidelines, which are fundamental
Compliant with Key         elements for an EVM system and are included in Treasury’s abbreviated
Required Guidelines        10 EVM guidelines. 34 We found that IRDM’s EVM system is not compliant
                           with these guidelines. For an overview of our findings on IRDM’s EVM
                           data reliability, see table 4 in appendix III.

                           For each selected guideline we found:

                           •    WBS: This ANSI guideline states that authorized work elements for
                                the program should be defined, which typically includes using a WBS



                           34
                             We assessed IRDM’s adherence to ANSI guidelines 1, 6, and 8. For DME projects
                           costing less than $20 million, such as IRDM, Treasury EVM guidance states that a project
                           only needs to follow a set of 10 ANSI guidelines (numbers 1, 2, 3, 6, 7, 8, 16, 22, 27, and
                           28). Treasury’s 10 guidelines include 9 of the 10 guidelines that are considered by experts
                           to be the most critical to follow. Treasury’s list does not include guideline 9. For a list of the
                           32 ANSI guidelines, see GAO-09-3SP pgs. 212-213.




                           Page 24                                                             GAO-12-59 IRS Management
       tailored for effective internal control. Further, a project’s schedule, 35
       cost estimate, and EVM system should be based on the same WBS,
       according to our cost guide. The WBSs used in IRDM’s schedules do
       not match the WBS used for EVM. The WBSs in four of the five IRDM
       schedules reflect detailed project-level tasks, while the WBS used for
       the EVM data is only broken down by contractor and government
       efforts, and does not include any project-level data. The WBS used to
       inform the 2011 IRDM cost estimate was not broken down by
       contractor or government data, and it did not provide costs for detailed
       tasks. Without a WBS, from which to measure progress and to serve
       as a consistent framework for the schedules and EVM, there is no
       basis for reliable EVM data, according to our cost guide. An IRS
       official said that the WBS in the IRDM schedules is not the primary
       source of financial information for IRDM. Instead, officials use IRS’s
       financial tracking system, which is much less detailed than the
       schedules’ WBS, to obtain project level data for EVM. This technique
       for gathering EVM data at a project level is contrary to the purpose of
       EVM, which is to integrate cost, schedule, and technical data from
       detailed work packages 36 that can be monitored for variances against
       the original plan. Since a resource-loaded schedule forms the
       foundation for the EVM baseline, both the schedule and the EVM data
       should be based on the same WBS, according to our cost guide. 37
       Because the financial tracking system only provides project level data,
       and financial information cannot be traced to the WBS elements in the
       schedules, the cost associated with the project tasks is unknown.
•      Sequencing: This ANSI guideline states that projects should have a
       schedule that describes the sequence of work—that is, a list of
       activities in the order in which they are to be carried out—and
       identifies significant task interdependencies required to meet project



 A schedule provides a time sequence for the duration of a program’s activities and helps
35

clarify both the dates for major milestones and the activities that drive the schedule. A
program schedule also provides the vehicle for developing a time-phased budget
baseline.
36
    Work packages are detailed tasks that are typically 4 to 6 weeks long.
37
  The WBS should be used as the outline for the schedule because the WBS defines the
work in lower levels of detail. Thus its framework provides the starting point for defining all
activities and tasks that will be used to develop the schedule. Furthermore, by breaking
the work into smaller, more manageable work elements, the WBS can be used to
integrate the scheduled activities and costs for accomplishing EVM. A WBS, therefore, is
an essential part of EVM cost, schedule, and technical monitoring because it provides a
consistent framework from which to measure progress.




Page 25                                                           GAO-12-59 IRS Management
     requirements. All of the IRDM schedules had significant problems with
     sequencing. For example, many predecessor and successor tasks
     were not linked to one another, which are necessary for properly
     sequencing work so that the schedule will update in response to
     changes. Because the schedule is missing so many of these “logic
     links,” it will not automatically recalculate forecasted start and finish
     dates of remaining activities. Thus, any activities that are late will not
     automatically recalculate the dates for affected successor activities.
     IRS officials said they are aware of some issues with missing links. As
     a result of these missing links in all of the IRDM schedules we
     reviewed, IRDM’s schedules are not reliable. Because the schedules
     form the basis for the performance measurement baseline 38 used to
     track cost and schedule variances in an EVM system, the data from
     the IRDM EVM system are not reliable.
•    Time-phased budget baseline: This ANSI guideline states that a
     program should establish and maintain a time-phased budget
     baseline, 39 at the control account level, against which performance
     can be measured. Resources must be accounted for in a schedule in
     order to develop this baseline, according to our cost guide. IRS was
     unable to show evidence that it has established and maintained a
     time-phased budget baseline for IRDM. Specifically, the program does
     not have one overall schedule, and none of the IRDM schedules we
     reviewed were completely resource-loaded.

IRS officials said although they do not have a program level schedule for
IRDM, the individual project schedules are linked through
interdependencies. However, we could not identify these links in our
analysis. IRS officials said the resource-loaded schedules do not show all
project resources because some resources, such as contractor personnel
being used across projects, are used in more than one IRDM project.
Without schedules that include the resources needed to complete tasks,
IRS was unable to prove that it had established and maintained a time-



 A performance measurement baseline is used in EVM to detect deviations from the plan
38

and give insight into problems and potential impacts.
39
  The time-phased budget baseline, against which performance is measured, is formed
from the performance measurement baseline, which is essentially the resource
consumption plan for the program. Deviations from the baseline identify areas where
management should focus attention. A performance measurement baseline represents
the cumulative value of a program’s planned work over time. It takes into account the
program activities that occur in a sequenced order, based on finite resources, with
budgets representing those resources spread over time.




Page 26                                                      GAO-12-59 IRS Management
                           phased budget baseline. This baseline is a critical EVM component for
                           measuring IRDM’s performance.


IRS Did Not Validate the   According to Treasury EVM guidance, an independent baseline validation
Baseline for IRDM’s EVM    should be conducted for a DME project like IRDM. As stated above, GAO
System                     and OMB consider an integrated baseline review to be a key element of
                           EVM data reliability, while Treasury guidance allows projects like IRDM to
                           complete a less-rigorous independent baseline validation. IRS did not
                           conduct an integrated baseline review or an independent baseline
                           validation for IRDM. IRS officials told us that many of the activities
                           typically done during an integrated baseline validation were performed—
                           such as developing a WBS and schedules. However, as previously
                           discussed we identified problems with the WBSs and schedules and the
                           baseline process. Without a comprehensive integrated baseline review or
                           independent baseline validation, and resolving any issues, IRS has not
                           sufficiently evaluated the validity of IRDM’s baseline. This calls into
                           question the reliability of IRDM’s EVM data, and could affect the
                           program’s ability to identify and mitigate risks.


IRS Is Not Conducting      OMB and Treasury require surveillance on EVM systems, and, according
Surveillance on IRDM’s     to our cost guide, surveillance should be conducted to check whether the
EVM System                 EVM system summarizes timely and reliable cost, schedule, and
                           performance information, among other things. IRS officials said IRS is not
                           performing surveillance on the IRDM EVM system because they did not
                           believe it was required and because, according to officials, IRS does not
                           have staff with the necessary technical skills to conduct surveillance. IRS
                           officials said Treasury is reviewing whether Department-level surveillance
                           would be efficient. It is important for the agency to conduct surveillance of
                           EVM systems to ensure that contractors are following their own
                           processes and satisfying ANSI guidelines.


                           The 2011 IRDM cost estimate does not fully meet best practices for a
Conclusions                reliable estimate. It is important for IRDM to have a cost estimate that
                           meets best practices to inform budgetary decisions and ensure that the
                           program is implemented as planned. This standard is particularly
                           important in a budgetary environment with scarce resources. IRDM’s
                           2011 cost estimate could be improved by using EPO’s expertise to
                           ensure the cost estimate follows best practices from our cost guide.
                           Additionally, more consistent guidance on using cost estimates to develop
                           budget requests could help program managers for IRDM, as well as other


                           Page 27                                              GAO-12-59 IRS Management
                      programs, make budget decisions that are supported by reliable cost
                      information.

                      IRS could increase the credibility of IRDM’s 2011 cost estimate by
                      ensuring that IRDM’s EVM data are reliable. Such reliability could also
                      allow for IRS to update projected costs for the remainder of IRDM’s
                      implementation. Using a WBS that is developed using best practices from
                      our cost guide could provide a baseline from which to measure progress,
                      a key component to an EVM system. Similarly, developing a single
                      integrated schedule for IRDM, that contains all resources needed to
                      implement the program, could provide more meaningful EVM data.
                      Finally, providing oversight of the EVM system, through an independent
                      baseline validation and EVM surveillance, could help identify potential
                      program risks and any possible issues with contractor performance.


                      To improve the quality of cost and budget information for IRS IT projects,
Recommendations for   we recommend that the Commissioner of Internal Revenue take the
Executive Action      following four actions:

                      1. Ensure that the IRDM life-cycle cost estimate is reliable and that
                         budget requests are justified by a reliable cost estimate that follows
                         best practices.
                      2. Require project managers to consult with EPO to determine if projects
                         could benefit from EPO assistance in updating cost estimates for
                         programs that exceed thresholds recommended by MITS’s Estimation
                         Procedures document. For those projects where EPO does not
                         update the cost estimate, IRS should require that the decision and
                         rationale be documented. EPO should use the information from
                         updated cost estimates to develop a historical repository of cost
                         estimation data.
                      3. Review all guidance applicable to cost estimates and take steps to
                         ensure that they are consistent. As a first step, IRS guidance should
                         require the use of current and reliable project cost estimates to inform
                         budget requests, in accordance with the Estimator’s Reference Guide.
                      4. Improve the reliability of IRDM’s EVM data, specifically:
                         •   address WBS issues by developing an EVM baseline for IRDM
                             that reflects the same WBS as the detailed schedule and IRDM
                             cost estimate;
                         •   address sequencing issues and enable the development of a time-
                             phased budget baseline by creating a single integrated master
                             schedule for IRDM that is properly sequenced and resource-




                      Page 28                                             GAO-12-59 IRS Management
                               loaded so that effective and meaningful EVM data can be
                               obtained to better manage the program;
                         •     conduct an independent baseline validation for the IRDM EVM
                               baseline; and
                         •     conduct independent surveillance of EVM systems to ensure that
                               data are reliable.

                     We provided a draft of this report to the Commissioner of Internal
Agency Comments      Revenue for his review and comment. We received written comments
and Our Evaluation   from the Deputy Commissioner for Operations Support, which are
                     reprinted in appendix IV. We sought clarification on IRS’s written
                     response in regards to whether it agreed with two of our
                     recommendations, and on a reference to OMB guidance. IRS provided us
                     with additional comments, which are summarized below. In addition, the
                     agency provided technical comments, which we incorporated into the
                     report as appropriate. IRS agreed with one of our four recommendations,
                     partially agreed with another, and disagreed with two.

                     While IRS’s comment letter did not address the recommendation, the
                     Director of Risk Management in MITS’s Strategy and Planning Office told
                     us the agency agrees with the recommendation to require certain IT
                     project managers to consult with EPO about updating cost estimates,
                     documenting decisions not to update cost estimates, and placing data
                     from updated cost estimates in a repository. Similarly, IRS’s comment
                     letter did not address our recommendation to ensure that its cost
                     estimation guidance is consistent. However, IRS officials said they
                     partially agree with this recommendation and have taken steps to ensure
                     that their estimation practices and procedures follow consistent,
                     documented guidance. They noted that in all instances, however, IRS IT
                     cost estimates will be based on the best information available at the time
                     the estimate is requested or required as opposed to our recommendation
                     that IRS require the use of current and reliable cost estimates to inform all
                     budget requests. We note in our report that using an unreliable cost
                     estimate could lead to budget requests that do not accurately reflect
                     program funding needs. Once done in conformance with guidance and
                     best practices, current and reliable cost estimates can be maintained
                     through normal required monitoring and cost tracking procedures, unless
                     significant changes in project circumstances warrant updating the cost
                     estimate.




                     Page 29                                              GAO-12-59 IRS Management
IRS disagreed with our recommendation to ensure that the IRDM life-
cycle cost estimate is reliable and that budget requests are justified by a
reliable cost estimate that follows best practices. In its comment letter,
IRS wrote that implementing the recommendation would require it to
spend resources that do not directly contribute to the successful
implementation of the IRDM program, and that the IRDM program is
within its budget and schedule. At the end of fiscal year 2011, IRDM was
under budget by more than 18 percent and behind schedule by almost 13
percent, equal to over 4 months behind schedule, according to IRDM’s
EVM data. Because the IRDM program does not have a reliable cost
estimate, budget authorities do not have reliable information to determine
an appropriate funding level. Such variances indicate that the current
funding level may not be appropriate. As we reported, IRS estimated in
July 2011 that it would take about 90 days, comprising 8 staff months (or
a direct staff cost likely less than $200,000, according to our calculations
using a top government salary rate and general benefits rate), to
complete a new cost estimate for IRDM. While we agree that federal
resources are tight, we believe that such an investment could produce
benefits that not only improve the reliability of the IRDM cost estimate, but
also better ensure that IRS requests the correct amount of resources to
ensure IRDM fully achieves successful implementation. Further, benefits
of a new estimate would also stretch beyond that program and provide an
important foundation for improving IRS cost estimates in general. The
OMB Capital Programming Guide directs agencies to develop sound cost
estimates based on our cost guide and states that during the budget
process, the credibility of the costs will be examined, and OMB and the
Congress will hold agencies accountable for meeting the schedule and
performance goals within the cost estimates. More reliable cost estimates
enable Congress and other budget authorities to make more complete
and informed decisions.

IRS also disagreed with our recommendation to improve the reliability of
EVM data, and stated that OMB’s revisions to Circular A-11 remove EVM
system requirements due to negative cost benefit. We disagree with IRS,
as Circular A-11, Appendix J and the Capital Programming Guide still
contain language that directs agencies to use EVM for major projects.
Former Circular A-11, section 300.7, instructed agencies to use EVM
system requirements to identify areas where problems are occurring
when reporting on ongoing investments. While current guidance for
Exhibit 300 no longer explicitly discusses EVM reporting on ongoing
investments under section 300.7, other sections of the guidance still
direct the use of EVM for managing IT capital assets and state that in
general cost, schedule and performance goals are to be controlled and


Page 30                                              GAO-12-59 IRS Management
monitored by using an EVM system. Moreover, our report assessed the
reliability of IRDM’s EVM data because the data were included in IRDM’s
2011 cost estimate and the data are used to track the program’s
progress. Regardless of OMB requirements, any data used for cost
estimation and program management, particularly when it helps to
support a budget request, should be reliable.

We will send copies of this report to the Chairmen and Ranking Members
of Senate and House committees and subcommittees that have
appropriation, authorization, and oversight responsibilities for IRS. We will
also send copies to the Commissioner of Internal Revenue, the Secretary
of the Treasury, the Chairman of the IRS Oversight Board, and the Acting
Director of the Office of Management and Budget. Copies are also
available at no charge on the GAO Web site at http://www.gao.gov.

If you or your staffs have any questions concerning this report, please
contact me at (202) 512-9110 or brostekm@gao.gov. Contact points for
our Offices of Congressional Relations and Public Affairs may be found
on the last page of this report. GAO staff who made major contributions to
this report are listed in appendix V.




Michael Brostek
Director, Tax Issues
Strategic Issues




Page 31                                              GAO-12-59 IRS Management
Appendix I: Scope and Methodology
             Appendix I: Scope and Methodology




             This report builds on our May 2011 report 1 on the Internal Revenue
             Service’s (IRS) Information Reporting and Document Matching (IRDM)
             program and analysis performed during a budget justification review of
             the IRDM program, conducted from June 2011 to August 2011, which we
             provided to Congress as technical assistance. In the May 2011 report, we
             assessed IRDM’s 2009 solution-concept based cost estimate. For the
             budget justification work and this report, we assessed IRDM’s 2007
             preliminary cost estimate. Also for this report, we assessed IRDM’s 2011
             cost estimate.

             To assess the extent to which the IRDM funding request is supported by
             a reliable cost estimate, and if not reliably supported, why not, we
             compared the Modernization and Information Technology Services
             (MITS) division’s 2011 IRDM cost estimate with the characteristics of a
             high-quality cost estimate, identified in the GAO Cost Estimating and
             Assessment Guide. 2 The 2011 IRDM cost estimate documentation
             included spend plans, the Exhibit 300, Earned Value Management (EVM)
             data, project schedules, and other documents. 3 Our cost guide, which is
             based on extensive research of best practices for estimating program
             schedules and costs, indicates that a high-quality, valid, and reliable cost
             estimate should be well documented, comprehensive, accurate, and
             credible; we analyzed the cost estimating practices used by MITS against
             these characteristics and rated each characteristic as being either: Met,
             Substantially Met, Partially Met, Minimally Met, or Not Met. To do so, we
             scored each of the individual key practices associated with cost and
             scheduling best practices on a scale of 1-5 (Does Not Meet = 1, Minimally
             Meets = 2, Partially Meets = 3, Substantially Meets = 4, and Meets = 5),
             and then averaged the individual practice scores to determine the overall
             rating. We shared our cost guide as well as our preliminary analysis of the
             IRDM 2011 cost estimate with program officials. When warranted, we
             updated our analyses based on the agency’s response and additional



             GAO, Information Reporting: IRS Could Improve Cost Basis and Transaction Settlement
             1

             Reporting Implementation, GAO-11-557 (Washington, D.C.: May 19, 2011).
             See GAO, Cost Estimating and Assessment Guide: Best Practices for Developing and
             2

             Managing Capital Program Costs, GAO-09-3SP (Washington, D.C.: March 2009).
             3
              The Exhibit 300 is a document required by the Office of Management and Budget to
             support IT projects. It includes the project’s desired outcome and budget justification.
             Earned Value Management is a project management tool that integrates the technical
             scope of work with schedule and cost elements for investment planning and control.




             Page 32                                                         GAO-12-59 IRS Management
Appendix I: Scope and Methodology




documentation provided to us. Once we determined that the 2011 cost
estimate did not fully meet best practices, we determined why this
occurred. To this end, we interviewed officials in IRDM’s Program
Management Office and MITS’s Estimation Program Office (EPO). We
also compared IRS guidance that addresses cost estimation—the EPO’s
Estimator’s Reference Guide, MITS’s Estimation Procedures document,
and IRS’s Information Technology Investment Planning and Management
Guide—to criteria in our cost guide.

To assess the extent to which IRS’s practices for capturing IRDM’s actual
costs and comparing them to estimated costs, or EVM, generate reliable
performance data, we compared the EVM data for IRDM and IRS’s
process for maintaining the data to the high-level EVM data reliability
tasks outlined in our cost guide. 4 We assessed the extent to which
IRDM’s EVM data adhered to three of the American National Standard
Institute’s (ANSI) 5 32 guidelines; we selected the three guidelines to
represent some of the fundamental steps for maintaining a reliable EVM
system, as identified in our cost guide, and because these guidelines are
also included in the Department of the Treasury’s Earned Value
Management Guide, which applies to IRS (see appendix III for a list of the
data reliability tasks and ANSI guidelines we reviewed). 6 In situations
where Treasury’s Earned Value Management Guide did not require
certain OMB or GAO best practices, we assessed IRDM’s EVM practices
against the Treasury guidance. To do this analysis, we compared the
work breakdown structures used in IRDM’s EVM system, schedules, and
cost estimates and identified differences in each. We also assessed each
of the five IRDM schedules 7 against scheduling best practices for



4
 GAO-09-3SP, see pg. 98 for a list of EVM data reliability tasks, we assessed whether
IRDM is implementing the first three.
5
 ANSI has a national EVM standard, comprised of 32 guidelines, that define acceptable
methods for agencies to evaluate an EVM system to determine if cost, schedule, and
technical performance data can be relied on for program management. We assessed
whether IRDM was following guideline numbers 1, 6, and 8.
6
 OMB provides guidance for cost estimation and EVM, but some departments and
agencies may scale ANSI guidelines to fit specific projects. See Office of Management
and Budget, Capital Programming Guide: Supplement to Circular A-11, Planning,
Budgeting, and Acquisition of Capital Assets, (Washington, D.C.: Executive Office of the
President, August 2011).
7
 IRS officials provided five schedules for IRDM, one for each of the four IRDM projects,
and a schedule depicting IRDM’s progress following IRS Enterprise Lifecycle guidance.




Page 33                                                        GAO-12-59 IRS Management
Appendix I: Scope and Methodology




ensuring that the activities are sequenced and related using network
logic, as identified in our cost guide.

For both objectives, we interviewed IRS officials in the MITS division,
specifically, officials from the IRDM Program Management Office and the
Investment Planning and Management Office, which includes EPO. We
spoke primarily with officials at IRS Headquarters in Washington, D.C.
and IRS’s division office in New Carrollton, Maryland, where the officials
responsible for IRDM are located. To assess the reliability of the cost
estimate data that we used to support findings in this report, we reviewed
relevant program documentation, such as cost estimation spreadsheets,
as available, to substantiate evidence obtained from interviews with
knowledgeable agency officials. We found the data we used to be
sufficiently reliable for the purposes of our report. As appropriate, we
attributed the sources of the data. We are making recommendations to
IRS to improve data reliability in the future.

We conducted this performance audit from August 2011 through January
2012 in accordance with generally accepted government auditing
standards. Those standards require that we plan and perform the audit to
obtain sufficient, appropriate evidence to provide a reasonable basis for
our findings and conclusions based on our audit objectives. We believe
that the evidence obtained provides a reasonable basis for our findings
and conclusions based on our audit objectives.




Page 34                                             GAO-12-59 IRS Management
Appendix II: Assessment of MITS’s Current
                                            Appendix II: Assessment of MITS’s Current
                                            IRDM Cost Estimate



IRDM Cost Estimate

                                            The following table outlines our assessment of the extent to which the
                                            Internal Revenue Service’s (IRS) 2011 Information Reporting and
                                            Document Matching (IRDM) program cost estimate meets best practices,
                                            depicted in figures 1-4.

Table 3: MITS IRDM Cost Estimate Alignment with Best Practices

 Best practices                    Overall                  Assessment of whether
 characteristics                   assessment               best practices met               Effect
 A comprehensive cost              Partially meets best
 estimate:                         practices for a
                                   comprehensive cost
                                   estimate.
 Includes all life-cycle costs.                             The estimate includes costs      A life cycle cost estimate should
 A life-cycle cost estimate                                 through FY 2014, but IRS         encompass all past, present, and
 provides a complete and                                    plans for the project to         future costs for every aspect of the
 structured accounting of all                               continue through FY 2016,        program, regardless of funding
 resources and associated cost                              and detailed costs are only      source, including all government and
 elements required to develop,                              provided for FY 2012.            contractor costs. Life-cycle cost
 produce, deploy, and sustain a                             (Minimally meets.)               estimates that include all costs can
 particular program. It should                                                               enhance program managers’ decision
 cover the inception of the                                                                  making by allowing them to evaluate
 program through its retirement.                                                             design trade off studies on a total
                                                                                             cost basis as well as on a technical
                                                                                             and performance basis.
      Completely defines the                                The estimate reflects the        Understanding the program—
      program, reflects the                                 current project schedule and     including the acquisition strategy,
      current schedule, and is                              contains high-level              technical definition, characteristics,
      technically reasonable.                               information about technical      system design features, and
 The cost estimate should be                                specifications but lacks         technologies to be included—is key to
 based on a documented                                      details that would completely    developing a credible cost estimate.
 technical baseline description,                            define the program. (Partially   Without these data, the cost estimator
 which provides a common                                    meets.)                          will not be able to identify the
 definition of the program,                                                                  technical and program parameters
 including detailed technical,                                                               that will bind the cost estimate.
 program, and schedule
 descriptions of the system.




                                            Page 35                                                     GAO-12-59 IRS Management
                                          Appendix II: Assessment of MITS’s Current
                                          IRDM Cost Estimate




Best practices                     Overall                Assessment of whether
characteristics                    assessment             best practices met              Effect
     Has a product-oriented                               Each of the four IRDM           A WBS provides a necessary
     work breakdown structure                             projects has a product-         framework for the program to develop
     (WBS), traceable to the                              oriented WBS, but the WBSs      a schedule and cost plan that can
     program’s technical scope,                           are not consistent and they     easily track technical
     at an appropriate level of                           are not traceable to the cost   accomplishments. A standard,
     detail.                                              estimate. (Partially meets.)    product-oriented WBS facilitates the
A WBS provides a basic                                                                    tracking of resource allocations and
framework for a variety of                                                                expenditures, which can give the
related activities like estimating                                                        agency insight to reliably estimate the
costs, developing schedules,                                                              cost of future similar programs.
identifying resources and
potential risks, and providing
the means for measuring
program status using EVM. It is
product-oriented if it allows a
program to track cost and
schedule by defined
deliverables, such as a
hardware or software
component.
     Documents all cost-                                  No ground rules were            Unless ground rules and assumptions
     influencing ground rules                             documented. Several             are clearly documented, the cost
     and assumptions.                                     documents included              estimate will not have a basis for
Cost estimates are typically                              assumptions, but none           assessing potential risks.
based on limited information                              discussed associated risks.     Furthermore, the estimate cannot be
and therefore need to be bound                            (Minimally meets.)              reconstructed when the original
by ground rules and                                                                       estimators are no longer available.
assumptions. Ground rules are
a set of estimating standards
that provide guidance and
common definitions, while
assumptions are judgments
about past, present, or future
conditions that may affect the
estimate. Any risks associated
with assumptions should be
identified and traced to specific
WBS elements.
A well documented cost          Minimally meets
estimate should:                best practices for a
                                well documented
                                cost estimate.




                                          Page 36                                                    GAO-12-59 IRS Management
                                        Appendix II: Assessment of MITS’s Current
                                        IRDM Cost Estimate




Best practices                   Overall                Assessment of whether
characteristics                  assessment             best practices met               Effect
    Capture the source data                             Data sources are listed, but     Data are the foundation of every cost
    used, the reliability of the                        the estimate is not consistent   estimate. Depending on data quality,
    data, and how the data                              with the source data that it     the estimate can range anywhere
    were made compatible                                cites bringing into question     from a mere guess to a highly
    with other data in the                              the reliability of the data.     defensible cost position. Data are
    estimate.                                           (Minimally meets.)               often in many different forms and
Data should be collected from                                                            need to be adjusted before being
primary sources. The source,                                                             used. The cost estimator needs
content, time, and units should                                                          information about the source and
be adequately documented.                                                                reliability of the data in order to know
Further, data should be                                                                  whether the data collected can be
analyzed to determine                                                                    used directly or need to be modified.
accuracy and reliability, and to
identify cost drivers.
    Describe the calculations                           Documentation does not fully     Poorly documented cost estimates
    and the methodology used                            explain how IRS derived          can cause a program’s credibility to
    to derive each element’s                            estimated costs. (Partially      suffer because the documentation
    cost.                                               meets.)                          cannot explain the rationale of the
Documentation should describe                                                            methodology or the calculations.
what calculation methods are                                                             Estimates that lack sufficient
used, as well as how they were                                                           documentation are not useful for
applied, and explain any                                                                 updates or information sharing and
anomalies.                                                                               can hinder understanding and proper
                                                                                         use.
     Describe how the estimate                          Labor cost calculations are      Without good documentation,
     was developed.                                     described at a high level, but   management and oversight
The data supporting the                                 the staffing level is not        organizations will not be convinced
estimate should be available                            traceable to the schedule.       that the estimate is credible;
and adequately documented so                            Hardware and software            supporting data, lessons learned, and
that the estimate can be easily                         calculations are not             reasons why costs changed will not
updated to reflect actual costs                         described, and costs are not     be available for future use; questions
or program changes.                                     consistent across                about the approach or data used to
                                                        documents. (Minimally            create the estimate cannot be
                                                        meets.)                          answered; and the scope of the
                                                                                         analysis cannot be thoroughly
                                                                                         defined.
     Discuss the technical                              Documentation links              Because the technical baseline is
     baseline description.                              hardware and software            intended to serve as the basis for
A technical baseline description                        specifications to costs, but     developing a cost estimate, it should
provides a common definition                            project-specific costs are       be discussed in the cost estimate
of the program, including                               only provided for FY 2012.       documentation. Without a technical
detailed technical, program,                            (Minimally meets.)               baseline, the cost estimate will not be
and schedule descriptions of                                                             based on a comprehensive program
the system, for a cost estimate                                                          description and will lack specific
to be built on. The data in the                                                          information regarding technical and
technical baseline should be                                                             program risks.
consistent with the cost
estimate.




                                        Page 37                                                     GAO-12-59 IRS Management
                                             Appendix II: Assessment of MITS’s Current
                                             IRDM Cost Estimate




Best practices                     Overall                   Assessment of whether
characteristics                    assessment                best practices met              Effect
     Provide evidence of                                     The estimate was developed      A cost estimate is not considered
     management review and                                   by IRDM managers, not cost      valid until management has approved
     acceptance.                                             estimators, and while it was    it. It is imperative that management
There should be a briefing to                                reviewed within the IRDM        understand how the estimate was
management, including a clear                                program, there is no            developed, including the risks
explanation of how the cost                                  evidence of review by the top   associated with the underlying data
estimate was derived.                                        IRDM managers. (Minimally       and methods.
Management’s acceptance of                                   meets.)
the cost estimate should be
documented.
An accurate cost estimate:         Minimally meets
                                   best practices for
                                   an accurate cost
                                   estimate.
      Produces unbiased results.                             No confidence levels are        A cost estimate is biased if the
Cost estimates should have an                                listed and documents do not     estimated work is overly conservative
uncertainty analysis, which                                  provide a range of possible     or too optimistic. Unless the estimate
determines where the estimate                                costs. Neither risk nor         is based on an assessment of the
falls against the range of all                               uncertainty is mentioned.       most likely costs and reflects the
possible costs.                                              (Does not meet.)                degree of uncertainty given all of the
                                                                                             risks considered, management will
                                                                                             not be able to make informed
                                                                                             decisions.
      Is properly adjusted for                               Documentation contains no       Adjusting for inflation is important
      inflation.                                             evidence that the cost          because in the development of an
Cost data should be adjusted                                 estimate is adjusted for        estimate, cost data must be
for inflation to ensure that                                 inflation. (Does not meet.)     expressed in like terms. If a mistake
comparisons and projections                                                                  is made or the inflation amount is not
are valid. Data should also be                                                               correct, cost overruns can result.
normalized to constant year
dollars to remove the effects of
inflation. Also, inflation
assumptions must be well
documented.
     Contains few mistakes.                                  Documentation does not          Without access to estimate details,
Results should be checked for                                specify whether the cost        one cannot be certain that
accuracy, double-counting, and                               estimate went through a         calculations are accurate or
omissions. Validating that a                                 quality control process, and    expressed consistently.
cost estimate is accurate                                    there are inconsistencies
requires thoroughly                                          among documents, but
understanding and                                            calculations are accurate.
investigating how the cost                                   (Partially meets.)
model was constructed.




                                             Page 38                                                    GAO-12-59 IRS Management
                                           Appendix II: Assessment of MITS’s Current
                                           IRDM Cost Estimate




Best practices                    Overall                  Assessment of whether
characteristics                   assessment               best practices met              Effect
     Is regularly updated to                               IRS does not maintain an        If a cost estimate is not updated, it
     reflect significant program                           updated IRDM cost estimate      can become more difficult to analyze
     changes.                                              in a single document.           changes in program costs and collect
The cost estimate should be                                However, IRDM costs are         cost and schedule data to support
updated to reflect significant                             tracked through the             future cost estimates. The cost
program changes, such as                                   program’s WBS and EVM           estimate should be updated when the
changes to schedules or other                              system, and IRS evaluates       technical baseline changes;
assumptions. Updates should                                them to develop spend           otherwise, it will lack credibility. A
also reflect actual costs so that                          plans, but we found that the    properly updated cost estimate can
the estimate always reflects the                           EVM data are not reliable.      provide decision makers with
current program status.                                    (Minimally meets.)              accurate information for assessing
                                                                                           alternative decisions.
     Documents and explains                                Variances between planned       Without a documented comparison
     variances between                                     and actual costs, and           between the current estimate
     planned and actual costs.                             explanations for the variance   (updated with actual costs) and the
Variances between planned                                  are documented in EVM           old estimate, cost estimators cannot
and actual costs should be                                 data, but the EVM data are      determine the level of variance
documented, explained, and                                 unreliable (Minimally meets.)   between the two estimates. That is,
reviewed. For any elements                                                                 the estimators cannot see how well
whose actual costs or                                                                      they are estimating and how the
schedules differ from the                                                                  program is changing over time.
estimate, the estimate should
discuss variances and lessons
learned.
    Reflects cost estimating                               Documentation contains no       Historical data provides the cost
    experiences from                                       evidence that the estimate is   estimator with insight into actual costs
    comparable programs.                                   based on a model that uses      on similar programs, including any
The estimate should be based                               historical records of cost      cost growth that occurred after the
on historical cost estimation                              estimating and actual           original estimate. As a result,
data and actual experiences                                experiences from                historical data can be used to
from other comparable                                      comparable programs. (Does      challenge optimistic assumptions and
programs. These data should                                not meet.)                      bring more realism to a cost estimate.
be reliable and relevant to the
new program.
A credible cost estimate          Does not meet best
includes:                         practices for a
                                  credible cost
                                  estimate.




                                           Page 39                                                    GAO-12-59 IRS Management
                                         Appendix II: Assessment of MITS’s Current
                                         IRDM Cost Estimate




Best practices                    Overall                      Assessment of whether
characteristics                   assessment                   best practices met                                Effect
     A sensitivity analysis that                               Documentation contains no                         Because uncertainty cannot be
     identifies a range of                                     evidence that a sensitivity                       avoided, it is necessary to identify the
     possible costs based on                                   analysis was performed.                           cost elements that represent the most
     varying inputs.                                           (Does not meet.)                                  risk. A sensitivity analysis reveals
A sensitivity analysis examines                                                                                  how the cost estimate is affected by a
how changes to key                                                                                               change in a single assumption, which
assumptions and inputs affect                                                                                    helps the cost estimator to
the estimate. The estimate                                                                                       understand which variable(s) most
should identify key cost drivers,                                                                                affects the cost estimate. Any
examine their parameters and                                                                                     sources of variation should be well
assumptions, and re-estimate                                                                                     documented and traceable.
the total cost by varying each
parameter between its
minimum and maximum range.
     A risk and uncertainty                                    Documentation contains no                         For management to make good
     analysis.                                                 evidence that a risk analysis                     decisions, the program estimate must
A risk and uncertainty analysis                                or an uncertainty analysis                        reflect the degree of uncertainty, so
recognizes the potential for                                   were done. (Does not meet.)                       that a level of confidence can be
error and attempts to quantify it                                                                                given about the estimate. An estimate
by identifying the effects of                                                                                    without risk and uncertainty analysis
changing key cost drivers.                                                                                       is unrealistic because it does not
                                                                                                                 assess the variability in the cost
                                                                                                                 estimate from such effects as
                                                                                                                 schedules slipping, missions
                                                                                                                 changing, and proposed solutions not
                                                                                                                 meeting users’ needs.
     Cross-checking of major                                   Documentation contains no                         If a cross-check demonstrates that
     cost elements.                                            evidence that cross-checks                        alternative methods can produce
A cross-check is done by using                                 were performed. (Does not                         similar results, then confidence in the
a different cost estimation                                    meet.)                                            estimate increases, leading to greater
method to see if it produces                                                                                     credibility.
similar results.
     A comparison to an                                        Documentation contains no                         An independent cost estimate is
     independent cost estimate                                 evidence that an                                  considered one of the best and most
     conducted by another                                      independent cost estimate                         reliable methods for validating an
     organization.                                             was performed. (Does not                          estimate. It provides an independent
A second, independent, cost                                    meet.)                                            view of expected program costs that
estimate should be performed                                                                                     tests the program office’s estimate for
by an organization outside of                                                                                    reasonableness. Without an
the program office’s influence.                                                                                  independent cost estimate, decision
It should be based on the same                                                                                   makers will lack insight into a
technical baseline, ground                                                                                       program’s potential costs because
rules, and assumptions, as the                                                                                   independent cost estimates
original estimate.                                                                                               frequently use different methods and
                                                                                                                 are less burdened with organizational
                                                                                                                 bias.
                                         Source: GAO analysis of IRS’s 2011 IRDM cost estimate and GAO 09-3SP.




                                         Page 40                                                                            GAO-12-59 IRS Management
Appendix III: High Level Assessment of the
                                            Appendix III: High Level Assessment of the
                                            Reliability of IRDM’s EVM System



Reliability of IRDM’s EVM System

Table 4: IRDM EVM Data Reliability

Data reliability task                                   Was task met? Explanation of GAO assessment
Maintain an earned value management (EVM)          No                           IRDM did not meet the three ANSI guidelines we assessed.
system that is compliant with the agency’s scaling
                    a
of ANSI guidelines.
•   Work Elements: Define the authorized work      •  No                        •     The WBS used in IRDM’s schedules did not match the
    elements for the program, typically done using                                    WBS used for EVM.
    a WBS.
•   Sequencing: Schedule the authorized work in a •   No                        •     The IRDM schedules did not have proper sequencing;
    manner that describes the sequence of work                                        related project activities were not linked.
    and identifies significant task
    interdependencies.
                                   b
•   Time-phased budget baseline: At the control         •     No                •     IRS was unable to show evidence that it established a
    account level, establish and maintain a time-                                     time-phased budget baseline for IRDM. IRDM does not
    phased budget baseline, against which                                             have one overall schedule, and none of the four IRDM
    program performance can be measured.                                              project schedules were completely resource loaded.
Conduct an integrated baseline review, or               No                      Neither an integrated baseline review nor an independent
                                                  c
independent baseline validation, for the program.                               baseline validation were conducted for IRDM.
Using independent and qualified staff, conduct          No                      Surveillance on the IRDM EVM system was not conducted and
                                d
surveillance on the EVM system.                                                 IRS does not have staff with sufficient technical skills to
                                                                                conduct surveillance.
                                            Sources: GAO analysis of IRDM EVM data, WBSs, and project schedules, as well as OMB and Treasury guidance and GAO-09-3SP.
                                            a
                                             The American National Standards Institute (ANSI) has a national EVM standard, comprised of 32
                                            guidelines, that enables agencies to evaluate an EVM system to determine if cost, schedule, and
                                            technical performance data can be relied on for program management.
                                            b
                                             The time-phased budget baseline, against which performance is measured, is formed from the
                                            performance measurement baseline, which is essentially the resource consumption plan for the
                                            program. Deviations from the baseline identify areas where management should focus attention. A
                                            performance measurement baseline represents the cumulative value of a program’s planned work
                                            over time. This baseline takes into account the program activities that occur in a sequenced order,
                                            based on finite resources, with budgets representing those resources spread over time.
                                            c
                                              An integrated baseline review is an assessment done by program management and contractors to
                                            verify that the program baseline is adequate and realistically portrays all authorized work according to
                                            the schedule. For a project of IRDM’s size, guidance from the Department of the Treasury states that
                                            an independent baseline validation, which appears to be a less rigorous version of an integrated
                                            baseline review, can be performed.
                                            d
                                             Surveillance is reviewing a contractor’s EVM system to observe ANSI compliance and how well a
                                            contractor is using its EVM system to manage cost, schedule, and technical performance.




                                            Page 41                                                                              GAO-12-59 IRS Management
Appendix IV: Comments from the Internal
              Appendix IV: Comments from the Internal
              Revenue Service



Revenue Service




              Page 42                                   GAO-12-59 IRS Management
Appendix IV: Comments from the Internal
Revenue Service




Page 43                                   GAO-12-59 IRS Management
Appendix V: GAO Contact and Staff
                  Appendix V: GAO Contact and Staff
                  Acknowledgments



Acknowledgments

                  Michael Brostek, (202) 512-9110 or brostekm@gao.gov
GAO Contact
                  In addition to the contact named above, Libby Mixon, Assistant Director;
Staff             Laurel Ball; Amy Bowser; Bill Cordrey; Jennifer Echard; Robert Gebhart;
Acknowledgments   Paul Middleton; Donna Miller; Sabine Paul; Karen Richey; Stacy Steele;
                  and Lindsay Swenson made key contributions to this report.




(450944)
                  Page 44                                            GAO-12-59 IRS Management
GAO’s Mission         The Government Accountability Office, the audit, evaluation, and
                      investigative arm of Congress, exists to support Congress in meeting its
                      constitutional responsibilities and to help improve the performance and
                      accountability of the federal government for the American people. GAO
                      examines the use of public funds; evaluates federal programs and
                      policies; and provides analyses, recommendations, and other assistance
                      to help Congress make informed oversight, policy, and funding decisions.
                      GAO’s commitment to good government is reflected in its core values of
                      accountability, integrity, and reliability.

                      The fastest and easiest way to obtain copies of GAO documents at no
Obtaining Copies of   cost is through GAO’s website (www.gao.gov). Each weekday afternoon,
GAO Reports and       GAO posts on its website newly released reports, testimony, and
                      correspondence. To have GAO e-mail you a list of newly posted products,
Testimony             go to www.gao.gov and select “E-mail Updates.”

Order by Phone        The price of each GAO publication reflects GAO’s actual cost of
                      production and distribution and depends on the number of pages in the
                      publication and whether the publication is printed in color or black and
                      white. Pricing and ordering information is posted on GAO’s website,
                      http://www.gao.gov/ordering.htm.
                      Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
                      TDD (202) 512-2537.
                      Orders may be paid for using American Express, Discover Card,
                      MasterCard, Visa, check, or money order. Call for additional information.
                      Connect with GAO on Facebook, Flickr, Twitter, and YouTube.
Connect with GAO      Subscribe to our RSS Feeds or E-mail Updates. Listen to our Podcasts.
                      Visit GAO on the web at www.gao.gov.
                      Contact:
To Report Fraud,
Waste, and Abuse in   Website: www.gao.gov/fraudnet/fraudnet.htm
                      E-mail: fraudnet@gao.gov
Federal Programs      Automated answering system: (800) 424-5454 or (202) 512-7470

                      Katherine Siggerud, Managing Director, siggerudk@gao.gov, (202) 512-
Congressional         4400, U.S. Government Accountability Office, 441 G Street NW, Room
Relations             7125, Washington, DC 20548

                      Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800
Public Affairs        U.S. Government Accountability Office, 441 G Street NW, Room 7149
                      Washington, DC 20548




                        Please Print on Recycled Paper.