Flash Audit Alert - U.S. Office of Personnel Management's Infrastructure Improvement Project

Published by the Office of Personnel Management, Office of Inspector General on 2015-06-17.

Below is a raw (and likely hideous) rendition of the original report. (PDF)


                                                        Washington, DC 20415 

  Office of the                                         June 17, 2015
Inspector General



        FROM: 	                        PATRICKE.McFARLANDt:j?~~ (. ?IJ~L. .J
                                       Inspector General                    f/ ~ ~ ~'~ 	                      ..........-\ 

        SUBJECT: 	                     Flash Audit Alert - U.S. Office of Personnel Management's
                                       Infrastructure Improvement Project (Report No. 4A-CI-00-15-055)

        Executive Summary

        The U.S. Office of Personnel Management (OPM) Office ofthe Inspector General (OIG) is
        issuing this Flash Audit Alert to bring to your immediate attention serious concerns we have
        regarding the Office of the Chief Information Officer' s (OCIO) infrastructure improvement
        project (Project). 1 This Project includes a full overhaul ofthe agency's technical infrastructure
        by implementing additional information technology (IT) security controls and then migrating the
        entire infrastructure into a completely new environment (referred to as Shell).

        Our primary concern is that the OCIO has not followed U.S . Office of Management and Budget
        (OMB) requirements and project management best practices. The OCIO has initiated this project
        without a complete understanding of the scope of OPM' s existing technical infrastructure or the
        scale and costs of the effort required to migrate it to the new environment.

        In addition, we have concerns with the nontraditional Government procurement vehicle that was
        used to secure a sole-source contract with a vendor to manage the infrastructure overhaul. While
        we agree that the sole-source contract may have been appropriate for the initial phases of
        securing the existing technical environment, we do not agree that it is appropriate to use this
        vehicle for the long-term system migration efforts.

        We intend to conduct further oversight of this Project and may issue additional reports in the
        future. However, we have identified substantial issues requiring immediate action, and we are
        therefore issuing the following recommendations in this Flash Audit Alert, so that the OCIO can
        immediately begin taking steps to address these concerns. We provided a draft of this Alert to
        the OCIO for their review, but we did not receive any comments.

         This audit report has been d istributed to Federal officials who are responsible for the adm inistration of the audited
        program. This audit report may contain proprietary data which is protected by Federal law ( 18 U .S.C. 1905).
        Therefore, while this audit report is available under the Freedom of Information Act and made available to the
        public on the OIG webpage (http://www.opm.gov/our-inspector-general), caution needs to be exercised before
        releas ing the report to the general public as it may contain proprietary information that was redacted from the
        publicly distributed copy.

Honorable Katherine Archuleta                                                                        2

1) 	 Project Management Activities

   We were told that OPM officials initiated the Project to improve the security of its network
   and operating environment after learning of a significant security incident in March 2014.
   The initial plan was to make major security improvements to the existing environment and
   continue to operate OPM systems in their current location. During the process of
   implementing security upgrades, OPM determined that it would be more effective to
   completely overhaul the agency's IT infrastructure and architecture and move it into a
   completely new environment.

   The new plan involves hosting OPM systems in two commercial data centers. The new
   architecture will be a distributed computing environment, with no mainframe or legacy
   applications. We have been told by OCIO officials that no applications will be allowed to
   migrate to the new Shell environment unless they are rebuilt to be compatible with all new
   security and operating features of the new architecture. The phases of this Project include
   Tactical (shoring up the existing security environment), Shell (creating the new data center
   and IT architecture), Migration (migrating all OPM systems to the new architecture), and
   Cleanup (decommissioning existing hardware and systems). The current status is that the
   Tactical phase is complete, and the Shell phase is underway.

   While we agree in principle that this is an ideal future goal for the agency's IT environment,
   we have serious concerns regarding OPM's management ofthis Project. The Project is
   already underway and the agency has committed substantial funding, but it has not yet
   addressed several critical project management requirements, including, but not limited to:

   • 	 OPM has not yet identified the full scope and cost ofthis project;
   • 	 OPM has not prepared a 'Major IT Business Case' (formerly known as the OMB Exhibit
       300), as required by OMB for IT projects ofthis size and scope; and,
   • 	 OPM's overall project management process is missing a number of critical artifacts
       considered to be best practices by relevant organizations.

   As a result, there is a high risk that this Project will fail to meet the objectives of providing a
   secure operating environment for OPM systems and applications.

   Many critical OPM applications (including those that process annuity payments for Federal
   retirees, reimburse health insurance companies for claims payments, and manage background
   investigations) run on OPM's mainframe computers. These applications are based on legacy
   technology, and will need to be completely renovated to be compatible with OPM's proposed
   new IT architecture.

   To help put this in perspective, we reference OPM's Fiscal Year (FY) 2009 efforts to migrate
   a single financial system application from the mainframe. This project was relatively well
   managed and was subject to oversight from several independent entities, including the OIG,
   but it still required two years and over $30 million to complete. OPM's current initiative is
   much more massive than this prior project, as each individual application migration should
Honorable Katherine Archuleta                                                                    3

   be treated as its own project similar to this example. Furthermore, there are many other
   systems besides OPM's mainframe applications that will also need to be modified to some
   extent to be compatible.

   The Migration phase of this Project will clearly be a complex, expensive, and lengthy
   process. OPM currently estimates that it will take 18 to 24 months to complete. We believe
   this is overly optimistic and that the agency is highly unlikely to meet this target. In fact,
   OPM is still in the process of evaluating its existing IT architecture, including the
   identification of all mainframe applications that will need to be migrated, and other systems
   that will need to be redesigned. OCIO representatives are currently conducting a
   compatibility assessment for the "major OPM investments" as encompassed by three
   program offices: Retirement Services, Federal Investigative Services, and Human Resources
   Solutions. It was explained to us that this review only addresses approximately 80 percent of
   OPM ' s systems, with the remainder considered out of scope for this evaluation, but to be
   eventually addressed. This assessment is not scheduled for completion until next month
   (July 2015). It is difficult to see how the agency can estimate its timeline when it does not
   yet know the scope of the effort.

   Related to the unknown scope ofthe Project is the uncertainty of its overall cost. OPM has
   estimated that the Tactical and Shell phases of the Project will cost approximately $93
   million. OMB has included $21 million in the President's FY 2016 budget to fund part of
   this amount. Another $5 million was contributed by the Department of Homeland Security
   to support its continuous monitoring program, and the remaining $67 million is being
   collected from OPM's major program offices as a special assessment. However, this
   estimate does not include the costs to migrate the many existing applications to the new IT
   environment, which are likely to be substantial.

   When we asked about the funding for the Migration phase, we were told, in essence, that
   OPM would find the money somehow, and that program offices would be required to fund
   the migration of applications that they own from their existing budgets. However, program
   office budgets are intended to fund OPM' s core operations, not subsidize a major IT
   infrastructure project. It is unlikely that OPM will be able to fund the substantial migration
   costs related to this Project without a significantly adverse impact on its mission, unless it
   seeks dedicated funding through Congressional appropriation. Also, OPM's current budget
   approach seems to violate IT spending transparency principles promoted by OMB's budget
   guidance and its IT Dashboard initiative, which is intended to "shine [a] light onto the
   performance and spending of IT investments across the Federal Government."

   In addition to the undefined scope and uncertain budget, OPM has not completed other
   standard, and critical, project management steps. Control Objectives for Information and
   Related Technology (COBIT) is a framework created by the Information System Audit and
   Control Association (ISACA) for IT management and IT governance. The Committee of
   Sponsoring Organizations of the Treadway Commission (COSO) framework also identifies
   internal controls required for effective organizational management.
Honorable Katherine Archuleta                                                                     4

   COBIT and the COSO framework define best practices for major IT developments. Several
   examples of critical processes that OPM has not completed for this project include:

   • 	 Project charter;
   • 	 Comprehensive list of project stakeholders;
   • 	 Feasibility study to address scope and timeline in concert with budgetary
       justification/cost estimates;
   • 	 Impact assessment for existing systems and stakeholders;
   • 	 Quality assurance plan and procedures for contractor oversight;
   • 	 Technological infrastructure acquisition plan;
   • 	 High-level test plan; and,
   • 	 Implementation plan to include resource planning, readiness assessment plan, success
       factors, conversion plan, and back-out plan.

   In our opinion, the project management approach for this major infrastructure overhaul is
   entirely inadequate, and introduces a very high risk of project failure . The correct approach
   would be to use the OMB budget process to request project funding using the OMB-required
   Major IT Business Case (Exhibit 300) process. This would require OPM to fully evaluate the
   costs, benefits, and risks associated with its planned Project, and present its business case to
   OMB to seek approval and funding.

   OMB Circular A-ll Appendix 6 defines capital budgeting requirements for capital asset
   projects. The basic concepts are that capital asset projects require proper planning,
   cost/benefit analysis, financing, and risk management. This includes demonstrating that the
   return on investment exceeds the cost of funds used, and an analysis of the "investment' s
   total life-cycle costs and benefits, including the total budget authority required for the
   asset . .. "

   Furthermore, the financing principles outlined in the Circular state that " Good budgeting
   requires that appropriations for the full cost of asset acquisition be enacted in advance to help
   ensure that all costs and benefits are fully taken into account at the time decisions are made
   to provide resources."

   Finally, the Circular requires risk management and earned value management throughout the
   life-cycle of the project: "The investment cost, schedule, and performance goals established
   through the Planning Phase ofthe investment are the basis for approval to procure the asset
   and the basis for assessing risk. During the Procurement Phase, performance-based
   management systems (earned value management system) must be used to provide contractor
   and Government management visibility on the achievement of, or deviation from, goals until
   the asset is accepted and operational."

   OMB ' s FY 2016 IT Budget - Capital Planning Guidance further states that "Together, the
   Major IT Business Cases and Major IT Business Case Details provide the budgetary and
   management information necessary for sound planning, management, and governance of IT
   investments. These documents help agencies explicitly align IT investments with strategic
   and performance goals, and ultimately provide value to the public by making investment and
Honorable Katherine Archuleta                                                                      5

   management information more transparent." OMB expects that artifacts, documents, and
   associated data similar to those defined by the COBIT and COSO frameworks already exist
   when a Major IT Business Case is submitted as part of an agency's budget process.

   OPM officials informed us that the urgent and compelling nature of the situation required
   immediate action, and this is the reason that some of the required project management
   activities were not completed. We agree with and support the agency' s efforts to improve its
   IT security infrastructure through the Tactical phase of this Project. We understand and
   accept that immediate action was required and that it was appropriate to do so. However, the
   other phases ofthe project are clearly going to require long-term effort, and, to be successful,
   will require the disciplined processes associated with proper system development project

   Without these disciplined processes, there is a high risk that this Project will fail to meet all
   of its stated objectives. In addition, without a guaranteed source of funding in place, OPM
   may not have the internal resources necessary to complete the Migration phase, which is
   likely to be complex and expensive. In this scenario, the agency would be forced to
   indefinitely support multiple data centers, further stretching already inadequate resources,
   possibly making both environments less secure, and increasing costs to taxpayers. This
   outcome would be contrary to the stated goals of creating a more secure IT environment at a
   lower cost.

Recommendation 1

We recommend that OPM's OCIO complete an OMB Major IT Business Case document as part
of the FY 2017 budget process and submit this document to OMB for approval. Associated with
this effort, the OCIO should complete its assessment of the scope of the migration process, the
level of effort required to complete it, and its estimated costs. Furthermore, the OCIO should
implement the project management processes required by OMB and recommended by ISACA' s
COBIT and the COSO framework.

2) Sole-Source Contract

   OPM has secured a sole-source contract with a vendor to manage the infrastructure
   improvement project from start to finish. Although OPM completed a Justification for Other
   Than Full and Open Competition (JOFOC) to justify this contract, we do not agree that it is
   appropriate to use this contract for the entire Project.

   The initial phase of the Project covered the procurement, installation, and configuration of a
   variety of software tools designed to improve the IT security posture of the agency (the
   Tactical phase). We agree that recent security breaches at OPM warranted a thorough and
   immediate reaction to secure the existing environment, and that the JOFOC was appropriate
   for this tactical activity.

   However, the JOFOC also covered subsequent phases of the Project related to the
   development ofthe new Shell infrastructure, the migration of all ofOPM' s applications into
Honorable Katherine Archuleta                                                                     6

   this new environment, and decommissioning the old environment. Although the Shell phase
   is largely complete, there is still an opportunity to procure contractor support for the
   migration and cleanup phases ofthis project using the appropriate contracting vehicles.
   Without submitting this Project to an open competition, OPM has no benchmark to evaluate
   whether the costs charged by the sole-source vendor are reasonable and appropriate.

   As stated previously, we expect the Migration phase to be extremely complex and time
   consuming. It will likely require significant contractor support, with each application
   requiring a unique skill set. OPM may also determine that it would benefit from a contractor
   to oversee the Migration effort as a whole. We believe that contractor support for both
   application-specific migration and the Migration and Cleanup efforts as a whole are not
   justifiably covered by the existing sole-source contract. FAR 6.302 outlines seven scenarios
   where contracting without full and open competition may be appropriate, two of which relate
   to an unusual and compelling urgency and national security implications. However, we have
   not been provided evidence that the Migration and Cleanup phases ofthis project meet the
   FAR criteria for bypassing an open competition.

   We believe that OPM should gain a complete and thorough understanding ofthe scope of
   this Project, request funding from OMB via the appropriate avenues (See Recommendation
   1) and then subject the remainder of the project to contracting vehicles other than the sole
   source contract used for the Tactical and Shell phases.

Recommendation 2

We recommend that OPM not leverage its existing sole source contract for the Migration and
Cleanup phases of the infrastructure improvement project. Contractor support for these phases
should be procured using existing contracts already supporting legacy information systems or via
full and open competition.

If you have any questions about this Flash Audit Alert you can contact me, at 606-1200, or your
staff may wish to contact Michael R. Esser, Assistant Inspector General for Audits, at 606-2143.

cc: 	 Chris Canning
      Acting Chief of Staff

     Angela Bailey
     Chief Operating Officer

     Janet Barnes
     Director, Internal Oversight and Compliance

     Donna K. Seymour 

     Chief Information Officer