oversight

Y2K Computing Challenge: Day One Planning and Operations Guide

Published by the Government Accountability Office on 1999-10-01.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                   United States General Accounting Office
                   Accounting and Information
GAO                Management Division



October 1999
                   Y2K Computing Challenge:
                   Day One Planning and
                   Operations Guide




                          G A O
                         Accountability ✮ Integrity ✮ Reliability


GAO/AIMD-10.1.22
Preface

Federal agencies are making significant progress racing against time and renovating, validating,
and implementing their mission-critical information systems. Nevertheless, organizations
remain vulnerable to Y2K disruptions. Because most federal agencies are highly dependent on
information technology to carry out their missions, Year 2000-induced failures of one or more
critical systems may have an adverse impact on an organization’s ability to deliver critical
services.

The risk of failure is not limited to an organization’s internal information systems. Many federal
agencies depend on information and data provided by their business partners—including other
federal agencies, hundreds of state and local agencies, private-sector entities, and international
organizations. Finally, every organization depends on services provided by the public
infrastructure—including power, water, transportation, and voice and data telecommunications.

In August 1998, we published a guide to help agencies address business continuity and
contingency planning issues.1 The guide provides a conceptual framework for managing and
mitigating the risks of potential Year 2000-induced disruptions to agency operations. It also
calls for agencies to develop a “day zero” strategy, also known as a “Day One” strategy, to
manage the critical century rollover period.2

A Day One strategy comprises a comprehensive set of actions to be executed by a federal
agency during the last days of 1999 and the first days of 2000. It must be integrated with
agency business continuity and contingency plans, and should describe the key activities and
responsibilities of agency component organizations and staff.

Day One Operations
                 Pre-rollover                                         Post-rollover
              December 1999                                           January 2000

         28         29          30         31           1         2          3          4          5

        Tue        Wed        Thur        Fri        Sat        Sun        Mon         Tue       Wed


                                             Rollover
                                             Typical Day One Operations


The objectives of a Day One strategy are to (1) position an organization to readily identify Year

1Year   2000 Computing Crisis: Business Continuity and Contingency Planning (GAO/AIMD-10.1.19, issued as
an exposure draft in May 1998; issued final in August 1998). The guide is available at
<http://www.gao.gov/special.pubs/bcpguide.pdf>.
2Agen cies sh ould also be pr epar ed to deal with laten t post-r ollover pr oblems th at may emer ge days or mon th s
later, and with other key date-sensitive events such as the leap year.



GAO/AIMD-10.1.22 Year 2000 Day One Planning and Operations Guide                                                     1
2000-induced problems, take needed corrective actions, and minimize adverse impact on
agency operations and key business processes, and (2) provide information about an
organization’s Year 2000 condition to executive management, business partners, and the public.

The guide provides a conceptual framework for helping agencies develop a Day One strategy
and reduce the risk of adverse Y2K impact on agency operations. It builds upon our previously
issued Year 2000 business continuity and contingency planning guide, and draws on other
sources, including the Social Security Administration, International Business Machines
Corporation, and the Legislative Branch Y2K Group.

Because each agency is different, there is no single, cookie-cutter approach to Day One
planning. Some agencies are highly centralized, while others operate in a highly decentralized
environment. This guide addresses issues that will be common to most agencies; however, each
agency must tailor its Day One plan in response to its unique needs.

The guide addresses four phases supported by executive oversight:

     •   Initiation
     •   Rollover risk assessment, planning, and preparation
     •   Rehearsal
     •   Execution, monitoring, responding, and reporting

In addition to executive oversight, the four phases are united by the common thread of
accountability at all levels.

An electronic version of this guide is available from GAO’s World Wide Web at
<www.gao.gov/special.pubs/dayone.pdf>. If you have any comments or questions about the
guide, please contact us, or Mirko J. Dolak, Technical Assistant Director, at (202) 512-6362;
or E. Randolph Tekeley, Technical Assistant Director, at (202) 512-4070. We can also be
reached by e-mail at willemssenj.aimd@gao.gov, rhodesk.aimd@gao.gov,
dolakm.aimd@gao.gov, and tekeleye.aimd@gao.gov.




Joel C. Willemssen                                    Keith A. Rhodes
Director                                              Director
Civil Agencies Information Systems                    Office of Computer and Information
                                                      Technology Assessment




GAO/AIMD-10.1.22 Year 2000 Day One Planning and Operations Guide                                2
Contents




____________________________________________________________________________
Day One and the Year 2000 Problem                                          4




____________________________________________________________________________
Initiation                                                                 5




____________________________________________________________________________
Rollover Risk Assessment, Planning, and Preparation                        7




____________________________________________________________________________
Rehearsal                                                                 11




____________________________________________________________________________
Execution, Monitoring, Responding, and Reporting                          12




____________________________________________________________________________
Glossary                                                                  15




GAO/AIMD-10.1.22 Year 2000 Day One Planning and Operations Guide           3
Day One and the Year 2000

Although agencies are making significant progress in renovating and testing their mission-
critical systems, crossing the century boundary will nevertheless present many challenges. To
address these challenges, each agency should develop a Day One strategy for reducing risk to
its facilities, systems, programs, and services during the weekend of the critical century rollover.
Such a strategy should focus on actions to be taken shortly before, during, and after the
rollover.

This guide presents a structured approach to aid federal agencies in Day One planning and
management. The guide draws on (1) the Day One plan of the Social Security Administration,
(2) the rollover guidance3 developed by the International Business Machines Corporation, and
(3) the Day One Guide drafted by the Legislative Branch Y2K Group. It describes four phases
--supported by agency executive management--with each phase representing a major Day One
planning project activity or segment.

Year 2000 Day One Planning and Operations Structure




                                                Establish a Day One project work group and develop an
                               Initiation       overall Day One planning strategy. Develop master schedule
                                                and milestones and obtain executive support.




                             Rollover Risk       Assess the risk of internal and external Y2K failures.
                              Assessment,        Develop Day One plans and establish command center(s).
                                                 Identify activities and processes for the pre-rollover and
                               Planning,
                                                 post-event periods. Develop communications plan.
       Executive             Preparation
       Oversight

                                                Define and document rehearsal plans and rehearse selected
                               Rehearsal        Day One operations and teams. Update and revise Day One
                                                plan as needed.



                              Execution,        Execute rollover procedures and tests. Identify and resolve
                                                problems, report status.
                             Monitoring,
                             Responding,
                              Reporting




3Plan n in g for
            the 1999 to 2000 Rollover of IT Systems, Year 2000 Global Initiatives, International Business
Machines Corporation, June 1999. The guide is available at
<www.ibm.com/ibm/year2000/docs/rollover/rollover_english.pdf>.


GAO/AIMD-10.1.22 Year 2000 Day One Planning and Operations Guide                                              4
1.0 Initiation

Executive management needs to be fully aware that the century rollover period can pose risks
to an agency’s ability to deliver services. Executives responsible for the agencies’ core business
processes should work with the Chief Information Officer, the Chief Financial Officer, and the
Year 2000 program manager to develop a Day One strategy. Agency executives must dedicate
sufficient resources and staff for this task, and ensure that senior managers support this effort.

 Key Tasks

 1.1 Establish a Day One project work group, define roles, and assign responsibility
 1.2 Develop and document a high-level Day One planning strategy, and establish schedule
     and milestones
 1.3 Establish Day One leave and compensation policy
 1.4 Establish or activate mechanisms for planned reviews of Day One activities

1.1   Establish a Day One project work group, define roles, and assign responsibility

      Establish a Day One work group. The group should report to executive management
      and work closely or be aligned with the business continuity and contingency planning
      group. This group should also include representatives from the IRM office, major
      business units, and field organizations.

      Manage the Day One planning tasks and activities as a sub-project. Define roles and
      assign responsibilities for leading the planning effort and for performing analyses and
      developing Day One procedures and plans.

      Determine what major business units and field organizations need to develop Day One
      plans. The principal candidates are organizational components responsible for key
      business processes—usually components that were required to develop individual
      business continuity and contingency plans, and large field organizations. Appoint
      individuals to lead the development of Day One plans for each of the designated
      organizational components.

1.2   Develop and document a high-level Day One planning strategy, and establish schedule and
      milestones

      A high-level Day One planning strategy provides the agency’s executive management
      with an overview of key activities. The strategy should address the project management
      structure, the number and location of information coordination and/or command centers,
      the project’s relationship with the business continuity and contingency planning effort,
      metrics and reporting requirements, and initial cost and schedule estimates.
      Develop a schedule and milestones for the planning effort. Ensure time for rehearsing



GAO/AIMD-10.1.22 Year 2000 Day One Planning and Operations Guide                                 5
      the key teams needed to carry out each phase and function of the strategy.

      Coordinate Day One planning with the agency’s business continuity and contingency
      planning effort, and ensure that the Day One plan incorporates appropriate procedures
      and processes identified in the agency’s continuity of operations plan.

1.3   Establish Day One leave and compensation policy

      Address rollover leave and compensation issues. Work with human resources
      departments to determine special compensation requirements to deal with unused annual
      leave, overtime, holiday work, and standby. Consult the Office of Personnel
      Management for further guidance at <www.opm.gov/y2k/index.htm>.

1.4   Establish or activate mechanisms for planned reviews of Day One activities

      Establish or activate mechanisms to review the Day One planning process and strategy
      for omissions, feasibility, and key assumptions. Use vehicles such as quality assurance
      office staff to ensure that the business resumption teams established during the business
      continuity and contingency planning process are ready, and that the individual business
      continuity plans were adequately tested.




GAO/AIMD-10.1.22 Year 2000 Day One Planning and Operations Guide                                  6
2.0 Rollover Risk Assessment, Planning, and Preparation

Day One planning integrates and acts on the results of the rollover risk assessment. The output
of this process is a Day One plan for headquarters and--if required--for each of the designated
organizational units responsible for key business processes. The plan should focus on the key
Day One functions (monitoring, responding, and reporting) implemented within the framework
of the pre-rollover and post-rollover phases. Each plan should provide a description of the
activities, resources, staff roles, and timetables needed for its execution.


 Key Tasks

 2.1    Assess failure scenarios documented in business continuity and contingency plans
        and identify key risk areas, including risks faced by business partners
 2.2    Establish information coordination and/or command center(s)
 2.3    Define the pre-rollover and post-rollover responsibilities for the command center(s),
        major business units, facilities, and critical infrastructure components
 2.4    Develop agencywide schedule of key events for the pre- and post-rollover periods
 2.5    Develop or acquire and document facilities infrastructure checklists and templates
 2.6    Develop post-rollover test plans and procedures for testing of key business systems
        and processes
 2.7    Establish rapid response procedures; develop rules for the deployment of business
        resumption teams and for the execution of contingency and disaster recovery plans
 2.8    Develop rollover staffing plan
 2.9    Review vendor service agreements and availability
 2.10   Establish rollover security procedures for coordination/command center(s) and
        designated Day One sites
 2.11   Develop and document internal communications procedures; establish process for
        monitoring external events
 2.12   Develop external communications strategy and proced       dures



2.1    Assess failure scenarios documented in business continuity and contingency plans and
       identify key risk areas, including risks faced by business partners

        Identify key risk areas, including facilities, critical infrastructure components, and
       systems supporting core business processes, that should be addressed by the Day One
       plan. Rely on the results of Year 2000 risk assessment and testing activities to identify
       the most likely failure modes and scenarios. Consider the potential impact of external
       events, including the loss of electric power, telephone communications, water, and
       natural gas. Consider risks faced by business partners.




GAO/AIMD-10.1.22 Year 2000 Day One Planning and Operations Guide                                   7
2.2   Establish information coordination and/or command center(s)

      Establish information coordination and/or command center(s) to coordinate and report
      on agencywide rollover activities.

2.3   Define the pre-rollover and post-rollover responsibilities for the coordination/command
      center(s), major business units, facilities, and critical infrastructure components

      Define, for each organizational component responsible for a key business process, its
      pre- and post-rollover responsibilities, staffing, and problem resolution and escalation
      procedures. Specify events and conditions defined in the business continuity and
      contingency plan that will trigger contingency plans and activate business resumption
      teams. Review procedures and triggers for activating contingency plans to deliver
      services to constituencies served by business partners experiencing Year 2000-induced
      system and process failures.

      Specify data backup requirements, and identify systems and infrastructure components
      that are to remain operational, remain idle, or be shut down. Develop start-up plan for
      idled or shutdown systems and infrastructure components.

2.4   Develop agencywide schedule of key events for the pre- and post-rollover periods

      Develop agencywide schedule of key events and activities, including a schedule for
      closing business areas, shutting down systems, and facilities inspections.

2.5   Develop or acquire and distribute facilities infrastructure checklists and templates

      The century rollover may affect the availability of public utilities and the operations of
      building services, including elevators, air conditioning, and building security systems.
      Agencies should develop or acquire infrastructure checklists, data capture and reporting
      templates, and reporting procedures to assist building managers in assessing the rollover
      impact on agency facilities.

2.6   Develop post-rollover test plans and procedures for testing of key business systems and
      processes

      The post-rollover test and assessment process must be well defined and scripted to
      ensure adequate coverage and proper sequencing. For example, evaluation procedures
      may call for sequential evaluation beginning with hardware (mainframes, servers, PBX
      systems, routers, and switches), followed by connectivity testing (LANs and WANs), and
      ending with end-to-end evaluation and testing of mission-critical systems with live data.

      Where possible, use existing troubleshooting and problem reporting processes and
      procedures during the rollover period, including those specified in agency Continuity of
      Operations plans.



GAO/AIMD-10.1.22 Year 2000 Day One Planning and Operations Guide                                  8
2.7   Establish rapid response procedures; develop rules for the deployment of business
      resumption teams and for the execution of contingency and disaster recovery plans

      Define risk-escalation procedures, activation thresholds, and action plans for predefined
      failure scenarios, including the activation of business resumption teams. Consider
      activating and terminating a response based on risk-escalation thresholds for the
      degradation or loss of business systems and applications.

2.8   Develop rollover staffing plan

      Develop a rollover staffing plan to support Day One activities. Ensure that key technical
      staff are available to handle infrastructure problems, and that business resumption
      teams will be available to respond to problems affecting agency core business processes.
      Assess the availability of key staff to support Day One operations. Develop procedures
      for reaching all employees, including on-duty and on-call staff.

2.9   Review vendor service agreements and availability

      Review contract support arrangements to determine availability and coverage. Confirm
      the availability and contact information for key vendors. Avoid entering into an
      exclusive service arrangement with vendors serving other clients.

2.10 Establish rollover security procedures for coordination/command center(s) and designated
     Day One sites

      Review security plans to ensure their support for rollover activities. Ensure access to
      facilities in the event of security system failures, and provide additional physical security
      for critical facilities.

      Assess system and network security during the rollover period. Consider risks posed by
      remote access services and public web servers. Modify or reduce remote access to
      network resources during the rollover period. Develop plans to monitor, resolve, and
      report network security incidents.

2.11 Develop and document internal communications procedures; establish process for
     monitoring external events

      Develop and document internal communications strategy and process, both for intra-
      and inter-component communications, including communications between the Day One
      sites and the coordination/command center. Define status indicators and identify
      reportable events. Define modes of communication and reporting channels. Ensure that
      key Day One teams and staff have access to emergency communications equipment in
      case of the failure of public telecommunications networks.

      Establish process for the monitoring of external events, including communications with
      the President's Council on Year 2000 Conversion’s Information Coordination Center, to


GAO/AIMD-10.1.22 Year 2000 Day One Planning and Operations Guide                                  9
     obtain information about major infrastructure problems. Ensure that the
     coordination/command center(s) have access to information on the status of business
     partners, key suppliers, and service providers.

2.12 Develop external communications strategy and procedures

     Develop and document external communications strategy and process. Designate
     agency Day One spokesperson, and define the process for issuing public announcements
     and communicating with the media. Establish communication links with Information
     Coordination Center.




GAO/AIMD-10.1.22 Year 2000 Day One Planning and Operations Guide                           10
3.0 Rehearsal

The Day One plan describes a wide range of complex, interrelated activities and geographically
distributed processes that must be executed within a very short time frame. To ensure that the
Day One strategy is executable, the Day One plans and their key processes and timetables
should be reviewed, and, if feasible, rehearsed. Similarly, agencies may wish to rehearse the
operations and integration of the Year 2000 coordination/command center(s) and teams
responsible for the Day One activities in major business units, facilities, field organizations, and
infrastructure components.


Key Tasks

3.1   Develop and document Day One rehearsal plan
3.2   Rehearse selected Day One operations and teams
3.3   Update and revise Day One plans and procedures based upon lessons learned


3.1   Develop and document Day One rehearsal plan

       Define and document the Day One rehearsal plan. Ensure that management approves
      the plan. Disseminate applicable guidance and establish a help desk, preferably in the
      coordination/command center(s). Rehearsal plans should address objectives, scope,
      required equipment and personnel resources, likely scenarios, schedules and locations,
      and procedures.

3.2   Rehearse selected Day One operations and teams

      Rehearse the operations of the coordination/command center(s) and of selected Day One
      teams to ensure that the coordination/command center staff and team member are
      familiar with Day One procedures and their roles. Test emergency communications and
      security procedures designed to deal with the loss of electric power and public
      communications networks.

3.3   Update and revise Day One plans and procedures based upon lessons learned

      Identify and resolve shortcomings and problems noted during the rehearsals and testing,
      and revise the Day One plan and procedures accordingly. Retest, if feasible, critical
      processes, and update plans.




GAO/AIMD-10.1.22 Year 2000 Day One Planning and Operations Guide                                  11
4.0 Execution, Monitoring, Responding, and Reporting

Unlike schedules for most other plans, the Day One schedule is immovable, with the rollover to
occur at 12:00 A.M. on Saturday, January 1, 2000. The status—and success—of an agency’s
Year 2000 program will first be evident immediately after the rollover, with more detailed
information on the status of mission-critical systems and on the viability of agency core business
processes emerging during the post-rollover phase. The status information gathered during the
post-rollover phase needs to be communicated to executive management as well as to external
parties.

 Key Tasks

 Pre-rollover
 4.1 Activate information coordination/command center(s) and designated Day One sites
 4.2 Implement pre-rollover procedures specific to each site
 4.3 Implement planned risk prevention and risk reduction measures
 Post-rollover
 4.4 Conduct facilities inspection using infrastructure checklist
 4.5 Perform post-rollover tests, evaluations, and assessment
 4.6 Identify and report incidents, including problems, failures, and outages
 4.7 Perform problem and crisis management
 4.8 Implement recovery procedures and change control
 4.9 Activate, if necessary, business continuity and contingency plans
 4.10 Implement internal and external reporting procedures


4.1   Activate information coordination/command center(s) and designated Day One sites

      Activate coordination/command center(s) and designated sites, perform connectivity
      checks, and test reporting procedures and tools. Ensure that each site has accessible
      emergency contact information for local utilities, public safety organizations, key
      vendors, technical consultants, and contractors servicing critical equipment. Ensure
      that each site and business area addressed by a rollover strategy has a designated crisis
      intervention contact and that communication is established to the coordination/command
      center(s). Ensure that response teams are in communication, ready and awaiting
      dispatch.

4.2   Implement pre-rollover procedures specific to each site

      Implement pre-rollover procedures at all designated sites. Conduct operational
      inspections and readiness checks for normal and emergency backup services, including
      electric power, lighting, water, communications, and transportation, as appropriate.




GAO/AIMD-10.1.22 Year 2000 Day One Planning and Operations Guide                               12
4.3   Implement planned risk-prevention and risk-reduction measures

      Implement planned measures to prevent and reduce risks in business processes, systems,
      and applications. For example, perform backups of all critical files and data, including
      automated workflow rules, system configurations, database journals, system data, and
      application data.

      Implement shutdown or partial shutdown of processes and applications that may not be
      able to process correctly across the rollover event.

4.4   Conduct facilities inspection using infrastructure checklist

      Inspect facilities using infrastructure checklist. Report status and problems, and initiate
      corrective action.

4.5   Perform post-rollover tests, evaluations, and assessments

      Initiate post-rollover test and evaluation procedures. Test key business processes and
      supporting systems. Coordinate tests and evaluations with designated Day One sites and
      business partners.

4.6   Identify and report incidents, including problems, failures, and outages

      Report significant failures and outages. Provide status reports on key business processes
      and supporting systems. Identify and report on potential impact of failures and outages.

4.7   Perform problem and crisis management

      Document all reported incidents, assign priority, and dispatch appropriate problem
      response team. Periodically review and escalate response as appropriate. When called
      for, invoke emergency procedures. Track incidents, problems, and crises to closure.

4.8   Implement recovery procedures

      Follow recovery procedures for the rollover period. Pay special attention to systems and
      applications that were fully or partially shut down as a preventive measure. Test and
      implement corrections.

4.9   Activate, if necessary, business continuity and contingency plans

      Use escalation thresholds to invoke contingency plans. If a problem remains unresolved
      for a specific length of time and the impact to operations exceeds a predefined threshold,
      activate business resumption teams.




GAO/AIMD-10.1.22 Year 2000 Day One Planning and Operations Guide                               13
4.10 Implement internal and external reporting procedures

     Implement internal reporting procedures. Ensure that the coordination/command
     center(s) are provided with timely and accurate information on the rollover status of
     each designated site. Use standard data collection and reporting templates and tools to
     gather and report status information.

     Implement external communications strategy and process. If required, prepare and issue
     public announcements and communicate with the media. Ensure that the Information
     Coordination Center is provided with timely and accurate status information.




GAO/AIMD-10.1.22 Year 2000 Day One Planning and Operations Guide                           14
Glossary

The definitions in this glossary were developed by the project staff or were drawn from other
sources, including the Computer Dictionary: The Comprehensive Standard For Business,
School, Library, and Home, Microsoft Press, Washington, D.C., 1991; The Year 2000
Resource Book, Management Support Technology Corp., Framingham, Massachusetts, 1996;
The Year 2000 and 2-Digit Dates: A Guide for Planning and Implementation, International
Business Machines Corporation, 1997; and Denis Howe’s “Free On-line Dictionary of
computing at <foldoc.doc.is.ac.uk/>.

Business area             A grouping of business functions and processes focused on the
                          production of specific outputs.

Business function         A group of logically related tasks that are performed together to
                          accomplish a mission-oriented objective.

Business plan             An action plan that the enterprise will follow on a short-term and/or
                          long-term basis. It specifies the strategic and tactical objectives of the
                          enterprise over a period of time. The plan, therefore, will change
                          over time. Although a business plan is usually written in a style
                          unique to a specific enterprise, it should concisely describe "what" is
                          planned, "why" it is planned, "when" it will be implemented, by
                          "whom" it will be implemented, and "how" it will be assessed. The
                          architects of the plan are typically the principals of the enterprise.

Business resumption       Teams responsible for managing the implementation of contingency
teams                     and business resumption plans.

Contingency plan          In the context of the Year 2000 program, a plan for responding to
                          the loss or degradation of essential services due to a Year 2000-
                          related problem in an automated system. In general, a contingency
                          plan describes the steps the enterprise would take--including the
                          activation of manual or contract processes--to ensure the continuity
                          of its core business processes in the event of a Year 2000-induced
                          system failure.

Infrastructure            In the context of the Day One plan, the hardware, software, facilities,
                          and public utilities supporting the enterprise’s information
                          management functions.

Metrics                   Measures by which processes, resources, and products can be
                          assessed.




GAO/AIMD-10.1.22 Year 2000 Day One Planning and Operations Guide                                 15
Mission-critical         A system supporting a core business activity or process.
system

Quality assurance        Planned and systematic actions necessary to provide adequate
                         confidence that a product or service will satisfy given requirements
                         for quality.

Risk assessment          An activity performed to identify risks and estimate their probability
                         and the impact of their occurrence; used during system development
                         to provide an estimate of potential damage, loss, or harm that could
                         result from the failure to successfully develop individual system
                         components.

Risk management          A management approach designed to prevent and reduce risks,
                         including system-development risks, and lessen the impact of their
                         occurrence.

Test                     The process of exercising a product to identify differences between
                         expected and actual behavior.

Year 2000 problem        The potential problems that might be encountered by computer
                         hardware, software, or firmware in processing year-date data for
                         years beyond 1999.




GAO/AIMD-10.1.22 Year 2000 Day One Planning and Operations Guide                                16