oversight

Defense Computers: U.S. Transportation Command's Management of Y2K Operational Testing

Published by the Government Accountability Office on 1999-11-15.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                 United States General Accounting Office

GAO              Report to the Chairman of the
                 Subcommittee on Defense, Committee on
                 Appropriations, House of
                 Representatives

November 1999
                 DEFENSE
                 COMPUTERS

                 U.S. Transportation
                 Command’s
                 Management of Y2K
                 Operational Testing




GAO/AIMD-00-21
Contents



Letter                                                                                3


Appendixes   Appendix I: Briefing on Results of GAO Review of
               TRANSCOM Turbo Y2K-Part A OPEVAL                                      14
             Appendix II:   Objectives, Scope, and Methodology                       58
             Appendix III: GAO Contact and Staff Acknowledgements                    59


Tables       Table 1: Highlights of Key DOD-Recommended Year 2000
               Testing Processes                                                      7
             Table 2: Summary of TRANSCOM Satisfaction of JCS Evaluation
               Criteria on Deployment Evaluation                                      8




             Abbreviations

             AMC             Air Mobility Command
             CINC            Commanders-in-Chief
             DOD             Department of Defense
             JCS             Joint Chiefs of Staff
             MSC             Military Sealift Command
             MTMC            Military Traffic Management Command
             MTW             major theater war
             OPEVAL          operational evaluation
             TRANSCOM        U.S. Transportation Command
             Y2K             Year 2000




             Page 1                      GAO/AIMD-00-21 TRANSCOM Y2K Operational Testing
Page 2   GAO/AIMD-00-21 TRANSCOM Y2K Operational Testing
United States General Accounting Office                                                   Accounting and Information
Washington, D.C. 20548                                                                         Management Division



                                    B-282545                                                                             Leter




                                    November 15, 1999

                                    The Honorable Jerry Lewis
                                    Chairman
                                    Subcommittee on Defense
                                    Committee on Appropriations
                                    House of Representatives

                                    Dear Mr. Chairman:

                                    Complete and thorough Year 2000 (Y2K) end-to-end testing is essential to
                                    provide reasonable assurance that new or modified systems used to
                                    collectively support a core business function or mission operation will not
                                    jeopardize an organization’s ability to deliver products and services as a
                                    result of the Y2K computing problem. This is especially true for the
                                    Department of Defense (DOD) because it relies on a complex and broad
                                    array of interconnected computer systems—including weapons, command
                                    and control, satellite, inventory management, transportation management,
                                    health, financial, personnel and payment systems—to carry out its military
                                    operations and supporting business functions.

                                    At your request, we are reviewing DOD’s management of its various Year
                                    2000-related end-to-end testing activities. As part of our efforts, we
                                    assessed the U.S. Transportation Command’s (TRANSCOM) management
                                    of its end-to-end test of its ability to plan and execute joint major theater
                                    war (MTW) deployment operations, and determined what the results of this
                                    test show with respect to operational risks and readiness. 1 We briefed
                                    TRANSCOM officials on our findings on August 24, 1999, and made
                                    recommendations to correct the management weaknesses that we found.
                                    TRANSCOM took immediate action to address our recommendations, and
                                    on September 14, 1999, we briefed your office on our findings and
                                    TRANSCOM’s actions. The purpose of this letter is to summarize our
                                    briefing to your office. The briefing slides that we presented to your office
                                    are in appendix I, and the objectives, scope, and methodology of our review
                                    are detailed in appendix II. TRANSCOM provided oral comments on our
                                    briefing slides at the August 24, 1999, briefing, and we have incorporated


                                    1
                                     DOD refers to its combatant commands’ end-to-end tests as operational evaluations
                                    (OPEVAL).




                                    Page 3                             GAO/AIMD-00-21 TRANSCOM Y2K Operational Testing
                   B-282545




                   them as appropriate. We performed our audit work from March through
                   September 1999 in accordance with generally accepted government
                   auditing standards.



Results in Brief   Year 2000 end-to-end testing is an essential component of an effective Year
                   2000 testing program since Y2K-related problems can affect so many of the
                   systems owned and operated by an entity as well as systems belonging to
                   business partners and infrastructure providers. Moreover, to be effective,
                   end-to-end testing should be approached in a structured and disciplined
                   fashion. Both the Joint Chiefs of Staff (JCS) guidance to its combatant
                   commands on managing Year 2000 operational evaluations,2 (the term JCS
                   uses to refer to Year 2000 end-to-end testing) and our Year 2000 test
                   guidance3 define a number of key management controls to employ when
                   planning, executing, analyzing, and reporting on these tests and
                   evaluations.

                   We found that TRANSCOM’s deployment operational evaluation satisfied
                   most of the key processes that JCS’ guidance specifies. For example,
                   TRANSCOM established a Y2K task force to guide the evaluation effort,
                   which included surface and airlift experts, test and evaluation experts from
                   the Joint Interoperability Test Center and the Defense Office of Test and
                   Evaluation, component command and service representatives, and a public
                   affairs representative. Further, TRANSCOM performed a rehearsal before
                   conducting the test to ensure that all critical systems and interfaces were
                   operating correctly and that all staff knew their roles and responsibilities.

                   However, TRANSCOM had not satisfied key steps that are pivotal to
                   (1) fully disclosing and appropriately addressing risks associated with
                   limitations in the defined scope of the operational evaluation and
                   (2) accurately reporting on mission readiness and impacts. The result was
                   that the Year 2000 readiness of critical tasks associated with conducting a
                   MTW deployment—namely the use of commercial carriers and port
                   operations to deploy goods and people—was not known with sufficient
                   certainty to support TRANSCOM’s May 1999 reported position that it can



                   2
                       Joint Staff Year 2000 Operational Evaluation Guide, Version 3.0, April 1, 1999.
                   3
                    Year 2000 Computing Crisis: A Testing Guide (GAO/AIMD-10.1.21, issued as an exposure
                   draft in June 1998; issued in final in November 1998).




                   Page 4                                  GAO/AIMD-00-21 TRANSCOM Y2K Operational Testing
             B-282545




             conduct a MTW deployment in a Year 2000 environment with no material
             impact on operations.

             TRANSCOM has since either implemented or initiated appropriate actions
             to address our recommendations for correcting these weaknesses. For
             example, it amended its final reports on the evaluation to disclose the
             scope limitations. It is also working with its component commands to
             identify their major commercial carrier business partners, to assess their
             readiness and risks, and to develop risk mitigation strategies. Because
             TRANSCOM has initiated these and other actions, we are not making
             further recommendations at this time.



Background   As the single manager of Defense transportation, TRANSCOM’s mission is
             to coordinate the use of air, sea, and land transportation to deploy and
             sustain U.S. forces. To perform this mission, TRANSCOM relies on a wide
             array of information technology systems, including command and control
             systems, physical and human asset visibility and tracking systems,
             transportation feasibility and port analysis tools, and intelligence systems.
             TRANSCOM also depends heavily on partnerships with commercial
             transportation service providers such as Civil Reserve Air Fleet carriers,
             the commercial maritime industry, and railroad and trucking carriers. In
             total, these partners provide about 85 percent of Defense’s transportation
             needs.

             In August 1998, the Secretary of Defense directed the Commanders-in-
             Chief (CINC) to require its combatant commands, including TRANSCOM,
             to plan, execute, analyze, and report on a series of simulated Year 2000
             operational evaluations. The evaluations, which were to assess whether
             DOD can continue to perform critical military operations in a Year 2000
             environment, are one of three DOD end-to-end testing efforts.4

             The purpose of end-to-end testing is to verify that a defined set of
             interrelated systems, which collectively support an organizational core
             business area or function, interoperate as intended in an operational


             4
              In addition to conducting operational evaluations, the military services are conducting
             system integration testing and the functional business areas, such as personnel and health
             affairs, are conducting functional end-to-end tests. Each of these end-to-end testing
             activities is discussed in detail in Defense Computers: Management Controls Are Critical to
             Effective Year 2000 Testing (GAO/AIMD-99-172, June 30, 1999).




             Page 5                              GAO/AIMD-00-21 TRANSCOM Y2K Operational Testing
B-282545




environment (either actual or simulated). These interrelated systems
include not only those owned and managed by an organization, but also the
external systems with which they interface or that otherwise support the
business area or function. The CINC core business areas or functions are
referred to as “thin lines.”

The boundaries for end-to-end tests can vary depending on a given business
function’s system dependencies and criticality to the organizational
mission. Therefore, in managing end-to-end test activities, it is important to
analyze the interrelationships among core business functions and their
supporting systems and the mission impact and risk of date-induced system
failures and to use these analyses to define test boundaries. It is also
important to work early and continually with functional partners to ensure
that related end-to-end test activities are effectively coordinated and
integrated. Table 1 highlights key processes recommended by JCS’ Year
2000 operational evaluation guidance, which is consistent with our Year
2000 test guide.




Page 6                         GAO/AIMD-00-21 TRANSCOM Y2K Operational Testing
B-282545




Table 1: Highlights of Key DOD-Recommended Year 2000 Testing Processes


Planning       • Specify test assumptions and limitations
               • Establish a Year 2000 Task Force
               • Identify critical missions/tasks/systems
               • Verify that systems essential to the mission are Year 2000 compliant
               • Develop an operational evaluation plan to guide event planning and
                 execution
               • Identify and schedule support from other commands, DOD components,
                 etc.
               • Determine relevant and necessary resources (e.g., funding, personnel,
                 equipment)
               • Ensure that approved Year 2000 contingency plans are prepared
               • Develop risk management plan
               • Identify simulation needs and establish supporting testing environment
               • Develop data collection and analysis plan or approaches
Execution      • Conduct operational evaluation rehearsal
               • Follow configuration management policy
               • Perform baseline test for operational evaluation
               • Execute required Year 2000 date rollover tests
               • Collect and archive all Year 2000-relevant data and ensure that systems
                 are reset to current day operations
Analysis       • Categorize, document, and report failures
               • Determine mission impact of Year 2000 failures
               • Ensure that exit criteria are met
Reporting      • Prepare Year 2000 reports describing mission impact and readiness
               • Provide reports to Joint Staff within required time frames



TRANSCOM has already carried out two operational evaluations—the first,
conducted in April 1999, assessed its ability to manage a MTW deployment
operation and the second, conducted in May and June 1999, assessed its
ability to manage a MTW sustainment operation.5 A third evaluation, which
will retest the same 23 systems included in the May and June sustainment


5
 Deployment refers to all activities associated with systematically transporting personnel,
material, and other elements from a home station or origin through destination.
Sustainment refers to all activities associated with maintaining or supporting the deployed
personnel, material, and other elements.




Page 7                               GAO/AIMD-00-21 TRANSCOM Y2K Operational Testing
                         B-282545




                         evaluation, is planned for October 1999. In addition to these events,
                         TRANSCOM has taken part in other DOD Year 2000 exercises.



TRANSCOM                 As noted in table 2, we found that TRANSCOM had satisfied the majority of
                         the management process controls (17 of 21) specified in JCS’ operational
Implemented Many         evaluation guidance.
Important Management
Processes on Its
Deployment               Table 2: Summary of TRANSCOM Satisfaction of JCS Evaluation Criteria on
                         Deployment Evaluation
Operational Evaluation
                                                                 Number of primary      Number of primary
                         Phases                                   evaluation criteria     criteria satisfied
                         Planning                                                 11                      9
                         Execution                                                 5                      5
                         Analysis                                                  3                      2
                         Reporting                                                 2                      1
                         Total                                                    21                     17



                         Consistent with JCS guidance governing operational evaluation planning,
                         TRANSCOM established a Year 2000 task force, which included surface and
                         airlift experts, test and evaluation experts, system analysts, component and
                         service representatives, and a public affairs representative. It identified 15
                         critical tasks needed to carry out a MTW deployment mission and identified
                         a total of 23 “thin line” systems that support its capability to manage the
                         deployment mission. TRANSCOM also verified that the 23 systems were
                         certified as Year 2000 compliant. Further, TRANSCOM developed a test
                         plan that documented participant roles and responsibilities, critical
                         missions and tasks, test cases, and reporting requirements.

                         TRANSCOM also took effective steps in executing and reporting its
                         evaluation. For instance, before executing the operational evaluation,
                         TRANSCOM performed a rehearsal to ensure that all critical systems and
                         interfaces were operating correctly and that all staff knew their roles and
                         responsibilities. Before resetting systems to present day operational
                         conditions, TRANSCOM officials determined that the defined “thin line”
                         was completely exercised, all items in the master scenario events list were
                         performed, and that all data needed to make an assessment of the
                         command’s ability to perform its deployment mission were collected and



                         Page 8                         GAO/AIMD-00-21 TRANSCOM Y2K Operational Testing
                         B-282545




                         archived. Also, TRANSCOM ensured that required reports were submitted
                         on time and that they addressed specified reporting requirements.



TRANSCOM Did Not         While TRANSCOM followed most JCS-specified criteria needed to
                         effectively manage its evaluation, it did not adequately satisfy certain
Satisfy Some Key Steps   planning, analysis, and reporting criteria associated with the evaluation’s
Related to Testing       scope. Specifically, when planning the evaluation, TRANSCOM concluded
                         that it was not feasible or affordable to actively engage commercial
Limitations and          transportation providers and ports in its evaluation. As a result,
Deviations               TRANSCOM decided to rely on its commercial partners to ensure the
                         readiness of their respective transportation systems, and thus to assume
                         these partners’ ability to perform their military deployment and
                         sustainment support roles and responsibilities. Further, while TRANSCOM
                         officials stated that they communicated this assumption orally to JCS,
                         TRANSCOM’s operational evaluation plan did not specify this scope
                         limitation nor did it address the exclusion of these partners as a risk that
                         could adversely affect its ability to draw conclusions about TRANSCOM’s
                         mission readiness. Moreover, TRANSCOM did not verify the validity of this
                         assumption through any assessment of its key business partners’ readiness,
                         and in fact did not consider information available at the time of the
                         operational evaluation that raised concerns about the Year 2000 readiness
                         of the nation’s transportation sector. Despite this scope limitation,
                         TRANSCOM reported to JCS that critical deployment tasks could be
                         performed during calendar and leap year date rollovers with no significant
                         impact on its mission readiness.

                         As stressed in our briefing to TRANSCOM officials, Defense transportation
                         services for deployment and sustainment are provided primarily by the
                         commercial transportation sector. For example, Civil Reserve Air Fleet
                         carriers account for 90 percent of long-range passenger capability and
                         40 percent of long-range cargo capability. The commercial maritime
                         industry provides nearly all wartime sustainment capability. In addition,
                         commercial rail and truck carriers provide virtually all continental U.S.
                         surface transportation capability. TRANSCOM also depends on commercial
                         systems that support and maintain air and water port operations to fulfill
                         its deployment and sustainment missions. Therefore, without knowing
                         commercial carrier and port Year 2000 readiness, TRANSCOM’s conclusion
                         that all its critical missions and tasks could continue to be performed into
                         the Year 2000 without material disruption was not justified. Also, without
                         disclosing the scope limitation and related risks, TRANSCOM did not
                         provide DOD leadership with complete and reliable information upon



                         Page 9                        GAO/AIMD-00-21 TRANSCOM Y2K Operational Testing
                  B-282545




                  which to assess DOD’s institutional Year 2000 readiness to execute military
                  operations.

                  To address these concerns, we recommended in our August 24, 1999,
                  briefing that TRANSCOM amend its final reports to JCS to disclose the
                  scope limitations and related risks. We also recommended that
                  TRANSCOM assess and selectively verify the readiness of transportation
                  systems belonging to its commercial partners.



TRANSCOM Is       TRANSCOM agreed with our recommendations and is responding to them.
                  First, TRANSCOM amended its final reports to JCS to disclose the scope
Addressing Our    limitations of its operational evaluation and to qualify its previous
Recommendations   conclusion about Year 2000 readiness. In brief, the amended report states
                  that TRANSCOM’s ability to perform its full deployment and sustainment
                  missions requires a yet-to-be confirmed assumption that the commercial
                  transportation industry can perform its part of the missions.

                  Second, to assess commercial carrier readiness, TRANSCOM has been
                  seeking confirmation letters from commercial carriers whose systems
                  exchange data with TRANSCOM’s Global Transportation Network. The
                  Global Transportation Network is a system that provides in-transit visibility
                  of transportation assets, passengers, and cargo. As of August 24, 1999,
                  TRANSCOM had received confirmations on Y2K readiness from 20 of 27 of
                  the network’s data exchange partners.

                  Third, TRANSCOM has been working with other federal agencies, such as
                  the Department of Transportation, to assess transportation sector
                  readiness by reviewing industry assessments and Internet web sites. It also
                  plans to verify the reported readiness of selected transportation partners.




                  Page 10                       GAO/AIMD-00-21 TRANSCOM Y2K Operational Testing
              B-282545




              Fourth, TRANSCOM is working with its component commands (i.e., Air
              Mobility Command, Military Traffic Management Command, and Military
              Sealift Command)6 to identify their major commercial carrier business
              partners, identify these partners’ Year 2000 readiness and associated risks,
              and develop risk mitigation strategies. The first of these status reports was
              due September 15, and showed that significant progress had been made by
              TRANSCOM in determining the Year 2000 readiness of its domestic
              commercial transportation partners. For example, TRANSCOM determined
              that 89 percent of its Civil Reserve Air Fleet carriers, all major rail carriers,
              and 196 of 199 trucking carriers have reported that they are Year 2000
              compliant. In addition, most North American ports have compliance
              programs in place and all military active and ready reserve fleet ships are
              certified Year 2000 compliant. However, TRANSCOM is still in the process
              of determining the Year 2000 readiness of international ports, airfields, and
              other international commercial carriers that are key to accomplishing its
              deployment and sustainment missions.



Conclusions   By acting swiftly to address our recommendations made during the
              August 24, 1999, briefing, TRANSCOM is increasing the effectiveness and
              value of its operational evaluation and is mitigating the risks associated
              with being able to operate effectively in the year 2000. Further, it has
              ensured that DOD managers have complete and reliable information to use
              in making informed military decisions. As a result, they have largely
              satisfied the intent of those recommendations, and we are not making any
              further recommendations at this time.


              We are sending copies of this report to Representative John P. Murtha,
              Ranking Minority Member, Subcommittee on Defense, House
              Appropriations Committee; Senator John Warner, Chairman, and Senator
              Carl Levin, Ranking Minority Member, Senate Committee on Armed
              Services; Senator Ted Stevens, Chairman, and Senator Daniel Inouye,


              6
               TRANSCOM consists of three military service-level component commands, including the
              Army Military Traffic Management Command (MTMC), the Air Force Air Mobility Command
              (AMC), and the Navy Military Sealift Command (MSC). MTMC manages both land (truck
              and rail) transportation and water port operations. AMC provides strategic airlift and aerial
              refueling services, performs aeromedical evacuation, and operates special assignment
              aircraft (such as Air Force One). MSC employs a combination of government-owned and
              commercial ships to carry out sea transportation operations and also manages
              prepositioned ships.




              Page 11                              GAO/AIMD-00-21 TRANSCOM Y2K Operational Testing
B-282545




Ranking Minority Member, Subcommittee on Defense, Senate Committee
on Appropriations; Representative Floyd Spence, Chairman, and Ike
Skelton, Ranking Minority Member, House Committee on Armed Services.

We are also sending copies to the Honorable John Koskinen, Chair of the
President’s Year 2000 Conversion Council; the Honorable William Cohen,
Secretary of Defense; the Honorable John Hamre, Deputy Secretary of
Defense; General Henry Shelton, Chairman of the Joint Chiefs of Staff;
Arthur Money, Assistant Secretary of Defense for Command, Control,
Communications, and Intelligence; and the Honorable Jacob Lew, Director,
Office of Management and Budget. Copies will also be made available to
others upon request.

Should you or your staff have any questions concerning this report, please
contact me at (202) 512-6240. I can also be reached by e-mail at
brockj.aimd@gao.gov. Other points of contact and key contributors to this
report are listed in appendix III.

Sincerely yours,




Jack L. Brock, Jr.
Director, Governmentwide and Defense
 Information Systems




Page 12                      GAO/AIMD-00-21 TRANSCOM Y2K Operational Testing
Page 13   GAO/AIMD-00-21 TRANSCOM Y2K Operational Testing
Appendix I

Briefing on Results of GAO Review of
TRANSCOM Turbo Y2K-Part A OPEVAL                                              Appendx
                                                                                    Ii




           Results of GAO Review of
      TRANSCOM Turbo Y2K-Part A OPEVAL

             House Appropriations Committee

                   September 14, 1999
                                                                    1




                     Page 14    GAO/AIMD-00-21 TRANSCOM Y2K Operational Testing
                    Appendix I
                    Briefing on Results of GAO Review of
                    TRANSCOM Turbo Y2K-Part A OPEVAL




                                             Introduction

•   In August 1998, the Secretary of Defense directed the
    Commanders-in-Chief (CINC), who are responsible for Defense’s
    unified combatant commands, to plan, execute, and report on a
    series of simulated Year 2000 operational evaluations (Y2K
    OPEVALs).

•   The CINC Y2K OPEVALs are one of three Defense Y2K end-to-
    end test and evaluation efforts. GAO’s Y2K test guide advocates
    end-to-end testing, which is testing performed to verify that a
    defined set of interrelated systems (i.e., systems that collectively
    support an organizational core business function or operation)
    interoperate as intended in a Y2K environment.

•   The CINC core business functions/operations are referred to as
    “thin lines.”
                                                                                         2




                    Page 15                          GAO/AIMD-00-21 TRANSCOM Y2K Operational Testing
                   Appendix I
                   Briefing on Results of GAO Review of
                   TRANSCOM Turbo Y2K-Part A OPEVAL




                                              Objectives

•   At the request of the Chairman, House Appropriations Committee,
    Defense Subcommittee, GAO is reviewing selected OPEVALs to
    determine

    (1) if the OPEVAL was planned, executed, and documented in
    accordance with DOD guidelines and

    (2) what the OPEVAL results indicated concerning readiness and risks.

•   OPEVALs GAO reviewed included those conducted by
    TRANSCOM and SPACECOM and were selected in collaboration
    with the Defense Inspector General (IG) to ensure
     – appropriate coverage of all CINC OPEVALs and
     – no duplication of effort.

                                                                                        3




                   Page 16                          GAO/AIMD-00-21 TRANSCOM Y2K Operational Testing
                      Appendix I
                      Briefing on Results of GAO Review of
                      TRANSCOM Turbo Y2K-Part A OPEVAL




                                 Scope and Methodology

•   This briefing addresses the TRANSCOM Turbo Y2K - Part A
    (major theater war (MTW) deployment) OPEVAL. To satisfy
    objective (1), we
    – reviewed the OPEVAL plan, testing documents/records, and test
      results/reports,
    – interviewed TRANSCOM officials responsible for Y2K OPEVAL planning,
      execution, and reporting tasks,
    – observed the century date rollover test for the MTW deployment execution
      function, and
    – compared TRANSCOM’s planning, execution, analysis, and reporting
      actions against Defense OPEVAL guidance.




                                                                                           4




                      Page 17                          GAO/AIMD-00-21 TRANSCOM Y2K Operational Testing
                      Appendix I
                      Briefing on Results of GAO Review of
                      TRANSCOM Turbo Y2K-Part A OPEVAL




                                Scope and Methodology
•   To satisfy objective (2), we
     – reviewed TRANSCOM’s OPEVAL results, 7- and 30-day reports, and system
       problem tracking reports and
     – interviewed TRANSCOM officials and analysts responsible for developing
       OPEVAL assessment methodologies, interpreting evaluation metrics, and
       ensuring that evaluation exit criteria were met.


•   On August 24, 1999 we briefed TRANSCOM leadership on the results
    of our review and made recommendations to address our findings.
    TRANSCOM immediately acted on these recommendations. These
    actions are described where appropriate in this briefing.

•   We performed our work from March through September 1999 in
    accordance with generally accepted government auditing standards.

                                                                                           5




                      Page 18                          GAO/AIMD-00-21 TRANSCOM Y2K Operational Testing
                           Appendix I
                           Briefing on Results of GAO Review of
                           TRANSCOM Turbo Y2K-Part A OPEVAL




                                                    Background
DOD OPEVAL Overview

•   To assist the CINCs in planning, documenting, executing,
    analyzing, and reporting OPEVALs, the Joint Staff issued OPEVAL
    guidance. The guidance is divided into phases:
     –   identification and planning
     –   execution
     –   analysis
     –   reporting


•   The OPEVAL guidance is consistent with GAO’s end-to-end testing
    guidance and DOD’s Y2K management plan.




                                                                                                6




                           Page 19                          GAO/AIMD-00-21 TRANSCOM Y2K Operational Testing
                             Appendix I
                             Briefing on Results of GAO Review of
                             TRANSCOM Turbo Y2K-Part A OPEVAL




                                                    Background
TRANSCOM and Its OPEVAL Events’ Status
•   As the single manager of Defense transportation, TRANSCOM
    coordinates the use of air, sea, and land transportation to deploy and
    sustain U.S. forces.

•   To fulfill its deployment and sustainment missions, TRANSCOM
    depends on and partners with commercial transportation service
    providers. These partners provide about 85% of Defense’s
    transportation needs. For example,
     – Civil Reserve Air Fleet (CRAF) carriers account for 90% of long-range
       passenger capability and 40% of long-range cargo capability.
     – the commercial maritime industry provides nearly all wartime sustainment
       capability, and
     – commercial rail and truck carriers provide virtually all continental U.S. surface
       transportation capability.

                                                                                                  7




                             Page 20                          GAO/AIMD-00-21 TRANSCOM Y2K Operational Testing
                       Appendix I
                       Briefing on Results of GAO Review of
                       TRANSCOM Turbo Y2K-Part A OPEVAL




                                              Background

•   TRANSCOM also depends on commercial systems that support
    and maintain air and water port operations to fulfill its deployment
    and sustainment missions.

•   TRANSCOM identified an OPEVAL “thin line” within its deployment
    mission and an OPEVAL “thin line” within its sustainment mission.
    The deployment “thin line” consists of 15 critical tasks and 23
    systems that map to those tasks. The sustainment “thin line”
    consists of 7 critical tasks and 23 systems.

•   The following table describes the status of TRANSCOM’s OPEVAL
    events.



                                                                                            8




                       Page 21                          GAO/AIMD-00-21 TRANSCOM Y2K Operational Testing
                                                  Appendix I
                                                  Briefing on Results of GAO Review of
                                                  TRANSCOM Turbo Y2K-Part A OPEVAL




                                                                               Background
    Evaluation/Readiness           “Thin Line”                         Schedule                          Results
    Assessment Events
    OPEVAL Part A                  MTW deployment                      April 5-30, 1999                  1 soft c failure and 5 hardd failures;
                                   management                                                            no contingency plans invoked.



    OPEVAL Part B                  MTW sustainment                     May 24-June 4, 1999               1 hard failure; contingency plan
                                   management                                                            invoked.
    OPEVAL Part C                  MTW sustainment                     October 12-27, 1999
                                   management retest
    CCA a                          MTW deployment                      May 3-7, 1999                     Global Transportation Network
                                   management                                                            (GTN)e contingency plan
                                                                                                         evaluated.
    JMRR b                         Dual MTW deployment                 August 1999                       Classified assessment on Y2K
                                   and sustainment                                                       readiness of international ports and
                                                                                                         airfields.
a
  The Chairman’s Contingency Assessment (CCA) was designed to evaluate the ability of TRANSCOM to perform the MTW deployment mission in an
   environment degraded by Y2K failures.
b
  Joint Monthly Readiness Reviews (JMRR) focus on describing readiness deficiencies and corrective actions and are used by Defense to identify
   opportunities to improve warfighting effectiveness.
c
  A soft failure is a Y2K-related failure that is not immediately discernable. The effect may be cumulative and require several hours, days, or longer to
   manifest itself.
d
  A hard failure is a Y2K-related failure that results in an obvious adverse impact to the system. For example, the system shuts down, erroneous data is
   displayed, or unexpected actions occur.
e
  GTN is an integrated database containing all the data necessary to track the location of deploying units’ personnel and equipment, patients, sustainment
   cargo, and other vital resources as they move between theaters.


                                                                                                                                                 9




                                                  Page 22                                     GAO/AIMD-00-21 TRANSCOM Y2K Operational Testing
                     Appendix I
                     Briefing on Results of GAO Review of
                     TRANSCOM Turbo Y2K-Part A OPEVAL




                                            Background
Turbo Y2K-Part A OPEVAL
•   Turbo Y2K - Part A evaluated TRANSCOM’s ability to manage a joint
    MTW deployment operation.

•   Turbo Y2K - Part A OPEVAL examined one “thin line”--deployment
    management--and the capability of the system’s “thin line” to operate
    correctly at Y2K.

•   Turbo Y2K - Part A was conducted in collaboration with other DOD
    organizations, including Central Command (CENTCOM), the military
    services, and Defense agencies. It was intended to emulate a real-
    world MTW deployment operation.




                                                                                         10




                     Page 23                          GAO/AIMD-00-21 TRANSCOM Y2K Operational Testing
                            Appendix I
                            Briefing on Results of GAO Review of
                            TRANSCOM Turbo Y2K-Part A OPEVAL




                                                        Background
•   Turbo Y2K-Part A OPEVAL assessed century rollover and leap year
    critical dates.

•   Turbo Y2K-Part A OPEVAL was conducted during April 5-30, 1999.
    –   Rehearsal: March 29 - April 2, 1999.
    –   Baseline: April 5-9, 1999.
    –   Calendar year rollover: April 19-23, 1999.
    –   Leap year rollover: April 26-30, 1999.




                                                                                                11




                            Page 24                          GAO/AIMD-00-21 TRANSCOM Y2K Operational Testing
                                        Appendix I
                                        Briefing on Results of GAO Review of
                                        TRANSCOM Turbo Y2K-Part A OPEVAL




                                                                    Background
Turbo Y2K-Part A critical tasks                                                     “Thin Line” systems

1. Determine transportation and support availability                                GCCS/JOPES/GDSS/ADANS
                                                                                    CMARPS/ACFP
2. Coordinate and match transportation resources and requirements                   GCCS/JOPES/JFAST/ADANS
3. Provide for enroute support and clearances                                       No “thin line” systems used
4. Integrate deployment systems                                                     GTN/GCCS/JOPES/CMOS/DAAS
                                                                                    TCACCIS/GDSS/ADANS/RCAPS
                                                                                    CAPSII/C2IPS/Broker/CAMS-G081/IBS
                                                                                    WPS/GOPAX/IC3/CFM/AMS
5. Provide forces and mobility assets                                               C2IPS/GDSS/Broker/CAMS-G081
                                                                                    IC3/AMS/WPS/CFM
6. Provide terminal operations at ports of embarkation                              CAPSII/RCAPS/GDSS/WPS/AMS
                                                                                    GCCS/JOPES
7. Move forces and sustainment from origin to port of debarkation                   GCCS/JOPES/GDSS/ADANS/ACFP
                                                                                    C2IPS/GTN/TMDS/IBS/IC3/CFM
                                                                                    GOPAX/AMS/TCACCIS
8. Coordinate global strategic refueling                                            CMARPS/ADANS/GDSS/C2IPS/GTN
9. Provide global patient movement aeromedical evacuation                           No “thin line” systems used
10. Produce strategic and theater strategic intelligence and prepare intelligence   IDHS
    products
11. Disseminate and integrate theater strategic intelligence                        IDHS
12. Manage national military C4 systems worldwide for communicating strategic       No “thin line” systems used
    decisions and information
13. Provide strategic direction to forces worldwide                                 GDSS/C2IPS
14. Expand transportation support system                                            GDSS/ADANS/IC3/JFAST
15. Determine and validate forces and cargo to be deployed or redeployed            GCCS/JOPES


                                                                                                                  12




                                        Page 25                              GAO/AIMD-00-21 TRANSCOM Y2K Operational Testing
   Appendix I
   Briefing on Results of GAO Review of
   TRANSCOM Turbo Y2K-Part A OPEVAL




Results of GAO Review




                                                                       13




   Page 26                          GAO/AIMD-00-21 TRANSCOM Y2K Operational Testing
                        Appendix I
                        Briefing on Results of GAO Review of
                        TRANSCOM Turbo Y2K-Part A OPEVAL




                                                         Planning
Defense Test Criteria                                                 Result

Specify and validate test assumptions                                Partially satisfied
Establish a Y2K task force and assign responsibilities               Satisfied
Identify critical missions/tasks/systems                              Satisfied
Verify systems essential to mission are Y2K compliant/certified       Satisfied
Develop OPEVAL plan to guide event planning and execution             Satisfied
Identify and schedule CINC/Allied/Component/Agency support            Satisfied
Determine relevant and necessary resources (e.g., funding,            Satisfied
personnel, equipment),
Ensure approved Y2K contingency plans are prepared                    Satisfied
Develop risk management plan                                          Partially satisfied
Identify simulation needs and establish supporting environment Satisfied
Develop data collection and analysis plan or approaches               Satisfied


                                                                                            14




                        Page 27                          GAO/AIMD-00-21 TRANSCOM Y2K Operational Testing
                     Appendix I
                     Briefing on Results of GAO Review of
                     TRANSCOM Turbo Y2K-Part A OPEVAL




                                  Findings: Planning

Criteria: In planning for OPEVAL, CINCs are to define and validate
assumptions concerning the readiness of systems and the ability to
evaluate systems in light of real-world limitations.

Finding 1: To prevent potential corruption of live data during Turbo Y2K
Part A OPEVAL, TRANSCOM decided to simulate the operational “thin
line” systems environment.




                                                                                         15




                     Page 28                          GAO/AIMD-00-21 TRANSCOM Y2K Operational Testing
                          Appendix I
                          Briefing on Results of GAO Review of
                          TRANSCOM Turbo Y2K-Part A OPEVAL




                                       Findings: Planning
Finding 2: TRANSCOM determined that it would not be affordable or feasible
to include commercial transportation partners in its OPEVAL. Instead,
TRANSCOM decided to (1) limit the scope to its deployment management
capability, (2) rely on other Defense components to verify the Y2K readiness
of Defense transportation assets, and (3) assume the readiness of its
commercial partners’ assets (carriers and port operations). However, this
assumption was not based on assessment and verification of the partners’
readiness and was not stated in the OPEVAL plan. According to
TRANSCOM officials, this assumption and the limitations it imposed on the
scope of the OPEVAL was shared orally with the Joint Staff.

Finding 3: In response to our findings and recommendations, TRANSCOM
amended its 30-day reports to Joint Staff to disclose OPEVAL scope
limitations, and it strengthened its efforts to verify its assumption about the
readiness of commercial partners’ assets.

                                                                                              16




                          Page 29                          GAO/AIMD-00-21 TRANSCOM Y2K Operational Testing
                 Appendix I
                 Briefing on Results of GAO Review of
                 TRANSCOM Turbo Y2K-Part A OPEVAL




                               Findings: Planning
Criteria: A CINC Y2K Task Force composed of various
knowledgeable Y2K, test, and systems experts should be formed to
establish the base for all Y2K planning, coordination, execution, and
reporting.

Finding: TRANSCOM established a Y2K Task Force based on the
defined scope of OPEVAL and it defined roles and responsibilities
for each member. Members of the task force included Y2K subject
matter experts, surface and airlift experts, test and evaluation
experts from the Joint Interoperability Test Center (JITC) and the
Defense Office of Test and Evaluation, system analysts, component
command and service representatives, and a public affairs
representative.


                                                                                     17




                 Page 30                          GAO/AIMD-00-21 TRANSCOM Y2K Operational Testing
                  Appendix I
                  Briefing on Results of GAO Review of
                  TRANSCOM Turbo Y2K-Part A OPEVAL




                                  Finding: Planning
Criteria: CINCs need to analyze critical missions to determine the
most critical missions and identify the critical tasks supporting each
critical mission. In addition, the minimum number of integrated
automated information platforms/systems required to perform each
critical task must be identified (the “thin line”).

Finding: Consistent with its defined scope of the OPEVAL,
TRANSCOM identified 15 critical tasks that it needed to carry out its
management of a MTW deployment mission. In addition,
TRANSCOM identified a total of 23 “thin-line” systems that support
its capability to manage the deployment mission.




                                                                                      18




                  Page 31                          GAO/AIMD-00-21 TRANSCOM Y2K Operational Testing
                 Appendix I
                 Briefing on Results of GAO Review of
                 TRANSCOM Turbo Y2K-Part A OPEVAL




                                Finding: Planning
Criteria: Ensure that mission-critical “thin line” systems are
certified Y2K compliant.

Finding: Consistent with the defined scope of the OPEVAL,
TRANSCOM verified that all 23 mission-critical “thin line”
systems were certified Y2K compliant.




                                                                                     19




                 Page 32                          GAO/AIMD-00-21 TRANSCOM Y2K Operational Testing
                    Appendix I
                    Briefing on Results of GAO Review of
                    TRANSCOM Turbo Y2K-Part A OPEVAL




                                    Finding: Planning
Criteria: The Y2K task force should document how the Y2K
OPEVAL will be conducted, how data will be gathered and
analyzed, and what information the reports will contain.

Finding: TRANSCOM developed the Turbo Y2K Exercise
Directive and Test Plan to
 – ensure that mechanisms for evaluating critical dates and contingency
   plans for mission-critical systems are executed,
 – document participant roles and responsibilities,
 – link critical missions, critical tasks, architectures, test cases, and data
   elements, and
 – establish Y2K OPEVAL reporting requirements.




                                                                                        20




                    Page 33                          GAO/AIMD-00-21 TRANSCOM Y2K Operational Testing
                 Appendix I
                 Briefing on Results of GAO Review of
                 TRANSCOM Turbo Y2K-Part A OPEVAL




                               Findings: Planning
Criteria: When preparing for a Y2K OPEVAL, determine the
extent of participation of other CINCs, allies, components, and
agencies and coordinate their participation in the event.

Finding 1: TRANSCOM identified, coordinated, and scheduled
OPEVAL with CENTCOM, the transportation component
commands, Defense Logistics Agency, Defense Information
Systems Agency, Air Force, Army, and JITC to support the
OPEVAL.

Finding 2: TRANSCOM did not provide for participation in the
OPEVAL of commercial transportation service providers or the
commercial systems supporting air and water port operations
because neither was part of the defined OPEVAL “thin line.”

                                                                                     21




                 Page 34                          GAO/AIMD-00-21 TRANSCOM Y2K Operational Testing
                 Appendix I
                 Briefing on Results of GAO Review of
                 TRANSCOM Turbo Y2K-Part A OPEVAL




                                Finding: Planning

Criteria: The necessary resources (funding, personnel, training,
equipment, time frames, and external organization support)
should be identified and included in the plan.

Finding: TRANSCOM identified and planned for the necessary
resources. For example, it estimated a $10.4 million funding
requirement and earmarked the funds. In addition,
TRANSCOM coordinated the evaluation scenario and scripts
with all OPEVAL participants, acquired the systems hardware
and software to simulate a real-world MTW deployment
operation, and scheduled the appropriate staff to support Turbo
Y2K-Part A OPEVAL.




                                                                                     22




                 Page 35                          GAO/AIMD-00-21 TRANSCOM Y2K Operational Testing
                  Appendix I
                  Briefing on Results of GAO Review of
                  TRANSCOM Turbo Y2K-Part A OPEVAL




                              Findings: Planning
Criteria: To ensure that the defined Y2K OPEVAL exercise
objectives are met, it is essential to have contingency plans in
place prior to executing OPEVAL.

Finding 1: TRANSCOM ensured that approved contingency
plans for all specified mission-critical “thin line” systems had
been prepared prior to the start of OPEVAL.

Finding 2: During Turbo Y2K-Part A OPEVAL, TRANSCOM
invoked an approved contingency plan as a work-around for the
medical evacuation function. TRANSCOM officials recognized
that the medical evacuation systems were not Y2K compliant
and planned to use a contingency plan to address this
deficiency.

                                                                                      23




                  Page 36                          GAO/AIMD-00-21 TRANSCOM Y2K Operational Testing
                 Appendix I
                 Briefing on Results of GAO Review of
                 TRANSCOM Turbo Y2K-Part A OPEVAL




                               Findings: Planning
Criteria: CINC-unique risk management plans should be developed to
identify and mitigate system-related risks before they adversely affect
mission execution.

Finding 1: TRANSCOM developed a risk management plan that
identified risks and defined mitigation steps. For example, the plan
provided for determining the Y2K compliance of “thin line” systems
before conducting the OPEVAL and limiting OPEVAL to Y2K
compliant systems or systems with clearly delineated contingency
plans.




                                                                                     24




                 Page 37                          GAO/AIMD-00-21 TRANSCOM Y2K Operational Testing
                 Appendix I
                 Briefing on Results of GAO Review of
                 TRANSCOM Turbo Y2K-Part A OPEVAL




                               Findings: Planning

Finding 2: While the plan did not identify the exclusion of
commercial carriers and ports from OPEVAL as a risk,
TRANSCOM did initiate risk reduction steps by specifying that
confirmation letters should be sent to vendors whose systems
interface with TRANSCOM’s GTN to confirm the Y2K
compliance of these interfacing systems. However,
confirmation letters alone do not provide sufficient verification of
key business partners’ readiness. Moreover, limiting the
confirmation to systems that interface with GTN excludes the
providers’ critical systems that actually transport goods and
people, as well as the systems used to maintain these systems.




                                                                                     25




                 Page 38                          GAO/AIMD-00-21 TRANSCOM Y2K Operational Testing
                 Appendix I
                 Briefing on Results of GAO Review of
                 TRANSCOM Turbo Y2K-Part A OPEVAL




                               Findings: Planning

Finding 3: TRANSCOM reports that it recently expanded its
efforts to address the risk of excluding commercial carriers and
ports from OPEVAL. According to TRANSCOM, it is working
jointly with other federal agencies (e.g., Department of
Transportation) to determine the Y2K readiness of the civilian
transportation sector (air, sea, rail, and trucking) and plans to
develop risk mitigation strategies to address any Y2K
weaknesses in the sector. Further, TRANSCOM continues to
seek confirmation letter responses from GTN data exchange
partners (20 out of 27 received as of August 24, 1999), and, for
selected key commercial carrier partners, TRANSCOM plans to
verify readiness.




                                                                                     26




                 Page 39                          GAO/AIMD-00-21 TRANSCOM Y2K Operational Testing
                Appendix I
                Briefing on Results of GAO Review of
                TRANSCOM Turbo Y2K-Part A OPEVAL




                              Finding: Planning

Criteria: CINCs should (1) determine if simulations or manual
data input will be needed during the execution of the OPEVAL,
and, if needed, (2) ensure that an environment that can support
the simulation is planned for and acquired.

Finding: TRANSCOM officials identified the need for
simulations and manual data input and ensured that data
injection methodologies were included in the OPEVAL master
scenario events list (MSEL) and followed. For example, during
the evaluation of TRANSCOM’s ability to schedule air refueling
flights, TRANSCOM manually input data on fighter drag
requirements received from the Air Combat Command.



                                                                                    27




                Page 40                          GAO/AIMD-00-21 TRANSCOM Y2K Operational Testing
                 Appendix I
                 Briefing on Results of GAO Review of
                 TRANSCOM Turbo Y2K-Part A OPEVAL




                                Finding: Planning
Criteria: A plan should be prepared to help coordinate and
synchronize all OPEVAL data collection and assessment
activities.

Finding: Consistent with the specified scope of the OPEVAL,
TRANSCOM developed a data collection and analysis plan that
included, for example, (1) specific actions that should be
accomplished by OPEVAL participants prior to the start of and at
the completion of each OPEVAL, (2) ground rules for collecting
and documenting mission-critical system outputs, and (3)
direction on reviewing the critical tasks executed during
OPEVAL and determining the performance of the mission-critical
“thin line” systems.



                                                                                     28




                 Page 41                          GAO/AIMD-00-21 TRANSCOM Y2K Operational Testing
                        Appendix I
                        Briefing on Results of GAO Review of
                        TRANSCOM Turbo Y2K-Part A OPEVAL




                                                    Execution

Defense test criteria                                                Result

Conduct OPEVAL rehearsal                                             Satisfied

Follow configuration management policy                               Satisfied

Perform baseline test for OPEVAL                                     Satisfied

Execute required Y2K date rollover tests                             Satisfied
Collect and archive all Y2K-relevant data and ensure that            Satisfied
systems are reset to current day operations




                                                                                            29




                        Page 42                          GAO/AIMD-00-21 TRANSCOM Y2K Operational Testing
                    Appendix I
                    Briefing on Results of GAO Review of
                    TRANSCOM Turbo Y2K-Part A OPEVAL




                                Findings: Execution
Criteria: Prior to executing the Y2K OPEVAL, a rehearsal should be
conducted to ensure that all critical systems and interfaces identified
in the system architecture are operating correctly and that OPEVAL
staff know their roles and responsibilities.

Finding 1: TRANSCOM performed a rehearsal of the defined
OPEVAL from March 29 - April 2, 1999. During the rehearsal, (1) the
test readiness criteria were satisfied, (2) the data collection and
analysis methodology was verified, (3) the final baseline system
configuration was confirmed and documented, and (4) OPEVAL staff
practiced their roles and responsibilities.
Finding 2: A key staff member who participated in the Turbo Y2K-
Part A OPEVAL rehearsal did not participate in the actual execution
of critical date rollovers, which caused a schedule delay.
TRANSCOM officials were able to identify and correct the problem
with no significant impact on OPEVAL events.
                                                                                        30




                    Page 43                          GAO/AIMD-00-21 TRANSCOM Y2K Operational Testing
                 Appendix I
                 Briefing on Results of GAO Review of
                 TRANSCOM Turbo Y2K-Part A OPEVAL




                              Finding: Execution
Criteria: The systems configuration established for baseline testing
should not be changed unless authorized by the Y2K test director.

Finding: During our observation of Turbo Y2K-Part A OPEVAL, we
verified that changes made to the systems configuration were
authorized by the Y2K test director. We also confirmed that the
approved changes were made, tested, and documented in
accordance with configuration management policy.




                                                                                     31




                 Page 44                          GAO/AIMD-00-21 TRANSCOM Y2K Operational Testing
                 Appendix I
                 Briefing on Results of GAO Review of
                 TRANSCOM Turbo Y2K-Part A OPEVAL




                                Finding: Execution
Criteria: A baseline Y2K test should be executed to establish
expected results data that will be compared to output data
captured during the Y2K date rollover tests to help establish
whether or not a failure is Y2K-related.

Finding: TRANSCOM conducted a baseline Y2K test. During
this test (1) all critical tasks were performed, (2) data were
collected as outlined in the data collection and analysis plan, and
(3) automated logging systems were operationally verified.




                                                                                     32




                 Page 45                          GAO/AIMD-00-21 TRANSCOM Y2K Operational Testing
                 Appendix I
                 Briefing on Results of GAO Review of
                 TRANSCOM Turbo Y2K-Part A OPEVAL




                               Finding: Execution
Criteria: The defined mission-critical “thin line” system
configuration should be executed using normal operating
procedures and a seamless continuity of operations during
critical Y2K date rollovers should be observed.

Finding: According to TRANSCOM’s OPEVAL reports, all 23
mission-critical “thin line” systems that TRANSCOM defined for
this OPEVAL were executed under normal operating conditions
during OPEVAL with no significant interruptions during critical
Y2K date rollovers.




                                                                                     33




                 Page 46                          GAO/AIMD-00-21 TRANSCOM Y2K Operational Testing
                 Appendix I
                 Briefing on Results of GAO Review of
                 TRANSCOM Turbo Y2K-Part A OPEVAL




                                    Finding: Execution
Criteria: Ensure that all data needed to conduct the evaluation
for the Y2K case have been captured prior to resetting the
system to current day operations requirements.

Finding: Prior to resetting the systems to present day operational
conditions, TRANSCOM officials determined that (1) the “thin-
line” was completely exercised, (2) all items in the master
scenario events list were performed, and (3) all data needed to
make an assessment of TRANSCOM’s ability to perform its
MTW deployment mission were collected and archived.




                                                                                     34




                 Page 47                          GAO/AIMD-00-21 TRANSCOM Y2K Operational Testing
                        Appendix I
                        Briefing on Results of GAO Review of
                        TRANSCOM Turbo Y2K-Part A OPEVAL




                                                      Analysis
Defense test criteria                                                Result

Categorize, document, and report failures                            Satisfied

Determine mission impact of Y2K failures                             Satisfied

Ensure Y2K OPEVAL exit criteria are met                              Partially satisfied




                                                                                            35




                        Page 48                          GAO/AIMD-00-21 TRANSCOM Y2K Operational Testing
                  Appendix I
                  Briefing on Results of GAO Review of
                  TRANSCOM Turbo Y2K-Part A OPEVAL




                                  Finding: Analysis
Criteria: All failures are to be identified and properly categorized
as either “hard” or “soft” failures and should be documented and
reported in accordance with the data collection and analysis
plan.

Finding: TRANSCOM identified six Y2K failures during OPEVAL
and categorized one as a soft failure and five as hard failures.
All failures were documented in accordance with DOD Y2K
requirements and reported to the Joint Staff Y2K office.
Examples of the failures include the incorrect display of the year
2000 as 100 and an incorrect time stamp date after the century
date rollover.




                                                                                      36




                  Page 49                          GAO/AIMD-00-21 TRANSCOM Y2K Operational Testing
                 Appendix I
                 Briefing on Results of GAO Review of
                 TRANSCOM Turbo Y2K-Part A OPEVAL




                               Finding: Analysis
Criteria: Determine the impact of a system failure on the
accomplishment of a critical mission.

Finding: TRANSCOM determined that all six Y2K failures had no
impact on its ability to perform the deployment management
mission.




                                                                                     37




                 Page 50                          GAO/AIMD-00-21 TRANSCOM Y2K Operational Testing
                     Appendix I
                     Briefing on Results of GAO Review of
                     TRANSCOM Turbo Y2K-Part A OPEVAL




                                      Findings: Analysis
Criteria: JCS defined nine exit criteria that OPEVAL results should be
measured against to ensure that critical tasks and missions can be
performed in a Y2K environment.

Finding 1: TRANSCOM measured its OPEVAL performance against
the nine Joint Staff exit criteria and concluded in its May 1999 report
that all nine were satisfied and that critical tasks and missions can be
performed across the calendar and leap year dates with no significant
impact on its deployment mission.




                                                                                         38




                     Page 51                          GAO/AIMD-00-21 TRANSCOM Y2K Operational Testing
                     Appendix I
                     Briefing on Results of GAO Review of
                     TRANSCOM Turbo Y2K-Part A OPEVAL




                                      Findings: Analysis
Finding 2: During our August 24, 1999, briefing, we told TRANSCOM
that this conclusion reaches beyond, and is not justified by, the defined
scope of OPEVAL because (1) TRANSCOM’s ability to perform the
deployment mission depends extensively on commercial partners’ Y2K
readiness and (2) OPEVAL did not address, and TRANSCOM did not
otherwise know, whether its commercial partners’ systems could
satisfy the critical tasks involving them. Thus, we recommended that
TRANSCOM amend its OPEVAL reports to qualify its conclusion and
that it take steps, in collaboration with other federal agencies, to
ascertain the Y2K readiness of its critical commercial partners.




                                                                                         39




                     Page 52                          GAO/AIMD-00-21 TRANSCOM Y2K Operational Testing
                    Appendix I
                    Briefing on Results of GAO Review of
                    TRANSCOM Turbo Y2K-Part A OPEVAL




                                     Findings: Analysis
Finding 3: In response, TRANSCOM amended its OPEVAL Part A and
B 30-day reports to disclose that the scope of the OPEVALs was
limited to its ability to perform critical command, control, and
synchronization tasks in support of MTW operations in a Y2K
environment. The amended reports explicitly state that TRANSCOM’s
ability to perform its full deployment and sustainment missions requires
a yet-to-be confirmed assumption that the commercial transportation
sector can perform its part of the missions. Also, as described earlier,
TRANSCOM reports that it reemphasized and expanded its efforts to
determine the Y2K readiness of the commercial transportation sector in
general and selected key carriers in particular.




                                                                                        40




                    Page 53                          GAO/AIMD-00-21 TRANSCOM Y2K Operational Testing
                         Appendix I
                         Briefing on Results of GAO Review of
                         TRANSCOM Turbo Y2K-Part A OPEVAL




                                                     Reporting

Defense test criteria                                                 Result

Prepare Y2K reports describing mission impact and readiness           Partially satisfied

Provide reports to Joint Staff J7 within required time frames         Satisfied




                                                                                             41




                         Page 54                          GAO/AIMD-00-21 TRANSCOM Y2K Operational Testing
                      Appendix I
                      Briefing on Results of GAO Review of
                      TRANSCOM Turbo Y2K-Part A OPEVAL




                                 Findings: Reporting
Criteria: CINCs are to prepare Y2K reports describing mission impact and
readiness.

Finding 1: TRANSCOM provided its May 1999 Y2K reports to Joint Staff,
which concluded that all critical tasks supporting MTW unit deployment can
be performed with no significant impact on readiness caused by potential
Y2K failures.

Finding 2: The May 1999 OPEVAL reports did not describe the mission
impact of not evaluating or otherwise knowing the Y2K readiness of the
commercial transportation providers and port operations. However,
TRANSCOM has since amended its OPEVAL Part A and B 30-day reports,
as we recommended, to disclose that its missions would be adversely
affected if the commercial transportation sector was unable to perform its
very important role in TRANSCOM’s deployment and sustainment missions.

                                                                                          42




                      Page 55                          GAO/AIMD-00-21 TRANSCOM Y2K Operational Testing
                 Appendix I
                 Briefing on Results of GAO Review of
                 TRANSCOM Turbo Y2K-Part A OPEVAL




                              Finding: Reporting
Criteria: A preliminary report is required within 7 calendar days
after the completion of OPEVAL and a final report is required
within 30 calendar days. Both reports are to be provided to Joint
Staff.

Finding: TRANSCOM completed both reports within the required
time frames and provided them to Joint Staff.




                                                                                     43




                 Page 56                          GAO/AIMD-00-21 TRANSCOM Y2K Operational Testing
                          Appendix I
                          Briefing on Results of GAO Review of
                          TRANSCOM Turbo Y2K-Part A OPEVAL




                                                      Conclusions
•   TRANSCOM satisfied many of the Defense OPEVAL requirements for its
    defined “thin line.”

•   However, key steps that are pivotal to (1) fully disclosing and appropriately
    addressing any risks associated with limitations in the defined scope of
    OPEVAL and (2) accurately reporting on mission readiness and impacts,
    were not fully satisfied. The result is that the Y2K readiness of critical tasks
    associated with conducting a MTW deployment--namely the use of
    commercial carriers and port operations to deploy goods and people--were
    not known with sufficient surety to support TRANSCOM’s May 1999
    reported position that it can conduct a MTW deployment in a Y2K
    environment with no material impact on operations.

•   TRANSCOM has since either implemented or initiated appropriate actions
    to address our recommendations to correct these OPEVAL limitations. As a
    result, we are not making any further recommendations at this time.
                                                                                              44




                          Page 57                          GAO/AIMD-00-21 TRANSCOM Y2K Operational Testing
Appendix II

Objectives, Scope, and Methodology                                                        Appendx
                                                                                                Ii




              At the request of the Chairman, Subcommittee on Defense, House
              Committee on Appropriations, we selected the TRANSCOM Turbo Y2K-
              Part A OPEVAL for review to determine (1) whether the OPEVAL was
              planned, executed, and documented in accordance with DOD and GAO
              Year 2000 testing guidelines and (2) what the OPEVAL results indicated
              concerning readiness and risks. This OPEVAL was selected in collaboration
              with the DOD Inspector General to ensure appropriate coverage of all
              CINC OPEVALs and no duplication of effort.

              To meet our first objective, we reviewed TRANSCOM’s OPEVAL plan,
              testing documentation and records, and test results and associated reports.
              We also interviewed TRANSCOM officials responsible for Y2K OPEVAL
              planning, execution, and reporting tasks. Further, we observed the century
              date rollover test for the MTW deployment execution function and
              compared TRANSCOM’s OPEVAL planning, execution, analysis, and
              reporting actions against JCS OPEVAL guidance and our Y2K testing guide.

              To meet our second objective, we reviewed TRANSCOM’s OPEVAL results,
              including its 7- and 30-day reports and system problem tracking reports. We
              also interviewed TRANSCOM officials and analysts responsible for
              developing OPEVAL assessment methodologies, interpreting evaluation
              metrics, and ensuring that evaluation exit criteria were met.

              We requested and received comments on a draft of our briefing, which was
              delivered on August 24, 1999. We incorporated those comments where
              appropriate. We performed our audit work from March through September
              1999 in accordance with generally accepted government auditing
              standards.




              Page 58                       GAO/AIMD-00-21 TRANSCOM Y2K Operational Testing
Appendix III

GAO Contact and Staff Acknowledgements                                                            AppendxIi




GAO Contact            Randolph C. Hite, (202) 512-6240




Acknowledgements       In addition to those named above, Ronald B. Bageant, Cristina T. Chaplain,
                       Richard B. Hung, Denice M. Millett, and Karen S. Sifford made key
                       contributions to this report.




(511660)       Leter   Page 59                      GAO/AIMD-00-21 TRANSCOM Y2K Operational Testing
Ordering Information

The first copy of each GAO report and testimony is free.
Additional copies are $2 each. Orders should be sent to the
following address, accompanied by a check or money order made
out to the Superintendent of Documents, when necessary, VISA and
MasterCard credit cards are accepted, also.

Orders for 100 or more copies to be mailed to a single address are
discounted 25 percent.

Orders by mail:

U.S. General Accounting Office
P.O. Box 37050
Washington, DC 20013

or visit:

Room 1100
700 4th St. NW (corner of 4th and G Sts. NW)
U.S. General Accounting Office
Washington, DC

Orders may also be placed by calling (202) 512-6000
or by using fax number (202) 512-6061, or TDD (202) 512-2537.

Each day, GAO issues a list of newly available reports and
testimony. To receive facsimile copies of the daily list or any list
from the past 30 days, please call (202) 512-6000 using a touchtone
phone. A recorded menu will provide information on how to obtain
these lists.

For information on how to access GAO reports on the INTERNET,
send an e-mail message with “info” in the body to:

info@www.gao.gov

or visit GAO’s World Wide Web Home Page at:

http://www.gao.gov
United States                       Bulk Rate
General Accounting Office      Postage & Fees Paid
Washington, D.C. 20548-0001           GAO
                                 Permit No. GI00
Official Business
Penalty for Private Use $300

Address Correction Requested