oversight

Defense Computers: U.S. Space Command's Management of Its Year 2000 Operational Testing

Published by the Government Accountability Office on 1999-11-15.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                 United States General Accounting Office

GAO              Report to the Chairman of the
                 Subcommittee on Defense, Committee on
                 Appropriations, House of
                 Representatives

November 1999
                 DEFENSE
                 COMPUTERS

                 U.S. Space Command’s
                 Management of Its
                 Year 2000 Operational
                 Testing




GAO/AIMD-00-30
Contents



Letter                                                                                   3


Appendixes   Appendix I: Briefing on Results of GAO Review of
               SPACECOM Space Control Y2K OPEVAL                                        14
             Appendix II:   Objectives, Scope, and Methodology                          67
             Appendix III: GAO Contact and Staff Acknowledgements                       68


Tables       Table 1: Summary of JCS Year 2000 Operational Evaluation
               Criteria                                                                  7
             Table 2: Summary of Space Command’s Satisfaction of JCS
               Evaluation Criteria for the Space Control Evaluation                      8




             Abbreviations

             DOD            Department of Defense
             JCS            Joint Chiefs of Staff
             OPEVAL         operational evaluation
             SPACECOM       U.S. Space Command
             Y2K            Year 2000



             Page 1                    GAO/AIMD-00-30 Space Command Y2K Operational Testing
Page 2   GAO/AIMD-00-30 Space Command Y2K Operational Testing
United States General Accounting Office                                                       Accounting and Information
Washington, D.C. 20548                                                                             Management Division



                                    B-282546                                                                                 Leter




                                    November 15, 1999

                                    The Honorable Jerry Lewis
                                    Chairman, Subcommittee on Defense
                                    Committee on Appropriations
                                    House of Representatives

                                    Dear Mr. Chairman:

                                    Complete and thorough end-to-end testing is essential to provide
                                    reasonable assurance that new or modified systems used to collectively
                                    support a core business function or mission operation will not jeopardize
                                    an organization’s ability to deliver products and services as a result of the
                                    Year 2000 (Y2K) computing problem. This is especially true for the
                                    Department of Defense (DOD) because it relies on a complex and broad
                                    array of interconnected computer systems—including weapons, command
                                    and control, satellite, inventory management, transportation management,
                                    health, financial, personnel and payment systems—to carryout its military
                                    operations and supporting business functions.

                                    At your request, we reviewed DOD’s management of various Year 2000-
                                    related end-to-end testing activities. As part of our efforts, we assessed the
                                    U.S. Space Command’s management of its end-to-end test of space control
                                    systems essential to major theater war (one of 16 operational evaluations
                                    for the command) and determined what the results of this test show with
                                    respect to operational risks and readiness. 1 We briefed Space Command
                                    officials on our findings on October 1, 1999, and made a recommendation
                                    to correct the management weaknesses that we found. Space Command
                                    immediately acted to address our recommendation. We then briefed your
                                    office on our findings and Space Command’s actions to address our
                                    recommendation on November 1, 1999. The purpose of this letter is to
                                    summarize our briefing to your office. The briefing slides that we presented
                                    to your office are in appendix I, and the objectives, scope, and
                                    methodology of our review are detailed in appendix II. Space Command
                                    provided oral comments on our briefing slides, and we have incorporated
                                    them as appropriate. We performed our audit work from March through



                                    1
                                        DOD refers to its combatant commands’ end-to-end tests as operational evaluations.




                                    Page 3                             GAO/AIMD-00-30 Space Command Y2K Operational Testing
                   B-282546




                   October 1999 in accordance with generally accepted government auditing
                   standards.



Results in Brief   Year 2000 end-to-end testing is an essential component of an effective Year
                   2000 testing program since Y2K-related problems can affect so many of the
                   systems owned and operated by an entity as well as systems belonging to
                   business partners and infrastructure providers. Moreover, to be effective,
                   end-to-end testing should be approached in a structured and disciplined
                   fashion. Both the Joint Chiefs of Staff (JCS) guidance to its combatant
                   commands on managing Year 2000 operational evaluations,2 (the term JCS
                   uses to refer to Year 2000 end-to-end testing) and our Year 2000 test
                   guidance3 define a number of key management controls to employ when
                   planning, executing, analyzing, and reporting on such test and evaluation
                   events.

                   We found that Space Command’s space control operational evaluation
                   satisfied 16 of 21 of the key processes prescribed by JCS guidance. For
                   example, the Command established a Y2K task force to guide the
                   evaluation effort, which included satellite/system specialists, test and
                   evaluation experts, system analysts, military component and service
                   representatives, and public affairs representatives. Further, the Command
                   performed a rehearsal before conducting the evaluation to ensure that all
                   critical systems and interfaces were operating correctly and that all staff
                   knew their roles and responsibilities.

                   In response to our concerns, Space Command has taken positive actions to
                   address the remaining five key processes. Three of the key processes were
                   addressed during the course of our review and two were addressed in
                   response to a recommendation we made at our briefing. During the course
                   of our review, Space Command began ensuring that contingency plans
                   were in place for its mission-critical systems, which it had not done before
                   conducting the space control operational evaluation. Also, after we found
                   that configuration management procedures were not always followed




                   2
                       Joint Staff Year 2000 Operational Evaluation Guide, Version 3.0, April 1, 1999.
                   3
                    Year 2000 Computing Crisis: A Testing Guide (GAO/AIMD-10.1.21, issued as an exposure
                   draft in June 1998; issued in final in November 1998).




                   Page 4                              GAO/AIMD-00-30 Space Command Y2K Operational Testing
             B-282546




             while executing the evaluation,4 Space Command initiated an effort to
             ensure that such procedures are followed in future evaluations. In addition,
             during our review, the Command amended its report to discuss its decision
             to exclude six communications systems from the evaluation and whether
             this adversely impacted the ability to draw conclusions about mission
             readiness.

             At the time of our October 1, 1999, briefing, Space Command still needed to
             address two partially satisfied key processes, which included (1) not
             documenting whether test cases for most intelligence systems met
             performance exit criteria and (2) not ensuring that 1 of 29 systems included
             in the evaluation was Y2K compliant. We therefore recommended that
             Space Command amend its final report to JCS to recognize the
             uncertainties and risks associated with its failure to take these steps and
             the actions underway or planned to address these uncertainties and risks.
             Without taking these steps, Space Command could not adequately know
             the Year 2000 readiness of critical tasks—collecting surveillance and
             intelligence data to disseminate warning messages—associated with
             conducting the space control mission. Because Space Command has
             subsequently amended its final report and plans to ensure that these
             weaknesses are not repeated in a November operational evaluation of its
             intelligence mission, we are not making further recommendations at this
             time.



Background   Space Command’s mission is to provide direct support to combatant
             commanders and military forces through the use of space-based satellites
             and other technologies needed for navigation, surveillance and
             reconnaissance, communications, environmental and attack warnings
             during war and peacetime operations. To perform this mission, Space
             Command relies on a wide array of information technology systems,
             including command and control systems, geographically dispersed radar
             sites, satellites, communications networks, and intelligence systems.

             In August 1998, the Secretary of Defense directed JCS to require its
             combatant commands, including Space Command, to plan, execute,


             4
              Configuration management involves establishing product baselines and systematically
             controlling changes made to those baselines. Without an effective configuration
             management process, organizations can lose control of the software product, potentially
             produce and use inconsistent product versions, and create operational problems.




             Page 5                          GAO/AIMD-00-30 Space Command Y2K Operational Testing
B-282546




analyze, and report on a series of simulated Year 2000 operational
evaluations. The evaluations, which were to assess whether DOD can
continue to perform critical military operations in a Year 2000 environment,
are one of three DOD end-to-end testing efforts.5

The purpose of end-to-end testing is to verify that a defined set of
interrelated systems, which collectively support an organizational core
business area or function, interoperate as intended in an operational
environment (either actual or simulated). These interrelated systems
include not only those owned and managed by an organization, but also the
external systems with which they interface or that otherwise support the
business area or function. The combatant commands’ core business areas
or functions are referred to as “thin lines.”

The boundaries for end-to-end tests can vary depending on a given business
function’s system dependencies and criticality to the organizational
mission. Therefore, in managing end-to-end test activities, it is important to
analyze the interrelationships among core business functions and their
supporting systems and the mission impact and risk of date-induced system
failures and to use these analyses to define test boundaries. It is also
important to work early and continually with functional partners to ensure
that related end-to-end test activities are effectively coordinated and
integrated. Table 1 summarizes key processes recommended by JCS’ Year
2000 operational evaluation guidance, which is consistent with our Year
2000 test guide.




5
 In addition to conducting operational evaluations, the military services are conducting
system integration testing, and the functional business areas, such as personnel and health
affairs, are conducting functional end-to-end tests. Each of these end-to-end testing
activities is discussed in detail in Defense Computers: Management Controls Are Critical to
Effective Year 2000 Testing (GAO/AIMD-99-172, June 30, 1999).




Page 6                          GAO/AIMD-00-30 Space Command Y2K Operational Testing
B-282546




Table 1: Summary of JCS Year 2000 Operational Evaluation Criteria


Planning    • Specify test assumptions and limitations
            • Establish a Year 2000 task force
            • Identify critical missions/tasks/systems
            • Verify that systems essential to mission are Year 2000 compliant
            • Develop an operational evaluation plan to guide event planning and
              execution
            • Identify and schedule support from other commands, DOD components,
              etc.
            • Determine relevant and necessary resources (e.g., funding, personnel,
              equipment, etc.)
            • Ensure approved Year 2000 contingency plans are prepared
            • Develop a risk management plan
            • Identify simulation needs and establish supporting testing environment
            • Develop data collection and analysis plan or approaches
Execution   • Conduct operational evaluation rehearsal
            • Follow configuration management policy
            • Perform baseline test for operational evaluation
            • Execute required Year 2000 date rollover tests
            • Collect and archive all Year 2000-relevant data and ensure that systems
              are reset to current day operations
Analysis    • Categorize, document, and report Year 2000 failures
            • Determine mission impact of Year 2000 failures
            • Ensure exit criteria are met
Reporting   • Prepare Year 2000 reports describing mission impact and readiness
            • Provide reports to JCS within required timeframes



Space Command has already completed 16 operational evaluations to
assess its ability to manage and provide combatant support during a major
theater war. These evaluations covered seven mission areas, including
(1) integrated tactical warning and attack assessment, (2) space control,
(3) force enhancement, (4) weather support, (5) command and control of
space forces, (6) space operations support, and (7) space lift. The space
control mission area provides (1) surveillance support to monitor, track,
identify, and catalog all orbiting space objects for collision avoidance and
(2) protection support to monitor, detect, assess, characterize, track, and
issue warnings about threats, both natural and man-made, against United




Page 7                         GAO/AIMD-00-30 Space Command Y2K Operational Testing
                       B-282546




                       States and allied space systems. The space control evaluation was
                       executed between March 11 and March 25, 1999.



Space Command          As noted in table 2 below, we found that, for its space control operational
                       evaluation, Space Command satisfied the majority of the management
Implemented Most       process controls (16 of 21) specified in JCS’ operational evaluation
Important Management   guidance.
Processes During Its
Space Control
Evaluation             Table 2: Summary of Space Command’s Satisfaction of JCS Evaluation Criteria for
                       the Space Control Evaluation

                                                                Number of primary      Number of primary
                       Phases                                    evaluation criteria     criteria satisfied
                       Planning                                                  11                      9
                       Execution                                                  5                      4
                       Analysis                                                   3                      2
                       Reporting                                                  2                      1
                       Total                                                     21                     16



                       Consistent with JCS guidance governing operational evaluation planning,
                       Space Command established a Year 2000 task force, which included
                       satellite/system specialists, test and evaluation experts, system analysts,
                       military component and service representatives, and public affairs
                       representatives. It identified 35 critical tasks that it needed to carry out the
                       space control mission in support of a major theater war. Space Command
                       also issued a directive to ensure testing resources would be made available
                       for operational evaluations and earmarked about $8 million for operational
                       evaluation activities—including the space control evaluation. Further,
                       Space Command developed a test plan that documented participant roles
                       and responsibilities, critical missions and tasks, test cases, and reporting
                       requirements.

                       Space Command also took effective steps in executing, analyzing, and
                       reporting on its evaluation. For instance, before executing the operational
                       evaluation, Space Command performed a rehearsal to ensure that all
                       critical systems and interfaces were operating correctly and that all staff
                       knew their roles and responsibilities. Before resetting systems to current
                       day operations, Space Command ensured that thin line systems were



                       Page 8                      GAO/AIMD-00-30 Space Command Y2K Operational Testing
                          B-282546




                          assessed, master scenario events were performed and deviations were
                          identified, and that all data needed to make an assessment of the
                          command’s ability to perform the space control mission were collected and
                          archived.



Space Command Acted       Following its operational evaluation, Space Command took action to
                          resolve three partially satisfied key processes. In doing so, it increased its
to Address Three          assurance with respect to the Y2K readiness of space control critical tasks
Partially Satisfied Key   involving intelligence and communications systems.
Processes                 First, before conducting its test, Space Command did not verify that
                          contingency plans were in place for the 29 systems included in the
                          evaluation. Instead, Space Command relied exclusively on system owners
                          to do so. As noted in JCS testing guidance, contingency plans identify
                          alternative systems or workaround procedures to use when performing a
                          mission in the event of a system disruption. As such, JCS guidance states
                          that it is essential that commands ensure that these plans are in place prior
                          to executing the operational evaluation so that they can be invoked in the
                          case of system failure. Subsequent to the evaluation, Space Command
                          began verifying that contingency plans are in place for its mission-critical
                          systems.

                          Second, while executing the evaluation, Space Command did not follow
                          configuration management procedures. JCS guidance specifies that system
                          configurations not be changed during testing unless authorized by the test
                          director. During the space control evaluation, changes were made to one
                          system after the baseline for the evaluation was established and without
                          authorization from the test director. These changes contributed to a “hard”
                          failure during testing. (Information on the nature of the system failure is
                          classified).6 After the evaluation, Space Command directed the 17th Test
                          Squadron and intelligence unit to review this deviation and its impact on
                          the command’s ability to determine mission readiness. On September 30,
                          1999, the intelligence unit and 17th Test Squadron reported that the
                          deviation did not materially affect mission readiness. To prevent similar
                          problems in future evaluations, Space Command directed the 17th Test
                          Squadron and intelligence unit to develop ways to improve testing


                          6
                           A “hard” failure is a Y2K-related failure that results in an obvious adverse impact to the
                          system. For example, the system shuts down, displays erroneous data, or performs other
                          unexpected actions.




                          Page 9                           GAO/AIMD-00-30 Space Command Y2K Operational Testing
                       B-282546




                       documentation and procedures with a special focus on ensuring that
                       documentation standards, configuration management procedures, and
                       baseline test requirements are followed.

                       Third, in reporting on the evaluation, Space Command did not specify how
                       its exclusion of six communications systems from the test impacted its
                       ability to draw conclusions about mission readiness. When planning the
                       evaluation, Space Command concluded that it would not include six
                       communications systems in the evaluation due to resource constraints or
                       because the systems were to be included in a future evaluation. As a result,
                       Space Command assumed that communications systems would be
                       available to perform critical tasks and disseminate time-sensitive warnings
                       to combatant commanders. While Space Command communicated this
                       assumption to JCS in its operational evaluation plan, it did not report on
                       how this scope limitation could adversely affect its ability to draw
                       conclusions about mission readiness. Instead, Space Command reported to
                       JCS that critical space control tasks could be performed across the
                       calendar and leap year dates with no significant impact on its mission
                       readiness. Space Command has since ensured that omitted
                       communications systems were included in other Year 2000 end-to-end
                       testing or operational evaluation events and disclosed this limitation in its
                       final report on the evaluation.



Space Command Is       At the time of our October 1, 1999, briefing, Space Command had not yet
                       addressed two partially satisfied key processes. First, in planning the
Acting to Address      evaluation, Space Command did not ensure that one intelligence system to
Recommendation         be tested was certified as compliant. Rather, it only verified that the
                       software application relevant to the evaluation was compliant. Year 2000
Made at the Briefing   compliance of an application in isolation is of very limited value unless the
                       system platform that it runs on, as well as other applications operating on
                       the system, is compliant. As such, both JCS guidance and GAO’s end-to-end
                       test guidance define system, not application, compliance as a precondition
                       to end-to-end testing.

                       Second, Space Command did not document whether intelligence systems
                       met system performance exit criteria for all test cases. Specifically, the
                       command was supposed to show whether it could process a predetermined
                       number of transactions within specific time constraints. While command
                       officials contend that this was done, only one-fifth of the transactions for
                       intelligence critical tasks were documented. Space Command officials




                       Page 10                    GAO/AIMD-00-30 Space Command Y2K Operational Testing
             B-282546




             stated that it was too time-consuming for operators to print screens for
             these tasks during the evaluation.

             At our briefing, we recommended that Space Command amend its final
             report to JCS to recognize the (1) uncertainties and risks associated with
             its failure to fully satisfy these criteria and (2) the actions it had underway
             or planned to address these uncertainties and risks. Space Command
             agreed with this recommendation. It plans to amend its final report to
             disclose these limitations and to pursue an alternative data collection
             strategy for its planned November 1999 operational evaluation of its
             intelligence mission in order to verify that intelligence systems/tasks fully
             meet performance criteria.



Conclusion   By acting swiftly to address our recommendation, made during the
             October 1, 1999, briefing, Space Command has demonstrated its
             commitment to improving management controls over Year 2000 testing
             activities and the effectiveness and value of its operational evaluation as
             well as mitigated the risks associated with being able to operate effectively
             in the Year 2000. Further, it has ensured that DOD managers have complete
             and reliable information to use in making informed military decisions. As a
             result, Space Command has satisfied the intent of our recommendation,
             and we are not making any further recommendations at this time.


             We are sending copies of this report to Representative John P. Murtha,
             Ranking Minority Member, Subcommittee on Defense, House
             Appropriations Committee; Senator John Warner, Chairman, and Senator
             Carl Levin, Ranking Minority Member, Senate Committee on Armed
             Services; Senator Ted Stevens, Chairman, and Senator Daniel Inouye,
             Ranking Minority Member, Subcommittee on Defense, Senate Committee
             on Appropriations; and Representative Floyd Spence, Chairman, and Ike
             Skelton, Ranking Minority Member, House Committee on Armed Services.

             We are also sending copies to the Honorable John Koskinen, Chair of the
             President’s Year 2000 Conversion Council; the Honorable William Cohen,
             Secretary of Defense; the Honorable John Hamre, Deputy Secretary of
             Defense; General Henry Shelton, Chairman of the Joint Chiefs of Staff;
             Arthur Money, Assistant Secretary of Defense for Command, Control,
             Communications, and Intelligence; and the Honorable Jacob Lew, Director,
             Office of Management and Budget. Copies will also be made available to
             others upon request.



             Page 11                     GAO/AIMD-00-30 Space Command Y2K Operational Testing
B-282546




Should you or your staff have any questions concerning this report, please
contact me at (202) 512-6240. I can also be reached by e-mail at
brockj.aimd@gao.gov. Other points of contact and key contributors to this
report are listed in appendix III.

Sincerely yours,




Jack L. Brock, Jr.
Director, Governmentwide and Defense
 Information Systems




Page 12                   GAO/AIMD-00-30 Space Command Y2K Operational Testing
Page 13   GAO/AIMD-00-30 Space Command Y2K Operational Testing
Appendix I

Briefing on Results of GAO Review of
SPACECOM Space Control Y2K OPEVAL                                                  Appendx
                                                                                         Ii




          Results of GAO Review of
     SPACECOM Space Control Y2K OPEVAL

             House Appropriations Committee

                    November 1, 1999

                                                                         1




                      Page 14   GAO/AIMD-00-30 Space Command Y2K Operational Testing
                     Appendix I
                     Briefing on Results of GAO Review of
                     SPACECOM Space Control Y2K OPEVAL




                                             Introduction

•   In August 1998, the Secretary of Defense directed the Commanders-
    in-Chief (CINC), who are responsible for Defense’s unified combatant
    commands, to plan, execute, and report on a series of simulated Year
    2000 operational evaluations (Y2K OPEVALs).

•   The CINC Y2K OPEVALs are one of three Defense Y2K end-to-end
    test and evaluation efforts. GAO’s Y2K Test Guide advocates end-to-
    end testing, which is testing performed to verify that a defined set of
    interrelated systems (i.e., systems that collectively support an
    organizational core business function or operation) interoperate as
    intended in a Y2K environment.

    The CINC core business functions/operations are referred to as “thin
    lines.” The “thin lines” consist of critical tasks, as well as systems that
    perform critical tasks.


                                                                                           2




                     Page 15                      GAO/AIMD-00-30 Space Command Y2K Operational Testing
                   Appendix I
                   Briefing on Results of GAO Review of
                   SPACECOM Space Control Y2K OPEVAL




                                              Objectives

•   At the request of the Chairman, House Appropriations Committee,
    Defense Subcommittee, GAO is reviewing selected OPEVALs to
    determine:

    (1) if the OPEVAL was planned, executed, and documented in
    accordance with DOD guidelines, and

    (2) what the OPEVAL results indicated concerning readiness and risks.

•   The OPEVALS reviewed by GAO included those conducted by
    Space and Transportation Commands and were selected in
    collaboration with the Defense Inspector General (IG) to ensure:
     – appropriate coverage of all CINC OPEVALs, and
     – no duplication of effort.
                                                                                         3




                   Page 16                      GAO/AIMD-00-30 Space Command Y2K Operational Testing
                    Appendix I
                    Briefing on Results of GAO Review of
                    SPACECOM Space Control Y2K OPEVAL




                              Scope and Methodology
This briefing addresses the Space Command OPEVAL for Space
Control. To satisfy objective (1), we

•   reviewed the OPEVAL plan, testing documents/records, and test
    results/reports;

•   interviewed Space Command officials responsible for Y2K
    OPEVAL planning, execution, and reporting tasks; and

•   compared Space Command’s planning, execution, analysis, and
    reporting actions with Defense OPEVAL guidance.




                                                                                          4




                    Page 17                      GAO/AIMD-00-30 Space Command Y2K Operational Testing
                        Appendix I
                        Briefing on Results of GAO Review of
                        SPACECOM Space Control Y2K OPEVAL




                                  Scope and Methodology
•   To satisfy objective (2), we

     • reviewed Space Command’s OPEVAL results, 7-and 30-day
       reports, and system problem tracking reports and
     • interviewed Space Command officials and analysts responsible for
       developing OPEVAL assessment methodologies, interpreting
       evaluation metrics, and ensuring that evaluation exit criteria were
       met.

•   On October 1, 1999, we briefed Space Command leadership on the
    results of our review and made a recommendation to address our
    findings. In agreeing to our recommendation, Space Command has
    taken action to address weaknesses identified during the review and
    plans to amend its final OPEVAL report accordingly.

•   We performed our work from March 1999 through October 1999 in
    accordance with generally accepted government auditing standards.
                                                                                              5




                        Page 18                      GAO/AIMD-00-30 Space Command Y2K Operational Testing
                         Appendix I
                         Briefing on Results of GAO Review of
                         SPACECOM Space Control Y2K OPEVAL




                                                 Background
DOD OPEVAL Overview

•   To assist the CINCs in planning, documenting, executing,
    analyzing, and reporting OPEVALs, the Joint Staff issued OPEVAL
    guidance. The guidance is divided into phases:
     –   planning,
     –   execution,
     –   analysis, and
     –   reporting

•   The OPEVAL guidance is consistent with GAO’s end-to-end testing
    guidance and DOD’s Y2K management plan.




                                                                                               6




                         Page 19                      GAO/AIMD-00-30 Space Command Y2K Operational Testing
                          Appendix I
                          Briefing on Results of GAO Review of
                          SPACECOM Space Control Y2K OPEVAL




                                                Background
Space Command OPEVAL Events’ Status

•   Space Command is responsible for providing continuous and real-time
    (1) warnings of air or space attacks against North America and (2) space
    control, surveillance, communications, and intelligence support to military
    operations worldwide.

•   To fulfill its missions, Space Command depends heavily on information
    systems and technology, including satellites, geographically dispersed
    radars/sensors, ground relay terminals/stations, and communication
    networks.




                                                                                                7




                          Page 20                      GAO/AIMD-00-30 Space Command Y2K Operational Testing
                                Appendix I
                                Briefing on Results of GAO Review of
                                SPACECOM Space Control Y2K OPEVAL




                                                       Background

•   Space Command identified 7 “thin lines” (missions) to be
    operationally evaluated: (1) Integrated Tactical Warning and Attack
    Assessment, (2) Space Control, (3) Force Enhancement,
    (4) Weather Support, (5) Command and Control (C2) of Space
    Forces, (6) Space Operations Support, and (7) Space Lift.

•   For the 7 “thin lines”, Space Command identified 92 critical tasks1
    and 86 supporting systems.

•   The following table describes the status of Space Command’s 16
    OPEVALs and the Chairman’s Contingency Assessment (CCA).


    1 The  number does not include critical tasks for the first two OPEVALS because critical tasks were not
    identified in reports for the first two OPEVALS.

                                                                                                         8




                                Page 21                       GAO/AIMD-00-30 Space Command Y2K Operational Testing
                                      Appendix I
                                      Briefing on Results of GAO Review of
                                      SPACECOM Space Control Y2K OPEVAL




                                                             Background
 Evaluation Events      “Thin Line”             Schedule           Results
 OPEVAL – North         Integrated Tactical     December 2-4,      Reported mission ready
 American               Warning and Attack      1998               0 “soft”1 failures
 Aerospace Defense      Assessment                                 0 “hard” 2 failures
 Command                (ITWAA)
 (NORAD)
 OPEVAL – NORAD         ITWAA                   February 16-28, Reported mission ready
                                                1999            2 “soft” failures
                                                                1 “hard” failure
 OPEVAL                 Space Control           March 15-25,    Reported mission ready
                                                  1999          2 “soft”1 failures
                                                                3 “hard”2 failures
 OPEVAL – Central       Command and             April 6-12, 1999   Reported mission ready
 Command                Control (C2) –                             1 “soft”1 failure
                        Space Forces                               0 “hard”2 failures
1
  A “soft” failure is a Y2K-related failure that is not immediately discernable. The effect may be cumulative
   and require several hours, days, or longer to manifest itself.
2
  A “hard” failure is a Y2K-related failure that results in an obvious adverse impact to the system. For
   example, the system shuts down, erroneous data is displayed, or unexpected actions occur.
3
  Information classified by the Department of Defense.




                                                                                                                9




                                      Page 22                       GAO/AIMD-00-30 Space Command Y2K Operational Testing
                                  Appendix I
                                  Briefing on Results of GAO Review of
                                  SPACECOM Space Control Y2K OPEVAL




                                                           Background
Evaluation Events   “Thin Line”             Schedule         Results
OPEVAL – Central    Global Positioning      April 23-May 1, Reported mission ready
Command             System - Space          1999            0 “soft” failures
                    Support/Space                           0 “hard” failures
                    Force Enhancement
OPEVAL – Central    Theater Ballistic       April 22-30,     Reported mission ready
Command             Missile Warning         1999             0 “soft” failures
                    (TBMW) – Space                           1 “hard” failure
                    Support/Space
                    Force Enhancement
OPEVAL – Central    Space Support/          May 1-2 &        Reported mission ready
Command             Satellite Control       May 10-11,       3 “soft”1 failures
                                              1999           0 “hard”2 failures
OPEVAL – Central    Terrestrial Weather     May 10-17,       Reported mission ready
Command              - Space Support/         1999           1 “soft”1 failure
                     Space Force                             0 “hard”2 failures
                     Enhancement




                                                                                                      10




                                  Page 23                      GAO/AIMD-00-30 Space Command Y2K Operational Testing
                                      Appendix I
                                      Briefing on Results of GAO Review of
                                      SPACECOM Space Control Y2K OPEVAL




                                                             Background
    Evaluation Events   “Thin Line”               Schedule                   Results
    OPEVAL              Communications –          June 9-15, 1999            Reported mission ready
                        Space Operations                                     0 “soft”1 failures
                        Support                                              0 “hard”2 failures
    CCA4                Intelligence,             June 14-18,1999            Classified3
                        Surveillance, &
                        Reconnaissance
    OPEVAL              Space Weather             June 19-July 14, 1999      Reported mission ready
                                                                             0 “soft”1 failures
                                                                             0 “hard”2 failures
    OPEVAL –            TBMW – C2 Space           July 15-31, 1999           Reported mission ready
    Central Command     Forces                                               0 “soft” failures
                                                                             1 “hard” failure
    OPEVAL – Central    Communications –         July 27, 1999               Reported mission ready
    Command             Space Force                                          1 “soft” failure
                        Enhancement                                          0 “hard failures
4
The CCA was designed to evaluate the ability of unified commands to perform missions in an
 environment degraded by Y2K failures.




                                                                                                            11




                                      Page 24                        GAO/AIMD-00-30 Space Command Y2K Operational Testing
                                  Appendix I
                                  Briefing on Results of GAO Review of
                                  SPACECOM Space Control Y2K OPEVAL




                                                        Background
Evaluation Events   “Thin Line”              Schedule                    Results
OPEVAL              Space Lift               July 22, 1999               Reported mission ready
                                                                         0 “soft” failures
                                                                         0 “hard” failures

OPEVAL              Space Lift               July 27, 1999               Reported mission ready
                                                                         0 “soft” failures
                                                                         0 “hard” failures

OPEVAL              Space Lift               August 2-6, 1999            Reported mission ready
                                                                         0 “soft” failures
                                                                         0 “hard” failures

OPEVAL              Space Lift               September 1-3, 1999         Reported mission ready
                                                                         0 “soft” failures
                                                                         0 “hard” failures




                                                                                                       12




                                  Page 25                       GAO/AIMD-00-30 Space Command Y2K Operational Testing
                     Appendix I
                     Briefing on Results of GAO Review of
                     SPACECOM Space Control Y2K OPEVAL




                                           Background
Space Control OPEVAL
•   The Space Control “thin line” includes providing (1) surveillance
    support to monitor, track, identify, and catalog all orbiting space
    objects for collision avoidance and (2) protection support to monitor,
    detect, assess, characterize, track, and issue warnings about threats,
    both natural and man-made, against U.S. and allied space systems.

•   The Space Control OPEVAL was conducted in collaboration with
    other DOD organizations, including Air Force, Army, and Navy Space
    Commands. It was intended to test real-world Space Control
    operations in a Y2K environment.




                                                                                         13




                     Page 26                      GAO/AIMD-00-30 Space Command Y2K Operational Testing
                             Appendix I
                             Briefing on Results of GAO Review of
                             SPACECOM Space Control Y2K OPEVAL




                                                        Background
•   The Space Control “thin line” included 50 systems supporting 35 critical
    tasks. The OPEVAL involved 29 of these systems.

•   The Space Control OPEVAL was completed between March 11-25, 1999
    and included these test execution events:
     –   Rehearsal: March 11, 1999.
     –   Baseline: March 20, 1999.
     –   Surveillance/Intelligence Testing: March 15-19, 1999.
     –   Protection Testing: March 22-25, 1999.


•   The test environment consisted of desktop computers; geographically
    dispersed ground radar sites; and partitioned IBM mainframes.



                                                                                                 14




                             Page 27                      GAO/AIMD-00-30 Space Command Y2K Operational Testing
                        Appendix I
                        Briefing on Results of GAO Review of
                        SPACECOM Space Control Y2K OPEVAL




                                                   Background
•   The required calendar and leap year events (September 8, 1999 to
    September 9, 1999; December 31, 1999 to January 1, 2000; February
    28, 2000 to February 29, 2000; and February 29, 2000 to March 1, 2000)
    were assessed as part of the OPEVAL.

•   The Space Control OPEVAL assessed 46 date dependent functions in
    29 systems using these dates.




                                                                                            15




                        Page 28                      GAO/AIMD-00-30 Space Command Y2K Operational Testing
                               Appendix I
                               Briefing on Results of GAO Review of
                               SPACECOM Space Control Y2K OPEVAL




                                                     Background
Space Control Critical Tasks                          Thin Line Systems

1. Maintain the current space environment             Navy Fence; Millstone; Thule; TOS; Altair;
database                                              MPDS; IDHS; SPADOC; CMP; CFE-R;
                                                      ICIG; NUIS; AMHS


2. Space Surveillance Network tasking                 MPDS; SPADOC; CMP


3. Observation control/tasking analysis               MPDS; SPADOC; CMP

4. Observation Processing                             SPADOC

5. Element Set Updates                                SPADOC

6. Transmit Field Element Sets                        SPADOC

7. Cross-tag/lost satellite/unknown observation       SPADOC
processing
8. Launch processing                                  SPADOC




                                                                                                   16




                               Page 29                      GAO/AIMD-00-30 Space Command Y2K Operational Testing
                               Appendix I
                               Briefing on Results of GAO Review of
                               SPACECOM Space Control Y2K OPEVAL




                                                     Background
Space Control Critical Tasks                          Thin Line Systems

 9. Maneuver Processing                               SPADOC

10. Manual Piece Separation Processing                SPADOC

11. Collision Avoidance                                SPADOC

12. Decay Processing/re-entry assessment              SPADOC

13. Break-up Processing                               SPADOC

14. Monitor and report status of all sensor sites   C MPDS; SPADOC; CMP

15. Manage sensor coverage                            MPDS; SPADOC; CMP

16. Perform event-related up-channel reporting        MPDS; SPADOC; CMP




                                                                                                   17




                               Page 30                      GAO/AIMD-00-30 Space Command Y2K Operational Testing
                                Appendix I
                                Briefing on Results of GAO Review of
                                SPACECOM Space Control Y2K OPEVAL




                                                        Background
Space Control Critical Tasks                            Thin Line Systems

 17. Perform extended collection/surveillance           Navy Fence, Millstone, Thule, TOS, Altair;
against foreign satellites to characterize status and   MPDS; IDHS; SPADOC; CMP; CFE-R;
performance parameters as well as support the           ICIG; NUIS; AMHS; OSAS; SDB; SMAT;
foreign space order of battle                           SMPAS

18. Monitor the space situation and collect and         Navy Fence, Millstone, Thule, TOS, Altair;
correlate data on potential and actual hostile          MPDS; IDHS; SPADOC; CMP; CFE-R;
activities against U.S. and designated allied space     ICIG; NUIS; AMHS; SDB; SATRAN
systems.

19. Assess data and determine intent                    SPADOC; AMHS; NUIS; OSAS; SDB;
                                                        SMAT; SMPAS
20. Inform the National Military Command Center of      MPDS; SPADOC; CMP
impending, current, and completed hostile space
activity
21. Characterize the results of a space attack          SPADOC




                                                                                                     18




                                Page 31                       GAO/AIMD-00-30 Space Command Y2K Operational Testing
                               Appendix I
                               Briefing on Results of GAO Review of
                               SPACECOM Space Control Y2K OPEVAL




                                                     Background
Space Control Critical Tasks                          Thin Line Systems

22. Provide designated authorities with situation     MPDS; SPADOC; CMP
reports

23. Provide technical support as required             MPDS; SPADOC; CMP


24. Provide assistance to routine peacetime space     MPDS; SPADOC; CMP
operations
25. Provide warning and assessment messages           MPDS; SPADOC; CMP

26. Provide countermeasure coordination/status        MPDS; SPADOC; CMP

27. Inform space system owner/operators and other     MPDS; SPADOC; CMP
designated authorities of selected countermeasures




                                                                                                   19




                               Page 32                      GAO/AIMD-00-30 Space Command Y2K Operational Testing
                                 Appendix I
                                 Briefing on Results of GAO Review of
                                 SPACECOM Space Control Y2K OPEVAL




                                                        Background
S pace Control C ritical Tasks                              Thin Line S ystem s

28. E valuate counterm easure effectiveness                 S PA D O C

29. Assist in planning for further counterm easures         S PA D O C

30. Inform N ational M ilitary C om m and C enter and       M P DS ; S PA D O C; C M P
other appropriate elem ents of the results of
counterm easure im plem entation

31. M aintain current docum entation of w orld-wide         S PA D O C; C FE -R; IC IG ; NU IS; S D B;
counter space capabilities                                  S M PA S ; O S AS; SM A T

32. Receive taskings (sensors)                              N avy Fence; M illstone; Thule; TO S; A ltair;
                                                            M P DS ; CM P

33. Schedule tracks (sensors)                               N avy Fence; M illstone; Thule; TO S; A ltair


34. Conduct tracks (sensors)                                N avy Fence; Millstone; Thule; TO S; A ltair

35. Transm it track data (sensors)                          N avy Fence; M illstone; Thule; TO S; A ltair;
                                                            M P DS ; IDH S



                                                                                                         20




                                 Page 33                      GAO/AIMD-00-30 Space Command Y2K Operational Testing
   Appendix I
   Briefing on Results of GAO Review of
   SPACECOM Space Control Y2K OPEVAL




Results of GAO Review




                                                                       21




   Page 34                      GAO/AIMD-00-30 Space Command Y2K Operational Testing
                        Appendix I
                        Briefing on Results of GAO Review of
                        SPACECOM Space Control Y2K OPEVAL




                                                        Planning
Defense Test Criteria                                                 Result

Specify test assumptions and limitations                             Satisfied
Establish a Y2K task force and assign responsibilities               Satisfied
Identify critical missions/tasks/systems                              Satisfied
Verify systems essential to mission are Y2K compliant/certified       Partially Satisfied
Develop OPEVAL plan to guide event planning and execution             Satisfied
Identify and schedule CINC/Allied/Component/Agency support            Satisfied
Determine relevant and necessary resources (e.g., funding,            Satisfied
personnel, equipment, etc.)
Ensure approved Y2K contingency plans are prepared                    Partially Satisfied
Develop risk management plan                                          Satisfied
Identify simulation needs and establish supporting environment Satisfied
Develop data collection and analysis plan or approaches               Satisfied


                                                                                            22




                        Page 35                      GAO/AIMD-00-30 Space Command Y2K Operational Testing
                        Appendix I
                        Briefing on Results of GAO Review of
                        SPACECOM Space Control Y2K OPEVAL




                                     Findings: Planning
Criteria: In planning for the OPEVAL, CINCs are to define assumptions
concerning the readiness of systems and the ability to evaluate systems in
light of real-world limitations.

Finding: Space Command identified real-world considerations and system
readiness limitations during Y2K planning meetings. These limitations
were disclosed in the OPEVAL Plan and to the JCS. Specifically, Space
Command reported that 29 of the 50 “thin line” systems would be included
and 21 would be excluded from the OPEVAL due to resource constraints,
or because they would be tested in other OPEVALs. Six of the 21
excluded systems were communications systems that were to be tested in
a future OPEVAL.




                                                                                            23




                        Page 36                      GAO/AIMD-00-30 Space Command Y2K Operational Testing
                 Appendix I
                 Briefing on Results of GAO Review of
                 SPACECOM Space Control Y2K OPEVAL




                              Findings: Planning
Criteria: A CINC Y2K Task Force composed of knowledgeable
Y2K, test, and systems experts should be formed to establish
the base for all Y2K planning, coordination, execution, and
reporting.

Finding: Consistent with the defined scope of the OPEVAL,
Space Command established a Y2K Task Force and it defined
roles and responsibilities with milestones for each member.
Members of the task force included satellite/system specialists,
test and evaluation experts, system analysts, and public affair
specialists from the Command’s Operations, Intelligence,
Planning, and Public Affairs units. They also included Air
Force’s Operational Test and Evaluation Center (AFOTEC) and
17th Test Squadron, military service, and NASA
representatives.
                                                                                     24




                 Page 37                      GAO/AIMD-00-30 Space Command Y2K Operational Testing
                  Appendix I
                  Briefing on Results of GAO Review of
                  SPACECOM Space Control Y2K OPEVAL




                                Findings: Planning
Criteria: CINCs need to analyze critical missions to determine
the most critical missions and identify the critical tasks
supporting each critical mission. In addition, the minimum
number of integrated automated information platforms/systems
required to perform each critical task or critical mission must be
identified (the “thin line”).

Finding: Consistent with the defined scope of the OPEVAL,
Space Command identified 35 critical tasks that needed to be
evaluated to determine mission readiness in a Y2K
environment. In addition, Space Command identified a total of
29 “thin line” systems to support these tasks.




                                                                                      25




                  Page 38                      GAO/AIMD-00-30 Space Command Y2K Operational Testing
                  Appendix I
                  Briefing on Results of GAO Review of
                  SPACECOM Space Control Y2K OPEVAL




                               Findings: Planning
Criteria: Ensure that mission-critical “thin line” systems are
certified Y2K compliant.

Finding: Consistent with the defined scope of the OPEVAL,
Space Command verified that 28 of 29 mission-critical, “thin
line” systems to be included in the OPEVAL were certified Y2K
compliant. The 29th system was not certified as compliant, but
was nevertheless included in the OPEVAL rather than invoking
the system’s contingency plan because, according to Space
Command officials, they verified that the application on the
system relevant to the OPEVAL was compliant. This is contrary
to JCS guidance and GAO’s end-to-end test guidance, which
defines system (not application) compliance as a precondition
to end-to-end testing. In short, Y2K compliance of an
application in isolation is of very limited value unless the system
platform that it runs on, and the other applications running on
the system that it interoperates with, are also compliant.
                                                                                      26




                  Page 39                      GAO/AIMD-00-30 Space Command Y2K Operational Testing
                 Appendix I
                 Briefing on Results of GAO Review of
                 SPACECOM Space Control Y2K OPEVAL




                                  Finding: Planning

Finding: At the time of our October 1, 1999, briefing, Space
Command had not addressed this concern. We therefore
recommended that this deviation be disclosed in the final
OPEVAL report. Space Command officials agreed with our
recommendation that the final report disclose this information
and now plans to revise the final report. Additionally, Space
Command stated that the system is scheduled to be
compliant in November 1999 and to be included in its
November 1999 operational evaluation of the intelligence
mission area.




                                                                                     27




                 Page 40                      GAO/AIMD-00-30 Space Command Y2K Operational Testing
                    Appendix I
                    Briefing on Results of GAO Review of
                    SPACECOM Space Control Y2K OPEVAL




                                  Findings: Planning
Criteria: The Y2K task force should document how the OPEVAL
will be conducted, data will be gathered and analyzed, and how
reports will be formatted.

Finding: Space Command developed an exercise directive and
test plan for the OPEVAL to:
 – ensure that mechanisms for evaluating critical dates and contingency
   plans for mission-critical systems are executed.
 – document participant roles and responsibilities.
 – link critical missions, critical tasks, architectures, test cases, and data
   elements.
 – report Y2K OPEVAL results.




                                                                                        28




                    Page 41                      GAO/AIMD-00-30 Space Command Y2K Operational Testing
                 Appendix I
                 Briefing on Results of GAO Review of
                 SPACECOM Space Control Y2K OPEVAL




                              Findings: Planning
Criteria: When preparing for a Y2K OPEVAL, determine the
extent of participation of other CINCs, allies, components, and
agencies and coordinate their participation in the event.

Finding: Consistent with the defined scope of the OPEVAL,
Space Command identified, coordinated, and scheduled the
OPEVAL with Y2K Task Force members from the Command’s
Operations, Intelligence, Planning, and Public Affairs units.
They also coordinated activities with Air Force’s Operational
Test and Evaluation Center (AFOTEC), 17th Test Squadron,
and Air Force, Army, and Navy Space Commands, and NASA.




                                                                                     29




                 Page 42                      GAO/AIMD-00-30 Space Command Y2K Operational Testing
                 Appendix I
                 Briefing on Results of GAO Review of
                 SPACECOM Space Control Y2K OPEVAL




                              Findings: Planning

Criteria: The necessary resources (funding, personnel, training,
equipment, time frames, and external organization support)
should be identified and included in the plan.

Finding: In November 1998, Space Command issued a directive
to ensure testing resources would be made available for Y2K
OPEVALS. About $8 million was earmarked for OPEVAL
activities, including the Space Control OPEVAL. Space
Command also coordinated the evaluation scenario and scripts
with all OPEVAL participants, acquired the systems hardware
and software to simulate space control events, and scheduled
37 test and operator personnel to help execute the OPEVAL.




                                                                                     30




                 Page 43                      GAO/AIMD-00-30 Space Command Y2K Operational Testing
                 Appendix I
                 Briefing on Results of GAO Review of
                 SPACECOM Space Control Y2K OPEVAL




                             Findings: Planning
Criteria: To ensure that Y2K exercise objectives are met, it is
essential to have contingency plans in place prior to executing
the OPEVAL.

Finding 1: Space Command’s approach to determining whether
contingency plans were in place prior to executing the OPEVAL
was to rely exclusively on system owners to ensure that this
criterion was met. Space Command did not take steps to verify
this criterion. Space Command has since initiated a review of
about 50 contingency plans.

Finding 2: During the OPEVAL, operators successfully
performed mission tasks in response to the 3 “hard” system
failures by invoking workarounds.

                                                                                     31




                 Page 44                      GAO/AIMD-00-30 Space Command Y2K Operational Testing
                Appendix I
                Briefing on Results of GAO Review of
                SPACECOM Space Control Y2K OPEVAL




                              Findings: Planning
Criteria: CINC-unique risk management plans should be developed to
identify and mitigate system-related risks before they adversely
impact mission execution.

Finding: Space Command identified OPEVAL risks and strategies for
managing these risks in its Space Control OPEVAL Plan. For
example, the Plan recognizes the risks associated with confusing
OPEVAL sensor observations with real-world observations. To
mitigate these risks, Space Command’s Plan provides strategies for
isolating systems’ execution of OPEVAL tasks/functions from real-
world system operations by electronic partitioning.




                                                                                    32




                Page 45                      GAO/AIMD-00-30 Space Command Y2K Operational Testing
                Appendix I
                Briefing on Results of GAO Review of
                SPACECOM Space Control Y2K OPEVAL




                             Findings: Planning

Criteria: CINCs should (1) determine if simulations or manual
data input will be needed during the execution of the OPEVAL,
and, if needed, (2) ensure that an environment which can
support the simulation is planned for and acquired.

Finding: Within the defined scope of the OPEVAL, Space
Command identified the simulations needed and manual data
inputs required for testing and ensured that data injection
methodologies were included in the OPEVAL Plan and master
scenario events list (MSEL). For example, Space Command
used simulated scenarios to perform satellite orbital changes
for tracking and cataloging purposes.



                                                                                    33




                Page 46                      GAO/AIMD-00-30 Space Command Y2K Operational Testing
                 Appendix I
                 Briefing on Results of GAO Review of
                 SPACECOM Space Control Y2K OPEVAL




                              Findings: Planning
Criteria: A plan should be prepared to help coordinate and
synchronize all OPEVAL data collection and assessment
activities.

Finding: Consistent with the specified scope of the OPEVAL,
Space Command developed a data collection and analysis plan
that included (1) specific actions that should be accomplished by
the OPEVAL participants prior to the start of and at the
completion of each OPEVAL, (2) ground rules for collecting and
documenting mission-critical system outputs, and (3) direction
on reviewing the critical tasks executed during the OPEVAL and
determining the performance of the mission-critical “thin line”
systems.



                                                                                     34




                 Page 47                      GAO/AIMD-00-30 Space Command Y2K Operational Testing
                        Appendix I
                        Briefing on Results of GAO Review of
                        SPACECOM Space Control Y2K OPEVAL




                                                   Execution

Defense Test Criteria                                                Result

Conduct OPEVAL rehearsal                                             Satisfied

Follow configuration management policies                             Partially Satisfied

Perform baseline test for OPEVAL                                     Satisfied

Execute required Y2K date rollover tests                             Satisfied
Collect and archive all Y2K-relevant data and ensure that
systems are reset to current day operations                          Satisfied




                                                                                            35




                        Page 48                      GAO/AIMD-00-30 Space Command Y2K Operational Testing
                  Appendix I
                  Briefing on Results of GAO Review of
                  SPACECOM Space Control Y2K OPEVAL




                             Findings: Execution
Criteria: Prior to executing the Y2K OPEVAL, a rehearsal should be
conducted to ensure that all critical systems and interfaces identified
in the system architecture are operating correctly and that OPEVAL
staff know their roles and responsibilities.

Finding: Space Command performed a rehearsal/test readiness
review on March 11, 1999. The rehearsal was used to (1) validate
that the test readiness review requirements, (2) verify data collection
and analysis methodologies, (3) confirm the baseline configuration
for testing, and (4) ensure OPEVAL staff practiced their roles and
responsibilities.




                                                                                      36




                  Page 49                      GAO/AIMD-00-30 Space Command Y2K Operational Testing
                 Appendix I
                 Briefing on Results of GAO Review of
                 SPACECOM Space Control Y2K OPEVAL




                              Findings: Execution
Criteria: The configurations of systems and architecture
established for OPEVAL testing should not be changed unless
authorized by the test director.

Finding: Changes were made to one system in the test
environment after the baseline for the OPEVAL was established
and without authorization from the test director. These changes
contributed to one “hard” system failure identified during the
OPEVAL. (Information on the nature of the system failure is
classified.) Space Command has since reinforced the need to
strictly follow configuration management policies during
OPEVALS and tasked the 17th Test Squadron and intelligence
unit to develop ways to better ensure that configuration
management over test baselines is enforced.


                                                                                     37




                 Page 50                      GAO/AIMD-00-30 Space Command Y2K Operational Testing
                     Appendix I
                     Briefing on Results of GAO Review of
                     SPACECOM Space Control Y2K OPEVAL




                                  Findings: Execution
Criteria: A baseline Y2K test should be executed to establish expected
results data that will be used to compare to output data captured during the
Y2K date rollover tests and to help establish whether or not a failure is
Y2K-related.
Finding: Space Command conducted a baseline Y2K test on
March 20, 1999. However, this test only covered the critical tasks
associated with space surveillance and protection and did not include
intelligence tasks because officials stated that baseline testing duplicated
rehearsal activities. This position is contrary to JCS guidance. According to
JCS guidance, the purpose of the rehearsal is to provide operators with an
opportunity to practice their responsibilities. In contrast, the baseline test is
to execute the master scenario events list under operational conditions to
establish expected outputs against which OPEVAL results can be
compared. Clearly, these two execution requirements, because they serve
different purposes, differ in terms of content, depth, and scope, and thus
are not duplicative.


                                                                                         38




                     Page 51                      GAO/AIMD-00-30 Space Command Y2K Operational Testing
                Appendix I
                Briefing on Results of GAO Review of
                SPACECOM Space Control Y2K OPEVAL




                              Findings: Execution
Finding: Space Command officials acknowledge the
differences between test rehearsals and baseline tests, but
explained that the rehearsal for intelligence systems was
expanded to satisfy baseline testing requirements,
including using the same quality and quantity of data
planned for the baseline test. To verify this, we reviewed
information subsequently provided by the test directorate
and found that it showed a level of testing rigor for these
systems that went beyond that normally required of a test
rehearsal. We also verified that baseline test results were
documented during the rehearsal. SPACECOM officials
have disclosed this deviation and its impact in its amended
OPEVAL report.

                                                                                    39




                Page 52                      GAO/AIMD-00-30 Space Command Y2K Operational Testing
                 Appendix I
                 Briefing on Results of GAO Review of
                 SPACECOM Space Control Y2K OPEVAL




                             Findings: Execution
Criteria: Mission-critical “thin line” systems should be executed
using normal operating procedures, and a seamless continuity of
operations during critical Y2K date rollovers should be observed.

Finding: For the Space Control OPEVAL, 29 systems were
tested and 3 experienced “hard” failures. According to Space
Command officials, it was not necessary to invoke contingency
plans for these failures because operators were able to perform
workarounds to complete mission tasks. For 2 of the 3 “hard”
failures, these workarounds were included in OPEVAL
documentation; however, for the third “hard” failure, OPEVAL
documentation was not prepared. According to Space
Command officials, in all 3 cases, operations were not disrupted
and tasks were completed seamlessly and continually.


                                                                                     40




                 Page 53                      GAO/AIMD-00-30 Space Command Y2K Operational Testing
                 Appendix I
                 Briefing on Results of GAO Review of
                 SPACECOM Space Control Y2K OPEVAL




                                 Findings: Execution
Criteria: Ensure that all data needed to conduct the evaluation
for the Y2K case has been captured prior to resetting the system
to current day operations requirements.

Finding: Prior to resetting the systems to present day operational
conditions, Space Command determined that (1) the 29 “thin-
line” systems were assessed, (2) the master scenario events
were performed and deviations were identified, and (3) all data
needed to make an assessment of Space Command’s ability to
perform the defined “thin line” were collected and archived.




                                                                                     41




                 Page 54                      GAO/AIMD-00-30 Space Command Y2K Operational Testing
                        Appendix I
                        Briefing on Results of GAO Review of
                        SPACECOM Space Control Y2K OPEVAL




                                                     Analysis
Defense Test Criteria                                                Result

Categorize, document, and report failures                            Satisfied

Determine mission impact of Y2K failures                             Satisfied

Ensure Y2K OPEVAL exit criteria are met                              Partially Satisfied




                                                                                            42




                        Page 55                      GAO/AIMD-00-30 Space Command Y2K Operational Testing
                  Appendix I
                  Briefing on Results of GAO Review of
                  SPACECOM Space Control Y2K OPEVAL




                                Findings: Analysis
Criteria: All failures are to be identified and properly categorized
as either “hard” or “soft” failures and should be documented and
reported in accordance with the data collection and analysis
plan.

Finding: Space Command identified 5 Y2K failures during the
Space Control OPEVAL and categorized 3 as “hard” and 2 as
“soft” failures. All system failures were documented in
accordance with DOD Y2K requirements and reported to the
Joint Staff Y2K office. Examples of the “hard” failures include a
system that did not display messages during and after the leap
year rollover and a system that did not automatically list file
names for operators in a viewer window.



                                                                                      43




                  Page 56                      GAO/AIMD-00-30 Space Command Y2K Operational Testing
                 Appendix I
                 Briefing on Results of GAO Review of
                 SPACECOM Space Control Y2K OPEVAL




                             Findings: Analysis
Criteria: Determine the impact of a failure on the accomplishment
of a critical mission.

Finding: Space Command determined that all 5 Y2K failures had
no significant impact on its ability to perform the Space Control
mission.




                                                                                     44




                 Page 57                      GAO/AIMD-00-30 Space Command Y2K Operational Testing
                 Appendix I
                 Briefing on Results of GAO Review of
                 SPACECOM Space Control Y2K OPEVAL




                                 Findings: Analysis
Criteria: JCS defined 9 exit criteria that OPEVAL results should
be measured to ensure that critical tasks and missions can be
performed in a Y2K environment.

Finding 1: Space Command measured its OPEVAL performance
against the 9 Joint Staff exit criteria and concluded that the
Space Control mission can be successfully performed in a Y2K
environment.




                                                                                     45




                 Page 58                      GAO/AIMD-00-30 Space Command Y2K Operational Testing
                   Appendix I
                   Briefing on Results of GAO Review of
                   SPACECOM Space Control Y2K OPEVAL




                                   Findings: Analysis
Finding 2: Space Command did not document that all the measures
of performance established as exit criteria for intelligence critical
tasks/systems were achieved. Measures of performance are used
to determine whether specified system functions are performed
within established time frames. Space Command’s measures of
performance for the critical tasks in its Space Control OPEVAL
included predetermined numbers of transactions to be executed
and time constraints within which transactions are to be executed.

However, Space Command only documented a portion of the
predetermined number of transactions specified in the OPEVAL
Plan for intelligence critical tasks. According to intelligence officials,
all predefined transactions were executed successfully but only
one-fifth were documented because it was too time-consuming to
print screens during testing. In the absence of the requisite test
results documentation, we could not validate this claim.
                                                                                       46




                   Page 59                      GAO/AIMD-00-30 Space Command Y2K Operational Testing
                 Appendix I
                 Briefing on Results of GAO Review of
                 SPACECOM Space Control Y2K OPEVAL




                                 Findings: Analysis

Finding 3: At the time of our briefing, Space Command had not
addressed this concern. We therefore recommended that the
final report be revised to reflect this deviation and to describe the
actions being taken to mitigate the resulting risks. Space
Command officials have since agreed that some alternative
measure should have been taken to document all test results
and thus ensure the OPEVAL’s integrity was not compromised.
The officials have also agreed to our recommendation and plan
to revise the final OPEVAL report to reflect this deviation and to
ensure that the November 1999 operational evaluation of its
intelligence mission area provides for fully documenting test
results.



                                                                                     47




                 Page 60                      GAO/AIMD-00-30 Space Command Y2K Operational Testing
                         Appendix I
                         Briefing on Results of GAO Review of
                         SPACECOM Space Control Y2K OPEVAL




                                                    Reporting

Defense Test Criteria                                                 Result

Prepare Y2K reports describing mission impact and readiness           Partially Satisfied

Provide reports to Joint Staff J7 within required time frames         Satisfied




                                                                                             48




                         Page 61                      GAO/AIMD-00-30 Space Command Y2K Operational Testing
                Appendix I
                Briefing on Results of GAO Review of
                SPACECOM Space Control Y2K OPEVAL




                           Findings: Reporting
Criteria: CINCs are to prepare Y2K reports describing mission
impact and readiness.

Finding 1: Space Command provided Y2K reports to the Joint
Staff which concluded that all critical tasks supporting Space
Control can be performed with no significant impact on
readiness caused by potential Y2K failures.

Finding 2: Space Command officials stated that the reports were
completed as required. However, the reports did not fully
describe the limitations in the scope of the OPEVAL and testing
deviations (i.e., the omission of 6 key communications systems,
the noncompliant system involved in the OPEVAL, configuration
changes made to a system after the baseline was established
for the OPEVAL, and the failure to fully document that

                                                                                    49




                Page 62                      GAO/AIMD-00-30 Space Command Y2K Operational Testing
                 Appendix I
                 Briefing on Results of GAO Review of
                 SPACECOM Space Control Y2K OPEVAL




                            Findings: Reporting
established performance criteria for intelligence tasks/systems
were satisfied) or the extent to which these limitations and
deviations affected the command’s ability to draw unqualified
conclusions about Space Control mission readiness.

Finding 3: Since completing the Space Control OPEVAL, Space
Command provided documentation that showed the 6
communications systems were included in other OPEVAL or
end-to-end tests. Also, Space Command has recently agreed to
revise its 30-day report to disclose testing deviations involving
the use of a noncompliant system in the OPEVAL, the failure to
follow configuration management procedures, and the failure to
fully document intelligence tasks/systems test results.


                                                                                     50




                 Page 63                      GAO/AIMD-00-30 Space Command Y2K Operational Testing
                 Appendix I
                 Briefing on Results of GAO Review of
                 SPACECOM Space Control Y2K OPEVAL




                            Findings: Reporting
Criteria: A preliminary report is required within 7 calendar days
after the completion of the OPEVAL and a final report is required
within 30 calendar days. Both reports are to be provided to Joint
Staff.

Finding: Space Command completed the 7- and 30-day reports
for the Space Control OPEVAL and provided them to Joint Staff.




                                                                                     51




                 Page 64                      GAO/AIMD-00-30 Space Command Y2K Operational Testing
                       Appendix I
                       Briefing on Results of GAO Review of
                       SPACECOM Space Control Y2K OPEVAL




                                                   Conclusions
•   Space Command satisfied many of the Defense OPEVAL
    requirements for its defined Space Control “thin line.”

•   However, key steps that are vital to (1) ensuring that only compliant
    systems or system contingency plans are used in the OPEVAL,
    (2) fully disclosing deviations from planned performance measures
    and the impact of doing so, and (3) accurately reporting mission
    readiness in light of OPEVAL scope limitations were not fully
    satisfied. As a result, the Y2K readiness of Space Control critical
    tasks involving intelligence and communications systems was not
    known with sufficient surety to support Space Command’s March
    1999 unqualified conclusion of mission readiness in a Y2K
    environment.


                                                                                           52




                       Page 65                      GAO/AIMD-00-30 Space Command Y2K Operational Testing
                       Appendix I
                       Briefing on Results of GAO Review of
                       SPACECOM Space Control Y2K OPEVAL




                                                   Conclusions
•   Since then, Space Command has taken steps to fill voids in its
    understanding of Space Control mission readiness by ensuring that
    omitted communications systems were included in other Y2K end-
    to-end testing or OPEVAL events. It has also taken additional
    action to improve testing procedures and documentation
    requirements and has agreed to address our recommendation for
    revising its final OPEVAL report to reflect deviations with regard to
    the performance and verification of intelligence systems’ Y2K
    compliance. Moreover, Space Command has acted to ensure that
    its planned November 1999 operational evaluation provides for fully
    documenting test results. Therefore, we are not making any further
    recommendations at this time.




                                                                                           53




                       Page 66                      GAO/AIMD-00-30 Space Command Y2K Operational Testing
Appendix II

Objectives, Scope, and Methodology                                                          Appendx
                                                                                                  Ii




              At the request of the Chairman, House Appropriations Committee,
              Subcommittee on Defense, we selected the Space Command Space Control
              evaluation for review to determine (1) if the evaluation was planned,
              executed, and documented in accordance with DOD guidelines, and
              (2) what the evaluation results indicated concerning readiness and risks.
              This operational evaluation was selected in collaboration with the Defense
              Inspector General to ensure appropriate coverage of all combatant
              command operational evaluations and no duplication of effort.

              To satisfy our first objective, we reviewed the evaluation plan, testing
              documentation and records, and test results and associated reports. We
              also interviewed Space Command officials responsible for Year 2000
              operational evaluation planning, execution, and reporting tasks. Further,
              we examined the century date rollover testing documents for the
              operational evaluation and compared Space Command’s operational
              evaluation planning, execution, analysis, and reporting actions against JCS
              operational evaluation guidance and our Year 2000 testing guide.

              To satisfy the second objective, we reviewed Space Command’s operational
              evaluation results, including its 7- and 30-day reports and system problem
              tracking reports. We also interviewed Space Command officials and
              analysts responsible for developing operational evaluation assessment
              methodologies, interpreting evaluation metrics, and ensuring that
              evaluation exit criteria were met.

              On October 1, 1999, we briefed Space Command leadership on the results
              of our review. Space Command provided oral comments on our briefing
              slides, and we have incorporated them as appropriate. We performed our
              work from March through October 1999 in accordance with generally
              accepted government auditing standards.




              Page 67                   GAO/AIMD-00-30 Space Command Y2K Operational Testing
Appendix III

GAO Contact and Staff Acknowledgements                                                              AppendxIi




GAO Contact            Randolph C. Hite, (202) 512-6240




Acknowledgements       In addition to those named above, Ronald B. Bageant, Cristina T. Chaplain,
                       Madhav Panwar, and Yvonne Vigil made key contributions to this report.




(511661)       Leter   Page 68                   GAO/AIMD-00-30 Space Command Y2K Operational Testing
Ordering Information

The first copy of each GAO report and testimony is free.
Additional copies are $2 each. Orders should be sent to the
following address, accompanied by a check or money order made
out to the Superintendent of Documents, when necessary, VISA and
MasterCard credit cards are accepted, also.

Orders for 100 or more copies to be mailed to a single address are
discounted 25 percent.

Orders by mail:

U.S. General Accounting Office
P.O. Box 37050
Washington, DC 20013

or visit:

Room 1100
700 4th St. NW (corner of 4th and G Sts. NW)
U.S. General Accounting Office
Washington, DC

Orders may also be placed by calling (202) 512-6000
or by using fax number (202) 512-6061, or TDD (202) 512-2537.

Each day, GAO issues a list of newly available reports and
testimony. To receive facsimile copies of the daily list or any list
from the past 30 days, please call (202) 512-6000 using a touchtone
phone. A recorded menu will provide information on how to obtain
these lists.

For information on how to access GAO reports on the INTERNET,
send an e-mail message with “info” in the body to:

info@www.gao.gov

or visit GAO’s World Wide Web Home Page at:

http://www.gao.gov
United States                       Bulk Rate
General Accounting Office      Postage & Fees Paid
Washington, D.C. 20548-0001           GAO
                                 Permit No. GI00
Official Business
Penalty for Private Use $300

Address Correction Requested