oversight

Defense Computers: DOD Y2K Functional End-to-End Testing Progress and Test Event Management

Published by the Government Accountability Office on 1999-10-18.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                 United States General Accounting Office

GAO              Report to the Chairman of the
                 Subcommittee on Defense, Committee on
                 Appropriations, House of
                 Representatives

October 1999
                 DEFENSE
                 COMPUTERS

                 DOD Y2K Functional
                 End-to-End Testing
                 Progress and Test
                 Event Management




GAO/AIMD-00-12
Contents



Letter                                                                                 3


Appendixes   Appendix I:    Briefing on DOD Y2K Functional End-to-End
               Testing: Progress and Test Event Management                            14
             Appendix II:   Objectives, Scope, and Methodology                        47
             Appendix III: Comments From the Department of Defense                    49
             Appendix IV: GAO Contact and Staff Acknowledgements                      53


Tables       Table 1: Summary of Recommended End-to-End Test Management
               Processes                                                               6
             Table 2: Summary of the Four Functional Area Test Approaches              7
             Table 3: Summary of Test Events Satisfying GAO Key Processes              9




             Abbreviations

             DISA      Defense Information Systems Agency
             DOD       Department of Defense
             JUSE      Joint User Switch Exercise
             OMB       Office of Management and Budget
             OSD       Office of the Secretary of Defense
             PSA       Principal Staff Assistant
             Y2K       Year 2000




             Page 1                             GAO/AIMD-00-12 DOD Y2K End-to-End Testing
Page 2   GAO/AIMD-00-12 DOD Y2K End-to-End Testing
United States General Accounting Office                                                     Accounting and Information
Washington, D.C. 20548                                                                           Management Division



                                    B-283564                                                                             Leter




                                    October 18, 1999

                                    The Honorable Jerry Lewis
                                    Chairman, Subcommittee on Defense
                                    Committee on Appropriations
                                    House of Representatives

                                    Dear Mr. Chairman:

                                    Complete and thorough Year 2000 (Y2K) testing is essential to provide
                                    reasonable assurance that new or modified systems process dates correctly
                                    and will not jeopardize an organization’s ability to perform core business
                                    operations after the millennium. This is especially true for the Department
                                    of Defense (DOD) which relies on a complex and broad array of
                                    interconnected computer systems—including weapons, command and
                                    control, satellite, inventory management, transportation management,
                                    health, financial, personnel and payment systems—to carry out its core
                                    business functions and military operations.

                                    At your request, we initiated a review of the effectiveness of DOD’s efforts
                                    to perform Year 2000-related end-to-end tests for its major business
                                    functions, including Health Affairs, Communications, Personnel, and
                                    Logistics. Together, these functional areas are performing thousands of
                                    end-to-end tests to ensure that key business processes and systems can
                                    continue operating into the year 2000. Specifically, for each functional area,
                                    we analyzed reported information on the status and progress of all test
                                    events. We also selected and reviewed a critical test event in each
                                    functional area to determine whether it was planned and managed in
                                    accordance with our Year 2000 testing guide.1 On September 14, 1999, we
                                    briefed you on the results of our review. This report provides a summary of
                                    our briefing and a recommendation to Defense for strengthening oversight
                                    of end-to-end testing for the Communications functional area. Subsequent
                                    to our briefing, logistics officials submitted additional information on the
                                    inclusion of installation telecommunications providers in related test
                                    events. We have clarified the briefing slides to reflect this. These
                                    clarifications, however, do not affect our overall conclusions and
                                    recommendation. The briefing slides are presented in appendix I, and our

                                    1
                                        Year 2000 Computing Crisis: A Testing Guide (GAO/AIMD-10.1.21, November 1998).




                                    Page 3                                       GAO/AIMD-00-12 DOD Y2K End-to-End Testing
                   B-283564




                   objectives, scope, and methodology are in appendix II. The Office of the
                   Assistant Secretary of Defense provided written comments on a draft of
                   this report. These comments are discussed at the end of this report and
                   reprinted in appendix III. We performed our audit work from March
                   through September 1999 in accordance with generally accepted
                   government auditing standards.



Results in Brief   Because Year 2000 conversions often involve numerous large
                   interconnecting systems with many external interfaces and extensive
                   supporting technology infrastructures, Year 2000 testing should be
                   approached in a structured and disciplined fashion. Our Year 2000 guidance
                   recommends that in planning and managing end-to-end tests, agencies
                   define test boundaries, secure the commitment of data exchange partners,
                   prepare test procedures and data, define exit criteria,2 and document test
                   results, among other steps. Each of the individual test events we attended
                   and reviewed within the four functional areas generally satisfied the key
                   processes that our Year 2000 test guide defines as necessary to effectively
                   plan, conduct, and report on end-to-end testing.3 Moreover, while the
                   events’ respective approaches to implementing the key processes varied,
                   these differences were appropriately based on consideration of the event’s
                   scope and complexity.

                   In addition, overall end-to-end test efforts within three of the four
                   functional areas were reported to be largely on schedule and expected to
                   be completed by October 1999. However, at the time we briefed the
                   Communications functional area on the results of our review, it could not
                   provide complete progress information. While information was
                   subsequently provided by Communications, it showed that the functional
                   area had not yet developed plans to test 31 mission-critical systems. We are
                   making a recommendation to Defense to ensure that these systems are
                   tested or that there is adequate justification for their exclusion from end-to-
                   end test events. While Defense only partially concurred with this
                   recommendation, it provided information showing the status of the
                   systems in question. We did not verify this information.



                   2
                       Exit criteria are test conditions or requirements for successfully completing testing.
                   3
                    Our observations are limited to the specific events we witnessed, and we cannot draw
                   conclusions regarding end-to-end testing from an overall functional area perspective.




                   Page 4                                           GAO/AIMD-00-12 DOD Y2K End-to-End Testing
             B-283564




Background   In August 1998, the Deputy Secretary of Defense recognized the need to
             ensure that various key lines of business or functional areas within the
             department could continue to operate effectively at and after the turn of
             the century. Therefore, the Deputy Secretary directed Office of the
             Secretary of Defense focal points, known as Principal Staff Assistants
             (PSAs), to verify that all functions would be unaffected by Year 2000 issues.
             In doing so, the PSAs were to (1) document mission-critical functions and
             systems supporting those functions, (2) coordinate, facilitate, and monitor
             Year 2000 end-to-end test and evaluation activities of services, agencies,
             and commands, and (3) in some cases, conduct Y2K end-to-end functional
             testing.

             The purpose of end-to-end testing is to verify that a defined set of
             interrelated systems, which collectively support an organizational core
             business area or function, interoperate as intended in an operational
             environment (either actual4 or simulated). These interrelated systems
             include not only those owned and managed by the organization, but also
             the external systems with which they interface or that otherwise support
             the core business area or function.

             The boundaries for end-to-end tests can vary depending on a given business
             function’s system dependencies and criticality to the organizational
             mission. Therefore, in managing end-to-end test activities, it is important to
             analyze the interrelationships among core business functions and their
             supporting systems, and the mission impact and risk of date-induced
             systems failures and to use these analyses to define test boundaries. It is
             also important to work early and continually with functional partners to
             ensure that related end-to-end test activities are effectively coordinated
             and integrated. As highlighted in table 1, our Year 2000 test guide, which
             has been adopted by the Office of Management and Budget (OMB),
             recommends that federal agencies take the following actions in planning
             and managing end-to-end tests.




             4
              Risks of testing in the production environment must be thoroughly analyzed and
             precautions taken to preclude damage to systems and data.




             Page 5                                      GAO/AIMD-00-12 DOD Y2K End-to-End Testing
                                                B-283564




Table 1: Summary of Recommended End-to-End Test Management Processes

Define the system boundaries of the end-to-end      Agencies should define boundaries for the end-to-end test based on an assessment
test(s)                                             of their mission-critical business functions, inter- and intraorganization system
                                                    dependencies, as well as the probabilities and impacts of any of these systems
                                                    suffering a date-related failure.
Secure the commitment of data exchange              Because end-to-end testing addresses business areas or functions that involve
partners                                            multiple internal and external organizations, participation by all key data exchange
                                                    partners should be solicited and obtained.
Establish an interorganizational test team          A team composed of representatives from each of the organizations participating in
                                                    the test should be formed to manage the planning, execution, and reporting of the
                                                    test.
Confirm Year 2000 compliance of                     In order to execute end-to-end testing and ensure that all systems in the chain of
telecommunications infrastructure                   support to core business areas function as intended, agencies should ensure that the
                                                    telecommunications infrastructure that interconnects the systems is compliant and
                                                    ready for testing.
Schedule and plan end-to-end test(s)                A plan should be developed specifying key tasks and requirements for test planning,
                                                    execution, and validation as well 25 milestones and resources associated with
                                                    performing these tasks.
Prepare end-to-end procedures and data              Interorganizational test procedures and data, including steps, cases, and input
                                                    conditions that verify the correct handling of critical dates, should be prepared and
                                                    approved by team representatives.
Define end-to-end test exit criteria                The conditions or requirements for successfully completing end-to-end testing need to
                                                    be established.
Execute end-to-end test(s)                          Tests should be executed in accordance with established plans and procedures.
Document test results                               Test results should be documented so that the data can be used to validate that test
                                                    exit criteria had been met and to assess and correct problems discovered during the
                                                    testing.
Correct Year 2000 defects                           On the basis of interoganization specified criteria, such as defect severity and test
                                                    exist criteria, defects identified during the test should be prioritized and corrected.
Ensure that end-to-end test exit criteria are met   Test results should be compared to test exit criteria to ensure that specified conditions
                                                    are met.




                                                Page 6                                         GAO/AIMD-00-12 DOD Y2K End-to-End Testing
                                            B-283564




                                            The table below explains how the four functional areas included in our
                                            review approached their end-to-end tests.



Table 2: Summary of the Four Functional Area Test Approaches

Function             Description of decomposition
Health Affairs       Health Affairs divided its function into three business processes: patient care, patient administration, and
                     medical logistics. Health Affairs then broke down each process into several sub-processes, termed “threads.”
Communications       Because Communications cross cuts all functional/operational areas, Communications is testing based on
                     system user. Therefore, Communications divided its function into 263 mission-critical systems. Various military
                     services, Defense agencies, and commanders-in-chief own these systems.
Personnel            Personnel divided its function into six areas: Army, Navy, Air Force, Marine Corps, Civilian, and DEERS/RAPIDS
                     (personnel systems). Personnel then broke down each area into sub-processes, termed “threads.” Personnel is
                     not conducting its own end-to-end tests. Instead, Personnel is participating in and observing service-level
                     testing.
Logistics            Logistics divided its function into four business processes: requisition, receipt, shipment, and inventory control
                     and asset status. Logistics then broke down each process into several sub-processes, termed “threads.”
                     Logistics tested these four processes in two phases: intracomponent (within each military service or Defense
                     agency) and intercomponent (joint testing with military services and Defense agencies).




                                            The test events we selected from each area to review ranged from a simple
                                            test involving two information systems located within one organization to
                                            an intricate test of DOD’s voice and data telecommunications networks
                                            involving several commands and multiple systems. Specifically, the Health
                                            Affairs test event we reviewed assessed the ability of two interfacing
                                            systems to issue and process blood requests after the calendar year
                                            rollover. The Communications test event that we observed was a portion of
                                            a larger test and assessed whether voice communications could be sent
                                            from Fort Monmouth, New Jersey, to St. Louis, Missouri, using DOD
                                            telecommunications networks and equipment and whether messages could
                                            be exchanged using the Defense Messaging System from the Strategic
                                            Command to the Atlantic Command. The Personnel test event assessed the
                                            Army’s ability to create active duty units for deployment from the Army
                                            Reserve and Army National Guard. Finally, the Logistics event focused on
                                            intercomponent testing—between the Army, Air Force, Navy, Marine
                                            Corps, and Defense Logistics Agency—and was designed to verify the Year
                                            2000 readiness of 17 of the 53 total logistics requisition and receipt
                                            processes.




                                            Page 7                                        GAO/AIMD-00-12 DOD Y2K End-to-End Testing
                       B-283564




End-to-End Tests       Available information for the respective areas indicates that, as of August
                       1999, end-to-end tests were largely on schedule and expected to be
Reported to Be on      completed by October 1999. In particular,
Schedule
                       • Health Affairs, which had three primary business processes, completed
                         testing for two—patient care and patient administration business
                         processes—and was on schedule to complete tests for the third—
                         medical logistics—by the end of September 1999.
                       • Personnel tests for the Army, Air Force, and Civilian areas had been
                         completed, while the Navy tests were scheduled to be done October 17,
                         1999. The Marine Corps was behind schedule on one test. However, it
                         completed the test by September 9, 1999.
                       • Logistics intra- and intercomponent tests, which involve four primary
                         business processes—requisition, shipment, receipt, and inventory
                         control and asset status—had been completed for intercomponent
                         transactions. Tests were scheduled to be done by the end of August 1999
                         and, according to Logistics officials, were completed on schedule.
                       • When we briefed the Communications functional area on the results of
                         our review in July 1999, it was unable to provide progress information
                         on all of its 263 mission-critical systems. Subsequently, Communications
                         reported that 77 mission-critical systems had completed testing and 155
                         systems did not require testing.5 The functional area also reported that
                         the remaining 31 mission-critical systems6 did not yet have plans for
                         testing and were considered to be behind schedule.



Selected End-to-End    We selected one test event from each functional area, determined whether
                       the key processes outlined in our Year 2000 testing guide were followed
Test Events Were       and found that DOD had completed the majority of the processes called for
Managed According to   in the guide. For example, for the four test events reviewed, DOD had
                       defined test boundaries, defined exit criteria that would be used to
GAO Guidance           determine when a test was successfully completed, and described how the
                       test results would be documented. While the event’s respective approaches
                       to implementing the key processes varied, these differences were based on


                       5
                        The Communications function considers systems that do not require end-to-end testing to
                       be developmental systems, those that do not process dates, and stand-alone systems.
                       6
                        According to Communications officials, some of these systems are satellite and control
                       systems, which may require waivers.




                       Page 8                                      GAO/AIMD-00-12 DOD Y2K End-to-End Testing
                                        B-283564




                                        the consideration of the event’s scope and complexity and inherent
                                        business risk. Our test guidance permits such differences when justified on
                                        the basis of business value and risk.

                                        Table 3 summarizes the results of our review. As the table notes, of the
                                        possible 44 key processes spanning the 4 test events, 34 were fully satisfied
                                        while another 2 were partially satisfied. For the remaining 8 key processes,
                                        4 were still in progress, and 4 processes concerning correcting defects
                                        found were “not applicable” because initial testing results had not yet
                                        disclosed Year 2000 defects. However, some of the test results that were
                                        obtained during our review were still being analyzed by DOD.



Table 3: Summary of Test Events Satisfying GAO Key Processes

Selected functional area test event                Fully satisfied           Partially satisfied          In progress          N/A         Total
Health Affairs                                                    8                              1                     1          1           11
Communications                                                    9                              0                     1          1           11
Personnel                                                         9                              0                     1          1           11
Logistics                                                         8                              1                     1          1           11
Total                                                            34                              2                     4          4           44


                                        Note: Due to differences in scope and complexity of the test events, the results of individual functions
                                        are not comparable.


                                        In all cases where we determined that the test events’ key processes called
                                        for in our guide had only been partially satisfied, the PSAs and test
                                        managers agreed to address our concerns and initiate corrective actions.
                                        For example:

                                        • While Health Affairs prepared procedures for its test event, these
                                          procedures were not sufficiently detailed and did not define each step to
                                          be executed or precisely define input data. As a result, it was necessary
                                          for system operators to augment the test procedures during the test’s
                                          execution. While this approach was satisfactorily carried out because
                                          the relative simplicity of the test event permitted face-to-face
                                          coordination and synchronization of the procedures, it was
                                          unnecessarily risky and could have been easily avoided by ensuring that
                                          test procedures were complete. Health Affairs officials agreed that more
                                          detailed procedures should have been established, and they committed
                                          to ensuring that other Health Affairs test events have them.



                                        Page 9                                             GAO/AIMD-00-12 DOD Y2K End-to-End Testing
                 B-283564




                 • Although the Logistics function is reliant on telecommunications
                   providers such as military installations and the Defense Information
                   Systems Agency (DISA), at the time of the test event we observed,
                   documentation offering assurances that installations’
                   telecommunications infrastructures were Y2K compliant was not
                   provided by Logistics functional managers. Our test guide states that, in
                   order to ensure that all systems in the chain of support function as
                   intended, the telecommunications infrastructure that interconnects the
                   systems must be compliant and ready for testing. Subsequent to our
                   review, Logistics officials provided information showing that
                   installations’ telecommunications infrastructures had been included in
                   installation test events. Logistics officials agreed, however, that they had
                   not yet confirmed the Y2K compliance of the infrastructures, and
                   reported that they have subsequently initiated steps to do so.



Conclusions      Given that virtually all Defense business functions and military operations
                 rely heavily on technology, it is vital that Year 2000 end-to-end testing
                 efforts be effectively planned and executed. All four of the individual test
                 events that we reviewed were well-managed because each either satisfied
                 or had steps underway or planned to address all relevant end-to-end
                 management key processes specified in our test guide. Moreover,
                 differences between the functional areas’ approaches to implementing
                 these key processes were generally commensurate with the events’ scope
                 and complexity. Finally, reported functional area status information
                 indicates that end-to-end tests are generally progressing on schedule.
                 However, DOD does not yet have assurance that all of its communications
                 systems will be Year 2000 compliant and, as such, should ensure that all
                 mission-critical communications systems are tested.



Recommendation   We recommend that the Secretary of Defense direct the Senior Civilian
                 Official of the Office of the Assistant Secretary of Defense for Command,
                 Control, Communications, and Intelligence to report to the Deputy
                 Secretary immediately on plans for end-to-end testing the 31 mission-
                 critical communications systems, including milestones for executing tests
                 and reporting test results, or to otherwise justify in writing to the Deputy
                 Secretary why any of the systems will not be included in an end-to-end test
                 event.




                 Page 10                               GAO/AIMD-00-12 DOD Y2K End-to-End Testing
                      B-283564




Agency Comments and   DOD concurred with our findings and partially concurred with our
                      recommendation to report to the Deputy Secretary on the status and plans
Our Evaluation        for Y2K testing of the 31 mission-critical communications systems
                      disclosed in our report.

                      In partially concurring on the recommendation, DOD stated that during the
                      July through August 1999 period of our review, testing data in the OSD Y2K
                      database was still evolving, and as a result, test data were incomplete for
                      many of the 31 systems. Since then, resolution has been reached on the
                      testing status of the 31 communications systems. DOD reported and
                      provided documentation to show that (1) Y2K testing for 14 of the 31
                      systems has been completed, (2) 9 systems do not process dates and are
                      exempt from end-to-end test requirements, (3) 4 systems are trusted
                      systems, which cannot be tested in a Y2K environment due to safety,
                      security, or operational necessity reasons, (4) 2 systems are developmental
                      systems that will not be deployed before the millennium rollover, (5) 1
                      system has been reclassified as a nonmission-critical system and does not
                      require additional testing, and (6) 1 system is scheduled to complete testing
                      by October 15, 1999. We have not verified the status information provided
                      by DOD.


                      We are sending copies of this report to Representative John P. Murtha,
                      Ranking Minority Member, Subcommittee on Defense, House
                      Appropriations Committee; Senator John Warner, Chairman, and Senator
                      Carl Levin, Ranking Minority Member, Senate Committee on Armed
                      Services; Senator Ted Stevens, Chairman, and Senator Daniel Inouye,
                      Ranking Minority Member, Subcommittee on Defense, Senate Committee
                      on Appropriations; and Representative Floyd Spence, Chairman, and Ike
                      Skelton, Ranking Minority Member, House Committee on Armed Services.

                      We are also sending copies to the Honorable John Koskinen, Chair of the
                      President’s Year 2000 Conversion Council; the Honorable William Cohen,
                      Secretary of Defense; the Honorable John Hamre, Deputy Secretary of
                      Defense; General Henry Shelton, Chairman of the Joint Chiefs of Staff;
                      Arthur Money, Senior Civilian Official of the Office of the Assistant
                      Secretary of Defense for Command, Control, Communications, and
                      Intelligence; and the Honorable Jacob Lew, Director of the Office of
                      Management and Budget. Copies will also be made available to others upon
                      request.




                      Page 11                               GAO/AIMD-00-12 DOD Y2K End-to-End Testing
B-283564




Should you or your staff have any questions concerning this report, please
contact me at (202) 512-6240. I can also be reached by e-mail at
brockj.aimd@gao.gov. Other points of contact and key contributors to this
report are listed in appendix IV.

Sincerely yours,




Jack L. Brock, Jr.
Director, Governmentwide and Defense
Information Systems




Page 12                             GAO/AIMD-00-12 DOD Y2K End-to-End Testing
B-283564




Page 13    GAO/AIMD-00-12 DOD Y2K End-to-End Testing
Appendix I

Briefing on DOD Y2K Functional End-to-End                                    Appenx
                                                                                  de
                                                                                   is




Testing: Progress and Test Event Management                                  Appendix
                                                                                    I




             DOD Y2K Functional End-to-End
             Testing: Progress and Test Event
                       Management

               House Appropriations Committee
                     September 14, 1999




                     Page 14         GAO/AIMD-00-12 DOD Y2K End-to-End Testing
                    Appendix I
                    Briefing on DOD Y2K Functional End-to-End
                    Testing: Progress and Test Event
                    Management




                        Briefing Overview

•   Introduction
•   Objectives
•   Scope and Methodology
•   Background
•   Objective 1: Status and Progress Information Summary
•   Objective 2: Results of Specific Test Events Reviewed
•   Findings
•   Conclusions
•   Recommendation




2




                    Page 15                                     GAO/AIMD-00-12 DOD Y2K End-to-End Testing
                    Appendix I
                    Briefing on DOD Y2K Functional End-to-End
                    Testing: Progress and Test Event
                    Management




                        Introduction
• In August 1998, the Deputy Secretary of Defense directed
  the Principal Staff Assistants (PSAs), who are responsible
  for Defense’s functional areas, to coordinate, facilitate and
  monitor a series of Year 2000 functional evaluations (Y2K
  functional testing) to ensure functions will operate properly
  in a Y2K environment.

• Y2K functional testing is one of three Defense Y2K end-
  to-end test and evaluation efforts. Y2K end-to-end testing
  is testing performed to verify that systems collectively
  supporting an organizational core business function or
  operation interoperate as intended in a Y2K environment.

• The PSAs core business processes are referred to as
  “threads.”
3




                    Page 16                                     GAO/AIMD-00-12 DOD Y2K End-to-End Testing
                    Appendix I
                    Briefing on DOD Y2K Functional End-to-End
                    Testing: Progress and Test Event
                    Management




                        GAO Objectives

• At the request of the Chairman, House Appropriations
  Committee, Defense Subcommittee, we initiated a
  review of the effectiveness of DOD Y2K functional end-
  to-end test management. The objectives of the review
  are to:

    • Assess status/progress on all test events within four
      functional areas: Health Affairs, Communications,
      Personnel, and Logistics

    • Review the management effectiveness of a critical
      test event for each of the four functional areas
4




                    Page 17                                     GAO/AIMD-00-12 DOD Y2K End-to-End Testing
                   Appendix I
                   Briefing on DOD Y2K Functional End-to-End
                   Testing: Progress and Test Event
                   Management




                       Scope and Methodology
• The test events we reviewed were:
   • Health Affairs
      Patient Care/Issuance and Processing of Blood
       Requests
   • Communications
      Joint User Switch Exercise (JUSE-99-Y2K)
   • Personnel
      Army Personnel/Mobilization/Reserve Unit
   • Logistics
      Joint tests of requisition and receipt processes



5




                   Page 18                                     GAO/AIMD-00-12 DOD Y2K End-to-End Testing
                    Appendix I
                    Briefing on DOD Y2K Functional End-to-End
                    Testing: Progress and Test Event
                    Management




                        Scope and Methodology
• For all test events, we compared reported status
  information to test plan milestones to identify variances.

• For the selected test events, we:
   • observed a part of the test event for each functional
     area and
   • compared event management activities to the key
     processes for end-to-end testing as specified in our
     Year 2000 Test Guide.

• The test events we reviewed were selected based on the
  PSA’s designation that the events were of key importance
  to the business function.

6




                    Page 19                                     GAO/AIMD-00-12 DOD Y2K End-to-End Testing
                    Appendix I
                    Briefing on DOD Y2K Functional End-to-End
                    Testing: Progress and Test Event
                    Management




                        Scope and Methodology

• The test events we reviewed assessed calendar year
  rollover only.

• We briefed the respective PSAs and test event
  managers on the results of our reviews following the
  conclusion of each event.

    •   Health Affairs, June 15, 1999
    •   Communications, July 12, 1999
    •   Personnel, July 15, 1999
    •   Logistics, August 17, 1999


7




                    Page 20                                     GAO/AIMD-00-12 DOD Y2K End-to-End Testing
                      Appendix I
                      Briefing on DOD Y2K Functional End-to-End
                      Testing: Progress and Test Event
                      Management




                          Scope and Methodology
• In commenting on the briefings, PSAs and test managers agreed
  with the results. This briefing consolidates the results of those
  individual briefings, and provides updated status information on
  each functional area’s testing progress. We also clarified the
  results of our analysis of the Logistics test event based on
  additional information provided by Logistics officials after our
  September 14, 1999 briefing.

• Due to differences in scope and complexity, the results of our
  review of the test events are not comparable across functions.

• We performed our audit work in accordance with generally
  accepted government auditing standards between March and
  September 1999.

8




                      Page 21                                     GAO/AIMD-00-12 DOD Y2K End-to-End Testing
                    Appendix I
                    Briefing on DOD Y2K Functional End-to-End
                    Testing: Progress and Test Event
                    Management




                        Background

• DOD requires PSAs to verify that their respective critical
  business processes can function properly in a Y2K
  environment.

• PSAs are to:
   • document mission-critical functions and systems
     supporting those functions,
   • coordinate, facilitate, and monitor Y2K end-to-end test
     and evaluation activities of services, agencies, and
     commands, and
   • in some cases, conduct Y2K end-to-end functional
     testing.

9




                    Page 22                                     GAO/AIMD-00-12 DOD Y2K End-to-End Testing
                     Appendix I
                     Briefing on DOD Y2K Functional End-to-End
                     Testing: Progress and Test Event
                     Management




                         Background

• The scope and complexity of the test events varied in
  terms of, for example, the numbers of systems tested and
  locations involved.

     • Health Affairs test event involved a two system cluster
       in a simulated Y2K environment. In contrast, the
       Logistics test event included 6 operational systems and
       multiple geographical dispersed installations.

     • Personnel test event involved only Army systems.
       Communications, Logistics and Health Affairs test
       events involved more than one Defense component.


10




                     Page 23                                     GAO/AIMD-00-12 DOD Y2K End-to-End Testing
                      Appendix I
                      Briefing on DOD Y2K Functional End-to-End
                      Testing: Progress and Test Event
                      Management




                          Background
• Army Personnel and Logistics chose different methods for
  testing threads that spanned more than one day.

     • Army Personnel divided its threads into what are
       termed “pairs” and tested transactions across each
       “pair” of systems using the critical rollover date.

     • Logistics used a 5-day test cycle that commences on
       the day before rollover (i.e., 9/30/1999, 12/31/1999, and
       2/28/2000). This strategy verifies the mission-critical
       thread across the date rollover, rather than verify all
       systems ability to correctly perform their respective
       function across the date rollover.
11




                      Page 24                                     GAO/AIMD-00-12 DOD Y2K End-to-End Testing
                   Appendix I
                   Briefing on DOD Y2K Functional End-to-End
                   Testing: Progress and Test Event
                   Management




                       Objective 1: Status and
                       Progress Information Summary
• Criteria: Management should collect timely, reliable test
  progress and status information in order to ensure that
  testing is progressing according to plans and that
  problems are resolved and testing requirements are met.

• Finding: Health Affairs, Personnel, and Logistics are
  collecting the progress and status information.

• Finding: At the time of our July 1999 briefing to the PSA,
  Communications was unable to provide us with complete
  information on functional test event’s progress for all
  mission-critical systems it had identified. Subsequently,
  additional information was provided that shows the status
  of all systems; however, Communications reported that
  tests have not yet been planned for 31 of the systems.
12




                   Page 25                                     GAO/AIMD-00-12 DOD Y2K End-to-End Testing
                   Appendix I
                   Briefing on DOD Y2K Functional End-to-End
                   Testing: Progress and Test Event
                   Management




                       Objective 1: Status and
                       Progress Information Summary
                       (continued)

• Finding: With the exception of the tests that need to be
  planned for the 31 Communications systems, the
  available test progress information shows that functional
  areas end-to-end test events are on schedule.




13




                   Page 26                                     GAO/AIMD-00-12 DOD Y2K End-to-End Testing
                      Appendix I
                      Briefing on DOD Y2K Functional End-to-End
                      Testing: Progress and Test Event
                      Management




                          Objective 1: Status and
                          Progress Information


     Health Affairs
     Test Type            Reported Status of Testing Efforts
     • Patient Care       26 out of 26 threads have been tested.
                          Testing was completed on schedule.

     • Patient            30 out of 30 threads have been tested.
       Administration     Testing was completed on schedule.

     • Medical            26 out of 55 threads have been tested.
       Logistics          Testing commenced on July 2, 1999 and is on schedule
                          to be completed by the end of September 1999.




14




                      Page 27                                     GAO/AIMD-00-12 DOD Y2K End-to-End Testing
                                                                Appendix I
                                                                Briefing on DOD Y2K Functional End-to-End
                                                                Testing: Progress and Test Event
                                                                Management




                                                                         Objective 1: Status and
                                                                         Progress Information (continued)
                Communication
                s
                Test Type Reported Status of Testing Events
                • Army                       Of 68 systems:
                                             19 have been tested, 4 have not been planned and are late, and 45 systems
                                             do not require tests.1
                • Air Force                  Of 32 systems:
                                             10 have been tested, 14 have not been planned and are late, and 8 systems
                                             do not require tests.1
                • Navy                       Of 123 systems:
                                             30 have been tested, 4 have not been planned and are late, and 89 systems
                                             do not require tests.1
                • Marine                     Of 6 systems:
                  Corps                      1 has been tested, 1 has not been planned and is late, and 4 systems do not
                                             require tests.1
                • CINCs                      Of 15 systems:
                                             5 have been tested, and 8 have not been planned and are late, and 2
                                             systems do not require tests.1
                • DISA                       Of 19 systems:
                                             12 have been tested and 7 systems do not require tests.1
     1Communications     relied on the Y2K Office database to identify systems that do not require testing. Systems that do not require tests are: 1) developmental systems, 2) do not process
     dates, or 3) are stand alone systems.

15




                                                                Page 28                                                                     GAO/AIMD-00-12 DOD Y2K End-to-End Testing
                                                        Appendix I
                                                        Briefing on DOD Y2K Functional End-to-End
                                                        Testing: Progress and Test Event
                                                        Management




                                                               Objective 1: Status and
                                                               Progress Information (continued)
                Personnel
                Test Type                 Reported Status of Testing Efforts
                • Army                    19 out of 19 threads have been tested.
                                          Testing was completed on schedule.
                • Navy                    Testing is in progress, 0 out of 18 threads
                                          are complete. The testing was started ahead
                                          of schedule and is planned to be completed
                                          by October 17, 1999.
                • Marine                  7 out of 7 threads have been tested.
                  Corps                   The testing began on time; however, 1 thread
                                          was not completed until September 9, 1999,
                                          instead of August 20, 1999.
                • Air Force               10 out of 10 threads have been tested.
                                          Testing was completed on schedule.
                • Civilian                2 out of 2 threads have been tested.
                                          Testing was completed on schedule.
                • DEERS/                  4 out of 4 threads have been tested.
                  RAPIDS2                 Testing was completed on schedule.

     2Defense   Enrollment Eligibility Reporting Systems (DEERS), Real Time Automated Personnel ID Card System (RAPIDS)



16




                                                        Page 29                                                           GAO/AIMD-00-12 DOD Y2K End-to-End Testing
                          Appendix I
                          Briefing on DOD Y2K Functional End-to-End
                          Testing: Progress and Test Event
                          Management




                              Objective 1: Status and
                              Progress Information (continued)

     Logistics
     Test Type                  Reported Status of Testing Efforts
     • Intra-component          All components have completed testing.
     • Inter-component          41 of 41 threads have been tested.
       (Requisition)            Testing was completed on schedule.
     • Inter-component          4 of 4 threads have been tested.
       (Shipment)               Testing was completed on schedule.
     • Inter-component          4 of 4 threads have been tested.
       (Receipt)                Testing was completed on schedule.
     • Inter-component          4 of 4 threads have been tested.
       (Inventory Control &     Testing was completed on schedule.
       Asset Status)




17




                          Page 30                                     GAO/AIMD-00-12 DOD Y2K End-to-End Testing
                                                                          Appendix I
                                                                          Briefing on DOD Y2K Functional End-to-End
                                                                          Testing: Progress and Test Event
                                                                          Management




                                                                                   Objective 2: Results of Specific
                                                                                   Test Events Reviewed
                      GAO Test Criteria                                                                           HA3   Comm3      Pers3     Logistics

                      Define the system boundaries of the end-to-end
                      test(s)
                      Secure the commitment of data exchange partners
 Satisfied
                      Establish an inter-organizational test team

In Progress Confirm Y2K compliance of telecommunications
                      Schedule and plan end-to-end test(s)

 Partially            Prepare end-to-end procedures and data
 Satisfied
                      Define end-to-end test exit criteria

                      Execute end-to-end test(s)
  Not
Applicable Document end-to-end test results
                      Correct Y2K defects

                      Ensure that end-to-end test exit criteria are met

     Note: Due to differences in scope and complexity of the test events, the results of individual functions are not comparable.

     3The   functional areas were abbreviated: Health Affairs (HA), Communications (Comm), and Personnel (Pers)

   18




                                                                          Page 31                                               GAO/AIMD-00-12 DOD Y2K End-to-End Testing
                     Appendix I
                     Briefing on DOD Y2K Functional End-to-End
                     Testing: Progress and Test Event
                     Management




                         Finding: End-to-end test
                         boundaries defined
• Criteria: Organizations need to assess their mission-critical
  business functions and dependencies and, on the basis of
  this assessment, define boundaries for end-to-end tests.

• Finding: All of the functional areas defined test boundaries
  for the test events we witnessed. For example,
  Communications defined boundaries, including geographic
  sites, deployed forces, satellites, and control centers.
  Health Affairs identified three critical functions: Patient Care,
  Patient Administration, and Medical Logistics. Patient Care
  was broken down into mission-critical threads, one of which
  was issuing and processing blood requests.

19




                     Page 32                                     GAO/AIMD-00-12 DOD Y2K End-to-End Testing
                  Appendix I
                  Briefing on DOD Y2K Functional End-to-End
                  Testing: Progress and Test Event
                  Management




                      Finding: Commitment of key
                      data exchange partners
                      secured
• Criteria: Because end-to-end testing addresses business
  areas or functions that involve multiple organizations,
  participation of all key business area data exchange
  partners should be secured.

• Finding: All of the functional areas secured the
  commitment of key data exchange partners. For example,
  Communications data exchange partners from different
  parts of the world agreed to send and receive data,
  including Ft. Monmouth, St. Louis, Atlantic Command, and
  Strategic Command for the test that we reviewed.




20




                  Page 33                                     GAO/AIMD-00-12 DOD Y2K End-to-End Testing
                   Appendix I
                   Briefing on DOD Y2K Functional End-to-End
                   Testing: Progress and Test Event
                   Management




                       Findings: Interorganizational
                       end-to-end teams established
• Criteria: A team composed of representatives from each
  of the organizations participating in the test should be
  formed to manage the planning, execution, and reporting
  of the test.
• Finding: All functional areas established an inter-
  organizational team that planned, executed, and reported
  on the test event. For example, Health Affairs established
  a team that included the TRICARE Management Activity
  Office, IV&V contractors, and technical support staff from
  Advanced Technology Integration Center.
• Finding: Logistics interorganizational team included
  relevant data exchange partners, such as the military
  services and Defense Logistics Agency.

21




                   Page 34                                     GAO/AIMD-00-12 DOD Y2K End-to-End Testing
                    Appendix I
                    Briefing on DOD Y2K Functional End-to-End
                    Testing: Progress and Test Event
                    Management




                        Findings: Y2K compliance of
                        telecommunications partially
                        ensured
• Criteria: In order to ensure that all systems in the chain of
  support function as intended, the telecommunications
  infrastructure that interconnects the systems must be
  compliant and ready for testing.

• Finding: Health Affairs ensured that its
  telecommunications infrastructure for the test (i.e., local
  area network) was Y2K compliant.

• Finding: Communications, Personnel, and Logistics end-
  to-end test events used the Unclassified (but Sensitive)
  Internet Protocol Router Network (NIPRNET). Defense
  Information Systems Agency reported that the NIPRNET
  has been certified Y2K compliant and is fully
  implemented.
22




                    Page 35                                     GAO/AIMD-00-12 DOD Y2K End-to-End Testing
                   Appendix I
                   Briefing on DOD Y2K Functional End-to-End
                   Testing: Progress and Test Event
                   Management




                       Findings: Y2K compliance of
                       telecommunications partially
                       ensured (continued)
• Finding: Communications and Personnel have verified
  that their local telecommunications infrastructure are Y2K
  compliant. Logistics had not verified that its local
  telecommunications infrastructure is Y2K compliant, but
  has since initiated steps to do so.




23




                   Page 36                                     GAO/AIMD-00-12 DOD Y2K End-to-End Testing
                    Appendix I
                    Briefing on DOD Y2K Functional End-to-End
                    Testing: Progress and Test Event
                    Management




                        Finding: End-to-end tests
                        planned and scheduled
• Criteria: A plan should be developed that defines key
  tasks and requirements in preparing for, executing, and
  documenting results of testing. Responsibility for
  performing key tasks and quality assurance/Independent
  Verification and Validation (IV&V) need to be established.

• Finding: All functional areas established plans that
  identified key tasks, schedules, roles and responsibilities
  for the participants in the test events. For example,
  detailed plans and schedules were provided in the Health
  Affairs Master Test Plan and Patient Care Event Plan,
  which addressed the IV&V contractor’s roles and
  responsibilities. Also, Personnel had an Army Master
  Test Plan and Army System Test Plans that identified key
  tasks, schedules, roles and responsibilities.
24




                    Page 37                                     GAO/AIMD-00-12 DOD Y2K End-to-End Testing
                    Appendix I
                    Briefing on DOD Y2K Functional End-to-End
                    Testing: Progress and Test Event
                    Management




                        Findings: End-to-end test
                        procedures prepared
• Criteria: Test procedures and data should include steps,
  cases, and input conditions that verify the correct handling
  of critical dates.

• Finding: Most functional areas prepared test procedures
  and data. For example, Army Personnel prepared baseline
  input test data and generated baseline output. Army
  Personnel then aged the baseline test data inputs and
  outputs to reflect a Y2K environment and used these to
  verify correct handling of the calendar year rollover date.

• Finding: Health Affairs developed high-level test
  procedures; however, it was necessary for operators to
  augment the test procedures to complete the test. Health
  Affairs officials agreed that more detailed procedures
  should have been established.
25




                    Page 38                                     GAO/AIMD-00-12 DOD Y2K End-to-End Testing
                    Appendix I
                    Briefing on DOD Y2K Functional End-to-End
                    Testing: Progress and Test Event
                    Management




                        Finding: Test exit criteria
                        defined
• Criteria: The conditions or requirements for successfully
  completing end-to-end testing need to be established.

• Finding: All functional areas defined test exit criteria for
  the specific test events. For example, Communications
  specified that the test exit criteria for successfully
  completing voice communications were to hear a person
  at the other end of the line with a specified clarity. It
  specified that the test exit criteria for successfully
  completing data communications were for all the message
  recipients to receive the “test” message with all
  attachments over a 72-hour period.


26




                    Page 39                                     GAO/AIMD-00-12 DOD Y2K End-to-End Testing
                    Appendix I
                    Briefing on DOD Y2K Functional End-to-End
                    Testing: Progress and Test Event
                    Management




                        Finding: End-to-end test plans
                        were followed
• Criteria: Tests should be executed in accordance with
  established plans and procedures.

• Finding: All test events adhered to the established plans
  and procedures. For instance, the Army Personnel test
  event was executed in accordance with both the master
  and system test plans. Similarly, the Logistics test event
  was executed in accordance with the test plans and
  procedures.




27




                    Page 40                                     GAO/AIMD-00-12 DOD Y2K End-to-End Testing
                    Appendix I
                    Briefing on DOD Y2K Functional End-to-End
                    Testing: Progress and Test Event
                    Management




                        Finding: Test results were
                        documented
• Criteria: End-to-end test results should be documented so
  that the data can be used to validate that test exit criteria
  have been met and to assess and correct problems
  discovered during testing.

• Finding: All functional areas documented test results.
  Both Health Affairs and Personnel documented results by
  printing out computer screen information. Logistics and
  Communications documented test results on a results
  reporting form.




28




                    Page 41                                     GAO/AIMD-00-12 DOD Y2K End-to-End Testing
                    Appendix I
                    Briefing on DOD Y2K Functional End-to-End
                    Testing: Progress and Test Event
                    Management




                        Finding: No Y2K defects were
                        found during the test event
• Criteria: On the basis of interorganization specified
  criteria, such as defect severity and test exit criteria,
  defects identified during the test should be prioritized and
  corrected.

• Finding: Initial results from the part of the test event that
  we observed indicated that none of the functional areas
  experienced Y2K defects. However, analysis of the test
  results has not yet been completed by all of the functional
  area data analysis centers.




29




                    Page 42                                     GAO/AIMD-00-12 DOD Y2K End-to-End Testing
                     Appendix I
                     Briefing on DOD Y2K Functional End-to-End
                     Testing: Progress and Test Event
                     Management




                         Finding: Effort to ensure
                         end-to-end test meets exit
                         criteria is ongoing
• Criteria: Test results should be compared to test exit
  criteria to ensure that specified conditions are met.

• Finding: All functional areas are still reviewing test results
  at the data collection centers. For example, Logistics’
  Exercise Operation Center is collecting results reporting
  forms and assessing the satisfaction of test exit criteria.
  In addition, Communications-Electronics Command is
  collecting test results for analysis and determination of
  test exit criteria satisfaction.




30




                     Page 43                                     GAO/AIMD-00-12 DOD Y2K End-to-End Testing
                    Appendix I
                    Briefing on DOD Y2K Functional End-to-End
                    Testing: Progress and Test Event
                    Management




                        Conclusions
• Three of the four PSAs were able to provide us with
  information needed to effectively track the progress and
  status of their respective inventories of end-to-end test
  events. Communications was not initially able to provide
  this information on all its test events. While additional
  status information was provided recently, this information
  identified mission-critical systems for which end-to-end
  testing had not yet been planned, thus leaving very little
  time to ensure that they are included in an end-to-end test
  event.

• The four test events that we reviewed were effectively
  managed based on the satisfaction of most, if not all, of
  the key end-to-end testing processes in our guidance.

31




                    Page 44                                     GAO/AIMD-00-12 DOD Y2K End-to-End Testing
                   Appendix I
                   Briefing on DOD Y2K Functional End-to-End
                   Testing: Progress and Test Event
                   Management




                       Conclusions
• The Logistics test event was somewhat limited by not
  ensuring the Year 2000 compliance of the end-to-end
  partners’ local telecommunications equipment connecting
  the partners to the NIPRNET. Logistics has initiated
  appropriate steps to do so.

• The test events’ respective approaches and the degree of
  test planning, execution, and reporting on the four test
  events varied. However, the variances were generally
  commensurate with the unique characteristics of the
  respective events. Such variability in test approaches is
  consistent with our test guidance.



32




                   Page 45                                     GAO/AIMD-00-12 DOD Y2K End-to-End Testing
                    Appendix I
                    Briefing on DOD Y2K Functional End-to-End
                    Testing: Progress and Test Event
                    Management




                        Recommendation

• We recommend that the Secretary of Defense direct the
  Senior Civilian Official of the Office of the Assistant
  Secretary of Defense for Command, Control
  Communications, and Intelligence to report to the Deputy
  Secretary immediately on plans for end-to-end testing the
  31 systems, including milestones for executing tests and
  reporting test results, or to otherwise justify in writing to
  the Deputy Secretary why any of the systems will not be
  included in an end-to-end test event.




33




                    Page 46                                     GAO/AIMD-00-12 DOD Y2K End-to-End Testing
Appendix II

Objectives, Scope, and Methodology                                                                        Appendix
                                                                                                                 I




              As requested by the House Committee on Appropriations, Subcommittee
              on Defense, our objectives were to (1) assess the status and progress of all
              test events within four functional areas—Health Affairs, Communications,
              Personnel, and Logistics—and (2) review the management effectiveness of
              a critical test event for each of the four functional areas. Together, these
              functional areas are performing thousands of end-to-end tests to ensure
              that key business processes and systems can continue operating into the
              year 2000.

              To meet our first objective, we obtained status and progress information
              for the aforementioned functional areas and compared the reported status
              information to milestones contained in individual functional test plans1 to
              identify variances. We discussed this information with DOD officials and
              personnel from the Office of the Assistant Secretary of Defense (Health
              Affairs), the Office of the Assistant Secretary of Defense for Command,
              Control, Communications, and Intelligence, the Office of the
              Undersecretary of Defense (Personnel and Readiness), and the Deputy
              Undersecretary of Defense (Logistics). Also, for each of the functional
              areas, where necessary, we obtained updated status and progress
              information on end-to-end test events.

              To meet our second objective, we selected one specific test event for each
              functional area. The four selected test events were based on each PSA’s
              designation that the test event was of key importance in ensuring that the
              function could continue unaffected at and after the turn of the century. The
              selected test events, and the dates and locations we observed the events,
              were

              • Health Affairs—Patient Care/Issuance and Processing of Blood
                Requests (May 18, 1999) at the Advanced Technology Integration Center
                in Falls Church, Virginia.
              • Communications—Joint User Switch Exercise (JUSE-99-Y2K)
                (June 10, 1999) at the Army Communications-Electronics Command,
                Fort Monmouth, New Jersey.
              • Personnel—Army Personnel/Mobilization/Reserve Unit (June 17, 1999)
                at the Army Personnel Command, Alexandria, Virginia.



              1
               Updated plans included in our review were the December 15, 1998, plan for
              communications; the January 1999, plan for health affairs; the January 31, 1999, plan for
              logistics; and the January 28, 1999, plan for personnel.




              Page 47                                      GAO/AIMD-00-12 DOD Y2K End-to-End Testing
Appendix II
Objectives, Scope, and Methodology




• Logistics—Intercomponent test of requisition and receipt processes
  (June 24-25, 1999) at the Navy Fleet Material Support Office in
  Mechanicsburg, Pennsylvania.

For the selected test events, we interviewed DOD officials and reviewed
pertinent documentation for each event, including test event plans,2
procedures, conditions, exit criteria, results, reports, defects, correction
action plans, and we observed the actual execution of the test event. We
then compared the particulars of each event to our Year 2000 test guide’s
end-to-end testing key processes, identified variances, and discussed with
test officials the reasons for and impacts of any variances.

To supplement our documentation reviews and observations, we
interviewed DOD officials, including those from the TRICARE Military
Health Systems, the Army Communications-Electronics Command, the
Office of the Deputy Chief of Staff for Personnel, and the Office of the
Deputy Undersecretary of Defense (Logistics); test event coordinators; and
test directors regarding additional clarifications after our visit. These
officials addressed telecommunications infrastructure Year 2000
compliance issues. They also provided additional documentation from the
test event we witnessed (i.e., test results, quick look, and final reports).
Due to the time criticality of the year 2000, as our reviews were completed
on each of the functional areas, we provided briefings detailing our
observations to each Defense PSA and test director as follows:

•   Health Affairs—June 15, 1999,
•   Communications—July 12, 1999,
•   Personnel—July 15, 1999, and
•   Logistics—August 17, 1999.

We performed our audit work primarily at DOD headquarters and at the
test event locations described above. We requested and received comments
on a draft of this report from DOD and incorporated those comments as
appropriate. We performed our audit work from March through September
1999 in accordance with generally accepted government auditing
standards.

2
 Test event plans included in our review were: Military Health Systems Patient Care
Functional Readiness Assessment Test Plan, v1.2, April 1999; Communications Joint User
Switch Exercise 99-Y2K Exercise Directive, April 1999; Army Personnel Command Test
Plan, June 1999; Army System Test Plans, June 1999, for EDAS, SIDPERS-3, and PEPDUS;
and Logistics Exercise Directive, May 1999.




Page 48                                    GAO/AIMD-00-12 DOD Y2K End-to-End Testing
Appendix III

Comments From the Department of Defense                            AppendxIi




               Page 49     GAO/AIMD-00-12 DOD Y2K End-to-End Testing
Appendix III
Comments From the Department of Defense




Page 50                                   GAO/AIMD-00-12 DOD Y2K End-to-End Testing
Appendix III
Comments From the Department of Defense




Page 51                                   GAO/AIMD-00-12 DOD Y2K End-to-End Testing
Appendix III
Comments From the Department of Defense




Page 52                                   GAO/AIMD-00-12 DOD Y2K End-to-End Testing
Appendix IV

GAO Contact and Staff Acknowledgements                                                            AppendixIV




GAO Contact           Randolph C. Hite, (202) 512-6240




Acknowledgements      In addition to those named above, Ronald B. Bageant, Cristina T. Chaplain,
                      Katherine I. Chu, Richard B. Hung, Myong S. Kim, Madhav S. Panwar, and
                      Alicia L. Sommers made key contributions to this report.




(511659)      Leter   Page 53                             GAO/AIMD-00-12 DOD Y2K End-to-End Testing
Ordering Information

The first copy of each GAO report and testimony is free.
Additional copies are $2 each. Orders should be sent to the
following address, accompanied by a check or money order made
out to the Superintendent of Documents, when necessary, VISA and
MasterCard credit cards are accepted, also.

Orders for 100 or more copies to be mailed to a single address are
discounted 25 percent.

Orders by mail:

U.S. General Accounting Office
P.O. Box 37050
Washington, DC 20013

or visit:

Room 1100
700 4th St. NW (corner of 4th and G Sts. NW)
U.S. General Accounting Office
Washington, DC

Orders may also be placed by calling (202) 512-6000
or by using fax number (202) 512-6061, or TDD (202) 512-2537.

Each day, GAO issues a list of newly available reports and
testimony. To receive facsimile copies of the daily list or any list
from the past 30 days, please call (202) 512-6000 using a touchtone
phone. A recorded menu will provide information on how to obtain
these lists.

For information on how to access GAO reports on the INTERNET,
send an e-mail message with “info” in the body to:

info@www.gao.gov

or visit GAO’s World Wide Web Home Page at:

http://www.gao.gov
United States                       Bulk Rate
General Accounting Office      Postage & Fees Paid
Washington, D.C. 20548-0001           GAO
                                 Permit No. GI00
Official Business
Penalty for Private Use $300

Address Correction Requested