oversight

Military Readiness: DOD Needs to Better Manage Automatic Test Equipment Modernization

Published by the Government Accountability Office on 2003-03-31.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

             United States General Accounting Office

GAO          Report to the Chairman, Subcommittee
             on National Security, Emerging Threats
             and International Relations, Committee
             on Government Reform, House of
             Representatives
March 2003
             MILITARY
             READINESS
             DOD Needs to Better
             Manage Automatic
             Test Equipment
             Modernization




GAO-03-451
                                               March 2003


                                               MILITARY READINESS

                                               DOD Needs to Better Manage Automatic
Highlights of GAO-03-451, a report to the
Chairman, Subcommittee on National             Test Equipment Modernization
Security, Emerging Threats and
International Relations, House Committee
on Government Reform




The services have billions of                  DOD and the services face growing concerns regarding obsolete automatic
dollars worth of outdated and                  test equipment, given the high costs of modernizing or replacing it and its
obsolete automatic test equipment              potential effect on aircraft readiness. The Navy and Air Force, for example,
(ATE) used to test components                  estimate that they will spend billions of dollars to modernize or replace
on military aircraft or weapon                 this equipment, much of which was acquired in the 1970s and 1980s. In the
systems. Department of Defense
(DOD) policy advocates the
                                               meantime, the aging testers are becoming increasingly out of date and more
development and acquisition of test            difficult to support. When the testers do not work properly, maintenance can
equipment that can be used on                  suffer and readiness can be adversely affected.
multiple types of weapon systems
and aircraft and used                          Since 1994, DOD policy has advocated the acquisition of test equipment
interchangeably between the                    that can be used on multiple weapon systems and aircraft and can be used
services.                                      interchangeably between the services; progress in this regard has been slow.
                                               For example, although the Navy set out in 1991 to replace 25 major tester
At the request of the                          types with one standard tester by 2000, budget cuts and delays in developing
Subcommittee’s Chairman,                       software have resulted in delays in completing the replacement of these
GAO examined the problems that                 obsolete testers until 2008. The Air Force has only recently initiated a test
the Air Force, Navy, and Marine
Corps are facing with this aging
                                               equipment modernization plan. However, little evidence suggests that
equipment and their efforts to                 consideration is being given to the acquisition of equipment that would have
comply with DOD policy.                        common utility for more than one weapon system as DOD policy advocates.
                                               For procurement of new weapon systems, the Air Force is giving little
                                               consideration to the use of a common tester, while a common tester is
                                               planned for use as the primary tester for the Joint Strike Fighter.
GAO recommends that the
Secretary of Defense reemphasize               Although DOD tasked the Navy as its Executive Agent for automatic test
the policy and reconsider the
                                               equipment in 1994, the agent has made only limited progress in achieving
organizational placement and
authority of the Executive Agent               compliance across all the services with DOD policy advocating the
for ATE.                                       development of common systems. While the Executive Agent can point to
                                               some successes in individual systems, its officials acknowledged that the
DOD concurred with GAO’s                       organization does not have sufficient authority or resources to fully
recommendations and agreed that                implement the policy and achieve the maximum commonality possible.
its Executive Agent for ATE
should be given the authority and
resources to more effectively
fulfill the Department’s
oversight responsibilities.




www.gao.gov/cgi-bin/getrpt?GAO-03-451.

To view the full report, including the scope
and methodology, click on the link above.      The Navy’s Consolidated Automated Support System will minimize unique types of testers.
For more information, contact Neal Curtin at
(757) 552-8100 or curtinn@gao.gov.
Contents


Letter                                                                                                            1
                       Results in Brief                                                                           3
                       Background                                                                                 4
                       Aging ATE Presents Major Challenges to DOD                                                 5
                       DOD Has Had Limited Success in Fostering Commonality                                       7
                       Conclusions                                                                               15
                       Recommendations for Executive Action                                                      16
                       Agency Comments and Our Evaluation                                                        16

Appendix I             Scope and Methodology                                                                     18



Appendix II            Comments from the Department of Defense                                                   20



Related GAO Products                                                                                             23



Figures
                       Figure 1: Schematic of an Automatic Test System, Including ATE
                                and Test Program Set Components                                                  2
                       Figure 2: CASS Station                                                                    9



                       Abbreviations

                       ATE          automatic test equipment
                       CASS         Consolidated Automated Support System
                       DOD          Department of Defense
                       GAO          General Accounting Office
                       JSF          Joint Strike Fighter


                       This is a work of the U.S. Government and is not subject to copyright protection in the
                       United States. It may be reproduced and distributed in its entirety without further
                       permission from GAO. It may contain copyrighted graphics, images or other materials.
                       Permission from the copyright holder may be necessary should you wish to reproduce
                       copyrighted materials separately from GAO’s product.




                       Page i                       GAO-03-451 DOD Automated Test Equipment Modernization
United States General Accounting Office
Washington, DC 20548




                                   March 31, 2003

                                   The Honorable Christopher Shays
                                   Chairman, Subcommittee on National Security,
                                   Emerging Threats and International Relations
                                   Committee on Government Reform
                                   House of Representatives

                                   Dear Mr. Chairman:

                                   One of the major challenges facing the military services is that of
                                   modernizing billions of dollars’ worth of aging and increasingly obsolete
                                   automatic test equipment1 (ATE) used to troubleshoot and diagnose
                                   components of aircraft or weapon systems. Since 1994, Department of
                                   Defense (DOD) policy has advocated a reduction in the proliferation of
                                   testers that are unique to one type of weapon system or aircraft, favoring
                                   the development or acquisition of testers that (1) are interoperable2 within
                                   a service and between the services and (2) can be used on many different
                                   components of multiple types of aircraft and weapon systems. As the
                                   services modernize ATE, they are challenged to adhere to DOD policy and
                                   reduce the number of unique testers.

                                   ATE with its test program sets, including test software, an interface device
                                   that connects the ATE to the item being tested, and documentation make
                                   up an automatic test system. (See fig. 1.)




                                   1
                                    The test hardware and software of an integrated assembly of stimulus, measurement,
                                   and switching components under computer control that is capable of processing
                                   software routines designed specifically to test a particular item or group of items.
                                   2
                                    “Interoperability” is the ability of systems to provide data or material to and accept the
                                   same from other systems and to operate effectively together.



                                   Page 1                         GAO-03-451 DOD Automated Test Equipment Modernization
Figure 1: Schematic of an Automatic Test System, Including ATE and Test Program
Set Components




ATE is used by the services at all maintenance levels (from flight lines
for routine maintenance to depots for major overhauls and factories for
production and acceptance testing) to test electronic systems and
components that are difficult or impossible to test manually, to isolate
system malfunctions, and to verify that systems are operating properly.
These testers can be made to examine a single aircraft system, various
components of an aircraft, or multiple components of different aircraft.

Because of your concerns regarding DOD’s continued reporting of spare
parts shortages and the potential impact that ATE obsolescence could
have on the readiness of military aircraft, you asked us to determine
whether DOD and the services are giving adequate attention to ATE
modernization efforts. Specifically, our objectives were to identify
(1) what problems the Air Force and Navy3 are facing with their ATE


3
 The term “Navy,” as presented in this report, represents Naval Aviation, which includes
the Marine Corps.



Page 2                       GAO-03-451 DOD Automated Test Equipment Modernization
                   and (2) how successful DOD, the Air Force, and the Navy have been in
                   addressing the proliferation of unique testers.

                   Our review included ATE for aircraft managed by the Air Force and the
                   Navy and included information on ATE acquisition for two fighter aircraft
                   currently under development: the multiservice Joint Strike Fighter (JSF)
                   and the Air Force’s F/A-22. Our scope and methodology are described in
                   more detail in appendix I. We performed our review from January 2002
                   through March 2003 in accordance with generally accepted government
                   auditing standards.


                   DOD and the services face growing concerns regarding obsolete ATE,
Results in Brief   given the high costs of modernizing or replacing this type of equipment
                   and its potential adverse impact on aircraft readiness. ATE acquired in the
                   1970s and 1980s is becoming increasingly out-of-date and more difficult
                   to support. These obsolescence issues are further aggravated by new
                   technologies that, in some cases, make ATE obsolete even before the new
                   testers can be fully fielded. Also, older testers are kept much longer than
                   initially planned because the weapon systems they support are being kept
                   longer. Repair parts for older ATE are becoming increasingly scarce, as
                   more contractors discontinue their support.

                   Although exact cost figures are not available, the services estimate that
                   they will need several billion dollars in the coming years either to acquire
                   new testers or modernize existing ones. Although the services do not
                   maintain data that allow them to measure the extent to which obsolete
                   ATE affects readiness, according to DOD readiness reports, only
                   28 percent of Air Force, Navy, and Marine Corps key aircraft models met
                   their readiness goals in 2002. Although a combination of factors affects
                   readiness goals, the availability of spare parts is a key contributor to
                   readiness, and the performance of ATE significantly affects the supply of
                   spare parts.

                   For years, DOD’s policy has aimed to minimize the acquisition of ATE
                   that is unique to a particular weapon system; however, the implementation
                   of this policy has been slow. In 1994, DOD appointed the Navy as its
                   Executive Agent to oversee policy implementation; however, according
                   to Executive Agent officials, the Navy has not had the authority or
                   resources to effectively carry out this oversight. The services lack ATE




                   Page 3                   GAO-03-451 DOD Automated Test Equipment Modernization
             modernization plans, and there is no DOD-wide approach to ensure that all
             ATE acquisitions and modernizations are identified in an early enough
             stage to ensure that commonality4 and interoperability are adequately
             considered. Without sufficient information concerning the magnitude of
             the services’ modernization efforts or a departmentwide approach to
             accomplish ATE modernization, the department faces a very expensive
             and time-consuming ATE modernization effort, with the continued
             proliferation of unique testers and no assurance that resources are
             allocated in the most effective manner. As a result, some ATE
             modernization and acquisition planning is being done with little
             consideration to commonality. For example, Executive Agent officials said
             that they have not had contact with the Air Force’s F/A-22 project office
             concerning ATE since 1994, and it does not appear that commonality is
             being considered or that unique ATE development will be minimized.

             We are making several recommendations aimed at reinforcing DOD’s
             stated goal of achieving more commonality of test equipment and
             strengthening the department’s oversight. DOD concurred with our
             recommendations and agreed to reemphasize its policy that common
             automatic test equipment be developed to the maximum extent possible.
             In addition, DOD agreed that its Executive Agent for ATE should be given
             the authority and resources to more effectively fulfill its oversight
             responsibilities. DOD’s comments on our report are reprinted in their
             entirety in appendix II.


             It is estimated that DOD employs more than 400 different tester types.
Background   This equipment is used to diagnose problems in aircraft avionics and
             weapon system components so that the component can be repaired and
             replaced on the aircraft or put into the supply system for future use. For
             example, testers may be used to diagnose problems with aircraft radars,
             guidance and control systems, or weapon systems. According to DOD, the
             department spent over $50 billion in its acquisition and support of ATE
             from 1980 through 1992, and the procurement was characterized by the
             proliferation of testers designed to support a specific weapon system or
             component. These testers are quickly becoming obsolete and more
             difficult and costly to maintain because they may no longer be in
             production and parts may not be readily available. Over the years, various
             studies have criticized the continued proliferation of unique ATE and


             4
                 Test equipment that can be used on multiple airframes and weapon systems.




             Page 4                         GAO-03-451 DOD Automated Test Equipment Modernization
                      highlighted the need for the development and acquisition of testers that
                      can be used to test more than one system or component.

                      In September 1993, the House Appropriations Committee recommended
                      that the Secretary of Defense develop a DOD-wide policy requiring ATE
                      commonality among the services, along with a formal implementation
                      mechanism with sufficient authority, staffing, and funding to ensure
                      compliance.5 In 1994, DOD established a policy stating that managers
                      of DOD programs should select families of testers or commercial
                      off-the-shelf components to meet all ATE acquisition needs and that the
                      introduction of unique testers should be minimized. DOD designated the
                      Navy at that time as its Executive Agent to oversee policy implementation
                      in all services, and identified a goal of reducing life-cycle costs and
                      providing greater ATE commonality and interoperability. Additional DOD
                      guidance published in 1996 and 1997 required that all ATE acquisitions be
                      part of the approved families of testers or commercial off-the-shelf.


                      DOD faces major challenges with aging and increasingly obsolete ATE.
Aging ATE Presents    These problems include the high costs of maintaining and replacing
Major Challenges      ATE and the declining availability of spare parts for the aging testers. In
                      addition, several DOD organizations, including the Navy Inspector
to DOD                General, have suggested that aging and obsolete ATE may adversely affect
                      aviation readiness.


Modernization Costs   Departmentwide estimates of funds needed for ATE modernization and
Are Substantial       acquisition are not readily available. However, according to Air Force and
                      Navy ATE managers, most of the services’ ATE is obsolete and will need to
                      be upgraded or replaced over the next several years. Our study confirmed
                      that replacement and modernization costs would be substantial. The Navy,
                      for example, spent about $1.5 billion from fiscal years 1990 through 2002
                      for the acquisition of its primary family of testers and plans to spend an
                      additional $430 million through fiscal year 2007. Additionally, the Navy
                      estimates that it plans to spend $584 million through fiscal year 2007 to
                      adapt existing test program sets necessary to perform specific tests of the
                      various aircraft components supported by this family of testers. The Navy
                      also anticipates spending an additional $584 million to develop program
                      test sets for new weapon system requirements.


                      5
                          House Report No 103-254, Sept. 22, 1993.




                      Page 5                          GAO-03-451 DOD Automated Test Equipment Modernization
                     Information on the Air Force’s spending for ATE modernization is
                     somewhat sketchy, as limited data are available centrally for individual
                     weapon systems. According to a recent study done for the Air Force, the
                     service has not developed a plan that allows modernization funding
                     requirements to be determined. However, estimates are available for
                     selected systems. The F-15 fighter program office, for example, is spending
                     approximately $325 million on just one tester that will be fielded in 2004. It
                     also plans to upgrade its electronic warfare tester, which is one of seven
                     primary testers for the aircraft, at a cost of over $40 million. A 2002 study
                     of B-52 bomber ATE identified obsolescence issues associated with six of
                     the aircraft’s seven major testers that will require more than $140 million
                     in the near future. Similarly, the upgrade of a unique B-1 bomber tester is
                     expected to exceed $15 million, even though the Air Force is considering
                     replacing this tester and has already begun planning the acquisition. The
                     latest estimate for the new tester is $190 million. Current ATE estimates
                     for the F/A-22, which is still under development, are not available.
                     However, estimates made early in the development phase exceeded
                     $1.5 billion.


Readiness Could Be   ATE is becoming increasingly out-of-date and more difficult to support.
Adversely Affected   And, according to service officials, using this outdated equipment to
                     perform required tests in a timely manner is becoming increasingly
                     challenging. Although the services could not quantify the extent that tester
                     problems affect readiness, service officials noted that without adequate
                     test equipment to diagnose problems, components cannot be repaired in
                     a timely manner and the mission capability of military aircraft can be
                     adversely affected. In August 2000, the Navy Inspector General identified
                     shortfalls in ATE as having a negative impact on naval aviation and, in
                     particular, on the availability of repaired components. During the same
                     time frame, a Navy operational advisory group, recognizing the importance
                     of ATE in maintaining readiness, ranked support equipment, including
                     ATE, as one of its top 20 readiness issues.

                     We have issued several reports in the recent past addressing the shortage
                     of spare parts—a potential result of ATE problems. In addition, according
                     to DOD readiness reports, only 28 percent of Air Force, Navy, and Marine
                     Corps key aircraft models met their readiness goals in fiscal year 2002.
                     Although difficulties in meeting these goals are caused by a complex
                     combination of interrelated logistical and operational factors, the shortage
                     of spare parts was a major cause. ATE plays a significant role in the supply
                     of available spares, since this equipment affects both how many parts are
                     taken out of service for repair and how quickly they are repaired and


                     Page 6                   GAO-03-451 DOD Automated Test Equipment Modernization
                       returned. We reported that maintenance and repair facilities routinely
                       work around spare parts shortages by removing a working part from one
                       aircraft to replace a nonworking part in another aircraft, a practice called
                       “cannibalization.”6 And, although the services do not record increases in
                       cannibalizations that are caused by ATE problems, the services use
                       cannibalization as a routine maintenance practice when testers are not
                       available or not working properly.

                       In July 2001, we reported that as a result of ATE not working properly,
                       unfilled requisitions were adversely affecting the mission capability of F-14
                       aircraft.7 In another case, more than 1,200 Air Force B-1 bomber
                       components were backlogged and could not be repaired because of the
                       same reason. Although we were unable to measure specific reductions in
                       the readiness of F-14 and B-1 aircraft as a result of this problem, mission
                       capable rates for the B-1 in fiscal years 1998-2002 averaged approximately
                       55 percent, compared with the goal of 67 percent, while mission capable
                       rates for the F-14D, during the same period, averaged 67 percent,
                       compared with a goal of 71 percent. Additionally, the Air Force’s 2002 B-52
                       study concluded that six of the seven major testers used to test B-52
                       components need to be modified or replaced or the availability of the
                       aircraft will be adversely affected as early as 2006. Air Force officials
                       believe that similar problems will continue unless the service undertakes a
                       major ATE modernization or replacement program.


                       Since the early 1990s, DOD policies have addressed the need for
DOD Has Had Limited    commonality in ATE acquisition and modernization. Although the services
Success in Fostering   have been making some progress, efforts to comply with these policies
                       have been slow. For example, although the Navy has developed a single
Commonality            family of testers to work on many of its aircraft components, after
                       11 years, the replacement of its obsolete testers aboard aircraft carriers
                       and shore maintenance facilities has not been completed. In addition,
                       strategic planning for the modernization of automatic test equipment at
                       Navy depots has only recently been initiated.




                       6
                        See U.S. General Accounting Office, Military Aircraft: Services Need Strategies to
                       Reduce Cannibalizations, GAO-02-86 (Washington, D.C.: Nov. 21, 2001).
                       7
                        See U.S. General Accounting Office, Navy Inventory: Parts Shortages Are Impacting
                       Operations and Maintenance Effectiveness, GAO-01-771 (Washington, D.C.: July 31, 2001).




                       Page 7                       GAO-03-451 DOD Automated Test Equipment Modernization
                       Historically, the Air Force has not had a service-level ATE standardization
                       policy and has essentially pursued unique ATE solutions for each weapon
                       system. Since individual aircraft program offices have been doing their
                       own planning for modernization, the Air Force has given little
                       consideration to having common ATE or testers that are interoperable
                       with those of other services. Planning for the Air Force’s latest aircraft
                       acquisition, the F/A-22, calls for the development of automatic test
                       equipment that will be unique to that aircraft. In August 2002, the Air
                       Force initiated a planning effort to determine its long-term servicewide
                       ATE modernization needs.


The Navy Has Been      According to Navy reports, obsolete ATE results in higher backlogs and
Slow in Fielding Its   increased flying hour costs, and adversely affects aircraft readiness. The
Common Tester          Navy recognized years ago, and prior to the establishment of DOD’s
                       1994 ATE standardization policy, that its ATE was becoming obsolete.
                       In the 1980s the Navy embarked upon an ATE standardization program to
                       replace 25 of its testers with one standard ATE family, the Consolidated
                       Automated Support System (CASS), to minimize unique types of testers.
                       The Navy designed CASS to be used at maintenance activities both ashore
                       and afloat. In 1991, the Navy began to produce CASS for the general
                       purpose testing of equipment such as radios, radars, and electro-optics.
                       (See fig. 2.)




                       Page 8                  GAO-03-451 DOD Automated Test Equipment Modernization
Figure 2: CASS Station




                         CASS’s replacement of 25 types of obsolete testers, in support of
                         2,458 weapon system components, was scheduled for completion by
                         fiscal year 2000. However, according to Navy officials, because of budget
                         cuts that caused delays in developing the test program sets, only 4 of the
                         25 have been completely replaced by CASS, and 8 test sets have been
                         partially replaced. Navy officials told us that the completion schedule has
                         slipped to fiscal year 2008 for aircraft carriers and shore maintenance
                         facilities and could be much longer for aviation depots.

                         The Navy reports that the replacement of these testers with CASS
                         stations, when complete, will reduce the number of test-related enlisted
                         occupational specialties from 32 to 4, thus reducing training requirements.
                         In addition, CASS will reduce the requirement for test equipment
                         operators aboard each aircraft carrier from 105 to 54, and at the same



                         Page 9                   GAO-03-451 DOD Automated Test Equipment Modernization
                          time reduce space requirements for testers from 2,700 to 1,900 square feet.
                          Spare parts needed to repair testers will be reduced from 30,000 to 3,800.
                          According to Navy officials, however, the revised completion schedule will
                          not allow for the timely replacement of aging ATE, and these delays will
                          adversely affect aircraft readiness.

                          In addition to schedule slippage, the original CASS equipment was
                          fielded about 10 years ago, uses 15-year-old technology and, according
                          to Navy ATE program managers, is in need of an upgrade. Accordingly,
                          by 2006, the first production units will have reached the point where wear
                          and obsolete components will drive supporting costs to unacceptable
                          levels and create a need for replacement and modernization. The Navy
                          has begun modernization planning for CASS, including upgrades through
                          fiscal year 2014.

                          Integrating CASS into Navy depots may further delay ATE commonality
                          within the service. For example, a 2001 Navy report, addressing total ATE
                          ownership costs, noted that the depots have not maximized the use of
                          CASS because of the limited availability of capital investment funds. In
                          addition, at one depot we found some reluctance to use CASS. This depot
                          had four CASS stations that had never been used—two were delivered in
                          1999 and installed in December 2000 and February 2001, while two others
                          delivered in 2000 were still in crates. Depot officials said that they had
                          elected not to put the equipment on-line, as they wanted to avoid paying
                          for overhead and maintenance, especially without the workload to justify
                          their use. They also noted that the development of the test program sets
                          needed to use the CASS has been slow, thereby slowing the fielding of the
                          equipment. The Navy has only recently begun a servicewide planning
                          effort to modernize its depot-level testers and determine how best to
                          integrate CASS into its depot maintenance strategy.


Air Force’s Approach      Unlike the Navy, the Air Force has not made commonality a priority but
Has Resulted in Limited   has pursued unique ATE solutions for each weapon system. In addition,
Commonality               it has only recently initiated efforts to collect information on ATE in
                          its inventory, including the equipment’s condition and its need for
                          modernization or replacement. Because the Air Force has not made
                          concerted efforts to use one system to service multiple aircraft platforms,
                          it has not taken advantage of efficiencies and potential savings such as
                          those expected by the Navy as a result of CASS.

                          Although the Air Force is developing plans to modernize its ATE, and
                          although its policy is to consider developing common testers, it does not


                          Page 10                 GAO-03-451 DOD Automated Test Equipment Modernization
    yet have an overall plan to guide its modernization efforts and has made
    limited progress in this area. Furthermore, it does not have a process in
    place to ensure that commonality is given adequate consideration in its
    ATE acquisition and modernization.

    The Air Force has been primarily upgrading—rather than replacing—aging
    ATE; leaving ATE management up to individual program managers. In
    most cases, it relies on contractors to provide support for ATE, leaving it
    vulnerable to contractors who may decide to stop supporting testers when
    maintaining them is no longer profitable.

    In early 2001, the Air Force organized the Warner Robins Air Logistics
    Center Automatic Test System Division to work with program offices
    on ATE issues. The Division has recently initiated efforts to establish a
    database of all contractors that are capable of supporting existing ATE
    to help identify emerging supportability issues. Although the office is
    responsible for fostering the adoption and use of common families
    of testers, it has no final decision-making authority regarding ATE
    modernizations and no control over funding decisions on these matters.
    Division officials told us that they work with individual project offices to
    encourage them to use common ATE, but individual project offices make
    the final decisions.

    In our opinion, leaving these ATE decisions to the individual Air Force
    project offices has led to some questionable and unnecessary
    expenditures. For example:

•   The Air Force will spend approximately $325 million to replace a tester for
    the F-15 with one that has been under development for almost 10 years
    and is already obsolete. The new tester, called the Electronic System Test
    Set, is not expected to be fielded until 2004. However, this electronic tester
    already needs an upgrade that will cost more than $24 million. Because the
    new tester will not be able to perform all the required tests, the Air Force
    will have to keep the old tester too.
•   The Air Force is spending over $15 million for an interim modernization of
    its intermediate automatic test equipment for its B-1 aircraft while, at the
    same time, a new tester is being developed. If the Air Force had taken
    the necessary steps to replace this obsolete tester in a timely manner,
    these duplicative costs could likely have been avoided, and overall ATE
    modernization costs reduced. According to an Air Force official, the
    program office should have begun the acquisition of a replacement tester
    several years ago, but funding was not available. The service is now
    considering acquiring a replacement tester estimated to cost $190 million.



    Page 11                  GAO-03-451 DOD Automated Test Equipment Modernization
                          The Air Force’s Warner Robins Air Logistics Center Automatic Test
                          System Division is developing a strategic plan that is expected to serve
                          as a management plan for meeting long-term ATE needs. The Division
                          plans to develop a baseline of its current tester capabilities, address
                          supportability and sustainability issues, and determine whether tester
                          failures adversely affect the availability of aircraft weapon systems. In
                          addition, it will evaluate replacement and modernization alternatives,
                          taking into account life-cycle costs and the potential for developing
                          common testers. The plan’s implementation is expected to take years
                          to complete.


Services’ Approaches in   While most of our work focused on ATE for the current aircraft inventory,
Developing Testers for    we also wanted to see how the services were approaching development of
Two New Aircraft Differ   testers for two new aircraft, the Joint Strike Fighter and the F/A-22. We
                          found that very different approaches are being taken in the development
                          of ATE for these two aircraft. The JSF, for example, will have a single
                          tester, made up almost entirely of commercial components, which will test
                          all components on the aircraft. The F/A-22 project office has no assurance
                          that commonality is being considered in its tester development or that
                          DOD’s policy to minimize unique ATE development is being followed.

                          The JSF originated in the early 1990s through the restructuring and
                          integration of several tactical aircraft and technology initiatives already
                          under way. The goal was to use the latest technology in a common family
                          of aircraft to meet the future strike requirements of the services and
                          U.S. allies. The JSF support strategy is built upon a single tester to be
                          used by the Air Force, Navy, and Marine Corps, as well as by foreign
                          partners, to test all avionics and weapon systems on the aircraft.

                          The JSF tester, referred to as the LM-STAR, is made up almost entirely
                          of commercially available components, contributing to readily available
                          spares and less complicated upgrades. It will be used during development
                          and after the aircraft is fielded. Vendors participating in the development
                          of avionics and weapon system components for the aircraft are required
                          to produce these components so that their testing can be done by the
                          LM-STAR. A total of $99 million has been allocated for the purchase and
                          support of 88 of these testers during the development phase. While a final
                          decision has not been made on whether maintenance support for the
                          aircraft will be provided by the contractor or at a military facility, the
                          system project office is taking steps to ensure that this tester can be
                          used regardless of where maintenance is accomplished.


                          Page 12                  GAO-03-451 DOD Automated Test Equipment Modernization
                      By contrast, Air Force F/A-22 program officials told us that they have not
                      made a decision as to what testers will be used to support this new
                      aircraft, which began development in 1991. The project office has not
                      ensured that all components for the F/A-22 can be tested with a single
                      tester. Project officials told us that the F/A-22 is a very complex aircraft
                      and that opportunities to take advantage of common equipment will be
                      limited. Yet, the same contractor that is developing the F/A-22 is also
                      involved in the JSF, which is also very advanced and complex and which
                      uses a common family of testers. While current projections of ATE costs
                      are not available, estimates made early in the F/A-22 development phase
                      exceeded $1.5 billion.8


DOD Oversight Needs   In 1993, the House Appropriations Committee recommended that a
Strengthening         DOD-wide policy be adopted requiring that the introduction of unique ATE
                      be minimized and that DOD establish an oversight system with sufficient
                      authority, staffing, and funding to ensure compliance. DOD established a
                      policy requiring the services to minimize unique types of testers to reduce
                      redundant investments and lessen long-term costs, leveraging its
                      investments in testers across the entire DOD establishment. In 1994,
                      DOD appointed the Navy as its Executive Agent for ATE to oversee the
                      implementation of this policy. As part of the tasking, the Executive Agent
                      for ATE was directed to establish a process so that programs proposing
                      not to use the DOD-designated standard of ATE families would have to
                      request a waiver. In accordance with the direction provided by DOD, the
                      Executive Agent established a waiver process. According to data provided
                      by the Executive Agent, since its inception, 30 requests for waivers were
                      submitted for their review. Our analysis indicated that 15 of these requests
                      resulted in waivers or concurrence. The remaining requests were never
                      finalized, were returned to the originating office for further action, or were


                      8
                        In providing technical comments on our draft report, the F/A-22 project office
                      reiterated that because of the complexity of this aircraft, opportunities to use common
                      test equipment were limited. The project office indicated that designing one set of ATE to
                      test all components could make that tester overly complex and expensive. In addition, the
                      project office indicated that it had taken advantage of commercial testers and incorporated
                      diagnostics into the avionics themselves. Finally, the project office indicated that the
                      estimate for ATE of more than $1.5 billion made early in the development phase was
                      correct but misleading since the support philosophy had changed. We continue to believe
                      that the F/A-22 project office has not ensured that tester commonality is being considered.
                      The project office was not able to provide information concerning the ATE used or planned
                      for the F/A-22 or estimates of ATE costs. Furthermore, there was no evidence of Executive
                      Agent involvement in the F/A-22 program since November 1994, and Executive Agent
                      officials do not know whether common testers are being considered.




                      Page 13                      GAO-03-451 DOD Automated Test Equipment Modernization
determined not to require waivers. According to Executive Agent officials,
the Executive Agent makes recommendations concerning the waiver
requests, but it does not have the authority to disapprove them.

Executive Agent officials told us however, that they have no assurance
that all tester acquisitions and modifications are identified or that all
required waivers are requested. As a result, they may not be aware of all
ATE modifications or acquisitions or they may not be made aware of
such until the process is already under way and it is too late to affect
any change. For example, the Air Force did not request a waiver for a
$77 million modification to ATE supporting the low altitude navigation and
targeting infrared for night (LANTIRN). LANTIRN is a pod system that
supports the F-15, F-16, and F-14 aircraft in low-level navigation and lazing
targets. In its technical comments on our draft report, however, Air Force
officials indicated that owing to the nature of the LANTIRN modification,
a DOD waiver was not required. We continue to believe, however, that
the Executive Agent should be notified of tester modifications of
this magnitude.

In addition to having no assurance that all tester acquisitions and
modifications are identified, Executive Agent officials told us they do
not have the necessary enforcement authority or resources to effectively
implement the waiver process even when they know of the planned
acquisition or modification. For example, Executive Agent officials held
several discussions with F/A-22 program officials, early in the development
phase, concerning the use of common testers; however, there was no
evidence of the Executive Agent’s involvement in F/A-22 ATE development
since November 1994. Executive Agent officials do not know whether
common testers are being considered.

As DOD’s Executive Agent for ATE, the Navy has achieved some success
in encouraging the development of common testers and in dealing with
technical issues affecting all services. In September 1998, the Executive
Agent for ATE reported that DOD had avoided $284 million in costs by
implementing DOD’s policy and cited one example in which the Army
and the Navy achieved savings of $80 million by jointly developing an
electro-optics test capability. Navy officials also told us that they believe
ATE planning for the Joint Strike Fighter, which calls for vendors to use
standardized test equipment or equipment having commercially available
components, can also be considered an accomplishment. In addition, the
Executive Agent established integrated process teams to research
technical issues dealing with tester commonality, such as efforts to
develop open systems architecture. In this regard, DOD provided funds


Page 14                  GAO-03-451 DOD Automated Test Equipment Modernization
              to the Executive Agent during fiscal years 1995 to 1998 for its research
              and development efforts. Currently, the Navy is leading a joint service
              technology project aimed at demonstrating that the most advance
              technologies can be combined into a single tester. The Executive Agent
              also implemented a process whereby ATE modernization and acquisitions
              would be reviewed for compliance with DOD policy, and developed the
              ATE Selection Process Guide and the ATE Master Plan to aid the services
              in complying with DOD’s ATE policies.

              ATE officials, responsible for oversight of ATE, noted that their role is
              essential; however, its current placement in one service (the Navy)
              makes it difficult to ensure other services comply with DOD guidance.
              A report recently prepared by a joint service working group9 noted
              continuing problems in the implementation of DOD policy, including
              ATE obsolescence, delays in modernization efforts, a lack of ATE
              interoperability among the services, upgrading difficulties, rising support
              costs, proliferation of equipment that is difficult to support, and systems
              that are not easily deployed.


              The services have made limited progress in achieving DOD’s commonality
Conclusions   goals for ATE, as established in the early 1990s. The department does not
              have a joint service forum or body that can oversee the total scope of
              ATE acquisition and modernization and better promote ATE commonality
              and the sharing of information and technology across platforms and
              services. DOD does not have sufficient information concerning the
              magnitude of the services’ modernization efforts or a departmentwide
              approach to accomplish ATE modernization in the most cost-effective
              manner. Without such an approach, the department faces a very expensive
              and time-consuming ATE modernization effort, with the continued
              proliferation of unique testers. It will also have no assurance that
              resources are allocated in the most effective manner to exploit
              commonality and commercially available technology and products. A
              single entity within DOD—rather than in one service—may be in the
              best position to provide overarching oversight and coordination between
              the services in planning for the modernization of ATE. We believe that
              high-level management commitment within DOD and all the services will



              9
               The DOD Executive Agent for ATE established this working group to develop a jointly
              funded demonstration project whereby the services would develop and share ATE
              innovative technologies for inclusion in future ATE acquisitions and modernizations.




              Page 15                     GAO-03-451 DOD Automated Test Equipment Modernization
                         be needed to achieve a cultural change that fosters the development of
                         common ATE.


                         We recommend that the Secretary of Defense reemphasize the policy that
Recommendations for      common ATE be developed to the maximum extent possible. We also
Executive Action         recommend that the Secretary reconsider whether placing its Executive
                         Agent for ATE in the Navy—or any single service—is the most effective
                         way to implement the policy. Wherever the Executive Agent is placed
                         organizationally, we recommend that the Secretary give it authority and
                         resources to

                     •   include representatives from all of the services, with a scope to include the
                         oversight of ATE acquisition and modifications for all weapon systems;
                     •   establish a mechanism to ensure that all ATE acquisitions and
                         modernizations are identified in an early enough stage to be able to
                         provide a comprehensive look at commonality and interoperability and to
                         ensure a coordinated effort between service entities;
                     •   direct the services to draw up modernization plans for its review so it can
                         identify opportunities to maximize commonality and technology sharing
                         between and within the services; and
                     •   continue efforts to research technical issues dealing with tester
                         commonality such as the development of open system architecture and
                         other joint service applications.


                         The Department of Defense provided written comments on a draft of this
Agency Comments          report, which are reprinted in their entirety in appendix II. The department
and Our Evaluation       also provided technical comments which we have incorporated, as
                         appropriate, into the report. DOD concurred with our recommendations
                         and agreed that it should reemphasize the policy that common automatic
                         test equipment be developed to the maximum extent possible. DOD
                         indicated that it would propose that an ATE acquisition policy statement
                         be included in the next issuance of DOD Instruction 5000.2, “Operation
                         of the Defense Acquisition System,” April 5, 2002. DOD also agreed to
                         reconsider whether the placement of its Executive Agent in the Navy—or
                         any single service—is the most effective way to implement its ATE policy.
                         The department further concurred that an Executive Agent for ATE
                         should have the authority and resources to direct the services to draw
                         up modernization plans for its review to maximize commonality,
                         interoperability, and technology sharing between the services. In this
                         regard, DOD agreed that there should be a mechanism to ensure all
                         automatic test equipment acquisitions and modernizations are identified in



                         Page 16                  GAO-03-451 DOD Automated Test Equipment Modernization
an early enough stage in order to have a coordinated effort among service
entities. Finally, DOD agreed that the Executive Agent for ATE should
include representatives from all services. DOD intends to use its authority
recently published in DOD Directive 5100.88, “DOD Executive Agent,”
September 3, 2002, to reconsider the placement of the Executive Agent
and to provide it with sufficient authority, resources, and mechanisms to
carry out its responsibilities. In addition, DOD intends to include the
funding for the Executive Agent as part of the Planning, Programming,
Budgeting and Execution process and to identify such funding separately
so that it is visible within the DOD budget.


As agreed with your office, unless you publicly announce its contents
earlier, we plan no further distribution of this report until 30 days from the
date of this report. At that time, we will send copies of this report to
interested congressional committees; the Secretaries of Defense, the Navy,
the Air Force, and the Army; the Commandant, U.S. Marine Corps; and the
Director, Office of Management and Budget. We will also make copies
available to other interested parties on request. In addition, the report will
be available at no charge on the GAO Web site at http://www.gao.gov/. If
you or your staff have any questions about the report, please contact me
at (757) 552-8100.

Key contributors to this assignment were Ken Knouse, William Meredith,
Harry Taylor, Hugh Brady, and Stefano Petrucci.

Sincerely yours,




Neal P. Curtin
Director, Defense Capabilities and Management




Page 17                  GAO-03-451 DOD Automated Test Equipment Modernization
             Appendix I: Scope and Methodology
Appendix I: Scope and Methodology


             We reviewed and analyzed available reports, briefings, documents, and
             records and interviewed officials at the Office of the Secretary of Defense
             and at Air Force and Navy headquarters organizations, Washington, D.C.;
             the Naval Air Systems Command located at Patuxent River, Maryland;
             Air Force Material Command and system program offices located at
             Wright-Patterson Air Force Base, Ohio; Warner Robins Air Logistics
             Center, Georgia; the North Island Naval Aviation Depot, California; the
             Navy’s Aviation Intermediate Maintenance Department, Oceana Master Jet
             Base, Virginia; and the intermediate maintenance department aboard an
             aircraft carrier based in San Diego, California. The Army was not included
             in the scope of this study because our focus was primarily on fixed-wing
             aircraft and because of the Army’s efforts to standardize its automatic test
             equipment (ATE) around a single family of testers, a situation similar to
             that of the Navy’s.

             To identify the problems that Air Force and Navy aviation (including
             Marine Corps) is facing with regard to ATE, we interviewed personnel
             responsible for policies and oversight, obtained applicable regulations and
             other guidance, and analyzed data provided by the services on various
             testers. We provided a proforma for the Air Force’s and Navy’s use in
             documenting their inventory of ATE, identifying obsolete testers, and
             providing estimates of modernization and replacement time frames and
             cost. The Navy’s data on ATE were provided by the central office that
             manages common test equipment—PMA-260, within the Naval Air Systems
             Command, and the Air Force’s Automatic Test System Division at Warner
             Robins Air Logistics Center. We also discussed obsolescence issues and
             ATE problems with the managers of shore-based, aircraft carrier, and
             depot maintenance activities. We reviewed and analyzed our prior reports
             and ongoing efforts, and reports of other organizations to provide a
             historical and contextual framework for evaluating ATE policies and
             issues, for documenting readiness rates of selected aircraft, and
             documenting the processes put in place by the Department of Defense
             (DOD) to oversee the services’ efforts to acquire and modernize ATE.

             To determine how successful DOD and the services have been in
             addressing the proliferation of unique testers, we held discussions with the
             responsible offices within each service and DOD, analyzed regulations and
             guidance, and reviewed studies and other documentation. We focused our
             work concerning this objective at the Navy office designated as DOD’s
             Executive Agent for Automatic Test Equipment—PMA-260 within the
             Naval Air Systems Command—and the Air Force’s Automatic Test System
             Division at Warner Robins Air Logistics Center. At these offices, which
             have responsibility for ATE acquisition or sustainment, modernization,


             Page 18                    GAO-03-451 DOD Automated Test Equipment Modernization
Appendix I: Scope and Methodology




and oversight, we held discussions with responsible officials, obtained
documentation regarding responsibilities and decisions, and reviewed files
for specific ATE acquisition and modernization programs. We also
obtained information from individual system program offices, for selected
aircraft, located at Wright-Patterson Air Force Base and selected Navy and
Air Force depots and intermediate maintenance facilities. Because we
found that Air Force testers are generally unique to specific aircraft, we
selected the F-15, B-1B, and B-2 for more detailed analysis, as these are
considered to be front-line aircraft depended upon heavily by the Air
Force to accomplish its mission. We also obtained information on ATE
acquisition for two fighter aircraft currently under development: the Joint
Strike Fighter and the F/A-22.

We performed our review from January 2002 through March 2003 in
accordance with generally accepted government auditing standards.




Page 19                    GAO-03-451 DOD Automated Test Equipment Modernization
                   Appendix II: Comments from the Department
Appendix II: Comments from the Department
                   of Defense



of Defense




         Page 20                 GAO-03-451 DOD Automated Test Equipment Modernization
          Appendix II: Comments from the Department
          of Defense




Page 21                 GAO-03-451 DOD Automated Test Equipment Modernization
          Appendix II: Comments from the Department
          of Defense




Page 22                 GAO-03-451 DOD Automated Test Equipment Modernization
             Related GAO Products
Related GAO Products


             Defense Inventory: Better Reporting on Spare Parts Spending Will
             Enhance Congressional Oversight. GAO-03-18. Washington, D.C.:
             October 24, 2002.

             Defense Inventory: Improved Industrial Base Assessments for Army
             War Reserve Spares Could Save Money. GAO-02-650. Washington, D.C.:
             July 12, 2002.

             Defense Inventory: Trends in Services’ Spare Parts Purchased from the
             Defense Logistics Agency. GAO-02-452. Washington, D.C.: April 30, 2002.

             Defense Logistics: Opportunities to Improve the Army’s and Navy’s
             Decision-Making Process for Weapons Systems Support. GAO-02-306.
             Washington, D.C.: February 28, 2002.

             Military Aircraft: Services Need Strategies to Reduce Cannibalizations.
             GAO-02-86. Washington, D.C.: November 21, 2001.

             Defense Logistics: Actions Needed to Overcome Capability Gaps in the
             Public Depot System. GAO-02-105. Washington, D.C.: October 12, 2001.

             Defense Logistics: Air Force Lacks Data to Assess Contractor Logistics
             Support Approaches. GAO-01-618. Washington, D.C.: September 7, 2001.

             Defense Inventory: Navy Spare Parts Quality Deficiency
             Reporting Program Needs Improvement. GAO-01-923. Washington, D.C.:
             August 16, 2001.

             Army Inventory: Parts Shortages Are Impacting Operations and
             Maintenance Effectiveness. GAO-01-772. Washington, D.C.: July 31, 2001.

             Navy Inventory: Parts Shortages Are Impacting Operations and
             Maintenance Effectiveness. GAO-01-771. Washington, D.C.: July 31, 2001.

             Air Force Inventory: Parts Shortages Are Impacting Operations and
             Maintenance Effectiveness. GAO-01-587. Washington, D.C.: June 27, 2001.

             Defense Inventory: Information on the Use of Spare Parts Funding Is
             Lacking. GAO-01-472. Washington, D.C.: June 11, 2001.

             Defense Inventory: Approach for Deciding Whether to Retain or
             Dispose of Items Needs Improvement. GAO-01-475. Washington, D.C.:
             May 25, 2001.


             Page 23                GAO-03-451 DOD Automated Test Equipment Modernization
           Related GAO Products




           Military Aircraft: Cannibalizations Adversely Affect Personnel and
           Maintenance. GAO-01-93T. Washington, D.C.: May 22, 2001.

           Defense Inventory: Army War Reserve Spare Parts Requirements Are
           Uncertain. GAO-01-425. Washington, D.C.: May 10, 2001.




(350137)   Page 24                GAO-03-451 DOD Automated Test Equipment Modernization
                         The General Accounting Office, the audit, evaluation and investigative arm of
GAO’s Mission            Congress, exists to support Congress in meeting its constitutional responsibilities
                         and to help improve the performance and accountability of the federal
                         government for the American people. GAO examines the use of public funds;
                         evaluates federal programs and policies; and provides analyses,
                         recommendations, and other assistance to help Congress make informed
                         oversight, policy, and funding decisions. GAO’s commitment to good government
                         is reflected in its core values of accountability, integrity, and reliability.


                         The fastest and easiest way to obtain copies of GAO documents at no cost is
Obtaining Copies of      through the Internet. GAO’s Web site (www.gao.gov) contains abstracts and full-
GAO Reports and          text files of current reports and testimony and an expanding archive of older
                         products. The Web site features a search engine to help you locate documents
Testimony                using key words and phrases. You can print these documents in their entirety,
                         including charts and other graphics.
                         Each day, GAO issues a list of newly released reports, testimony, and
                         correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site
                         daily. The list contains links to the full-text document files. To have GAO e-mail
                         this list to you every afternoon, go to www.gao.gov and select “Subscribe to daily
                         E-mail alert for newly released products” under the GAO Reports heading.


Order by Mail or Phone   The first copy of each printed report is free. Additional copies are $2 each. A
                         check or money order should be made out to the Superintendent of Documents.
                         GAO also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a
                         single address are discounted 25 percent. Orders should be sent to:
                         U.S. General Accounting Office
                         441 G Street NW, Room LM
                         Washington, D.C. 20548
                         To order by Phone:     Voice:    (202) 512-6000
                                                TDD:      (202) 512-2537
                                                Fax:      (202) 512-6061


                         Contact:
To Report Fraud,
                         Web site: www.gao.gov/fraudnet/fraudnet.htm
Waste, and Abuse in      E-mail: fraudnet@gao.gov
Federal Programs         Automated answering system: (800) 424-5454 or (202) 512-7470


                         Jeff Nelligan, managing director, NelliganJ@gao.gov (202) 512-4800
Public Affairs           U.S. General Accounting Office, 441 G Street NW, Room 7149
                         Washington, D.C. 20548