oversight

Depot Maintenance: Status of the Navy's Pearl Harbor Pilot Project

Published by the Government Accountability Office on 1999-09-10.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                   United States General Accounting Office

GAO                Report to the Chairman, Subcommittee
                   on Readiness and Management Support,
                   Committee on Armed Services, U.S.
                   Senate

September 1999
                   DEPOT
                   MAINTENANCE

                   Status of the Navy’s
                   Pearl Harbor Pilot
                   Project




GAO/NSIAD-99-199
Contents



Letter                                                                                          3


Appendixes             Appendix I   Scope and Methodology                                      26
                       Appendix II Description of the Pearl Harbor Pilot Test Plan and
                         Its Strengths and Weaknesses                                          29
                       Appendix III GAO Contacts and Staff Acknowledgments                     42


Related GAO Products                                                                           43


Tables                 Table 1: The Navy's Nine Test Metrics for Evaluating the Pearl Harbor
                         Pilot's Effectiveness                                                  7
                       Table 2: Preliminary Results, Strengths, and Weaknesses of the Pearl
                         Harbor Pilot Test Metrics                                             10
                       Table 3: Comparison of Ship Maintenance Activities in the Pearl
                         Harbor, Puget Sound, Portsmouth, and Norfolk Areas                    15
                       Table 4: Navy's Initial Two Metrics to Test Whether the Consolidated
                         Pearl Harbor NSY & IMF Increased Manpower Utilization and
                         Lowered Unit Cost of Production                                       29
                       Table 5: Metrics Three Through Six to Test Whether the Consolidated
                         Pearl Harbor NSY & IMF Increased Manpower Utilization and
                         Lowered Unit Cost of Production                                       31
                       Table 6: Metrics Seven Through Nine to Test Whether the Consolidated
                         Pearl Harbor NSY & IMF Increased Manpower Utilization and
                         Lowered Unit Cost of Production                                       32
                       Table 7: Baseline, Performance Expectations, and Preliminary
                         Results of the Test Plan                                              33


Figures                Figure 1: Breakdown of the Naval Shipyards' Fiscal Year 1997
                         Workloads by Major Customer                                           17
                       Figure 2: Comparison of Reported NWCF Revenues by Activity
                         Group for Fiscal Year 1998                                            20




                       Page 1                                   GAO/NSIAD-99-199 Depot Maintenance
Contents




Abbreviations

CNO         Chief of Naval Operations
CSMP        Current Ship Maintenance Program
DOD         Department of Defense
NAS         Naval Audit Service
NAVSEA      Naval Sea Systems Command
NIMF        Naval Intermediate Maintenance Facility
NSY & IMF   Naval Shipyard & Intermediate Maintenance Facility
NWCF        Navy Working Capital Fund
PACFLT      Pacific Fleet
PHNSY       Pearl Harbor Naval Shipyard



Page 2                                GAO/NSIAD-99-199 Depot Maintenance
United States General Accounting Office                                                   National Security and
Washington, D.C. 20548                                                             International Affairs Division



                                    B-280524                                                                          Leter




                                    September 10, 1999

                                    The Honorable James M. Inhofe
                                    Chairman, Subcommittee on Readiness
                                     and Management Support
                                    Committee on Armed Services
                                    United States Senate

                                    Dear Mr. Chairman:

                                    In recent years, the Navy has implemented many changes aimed at making
                                    its fleet support activities more efficient and effective. An underlying
                                    assumption was that consolidating similar activities within a region could
                                    eliminate the inefficiencies and redundancies. In 1998, as part of this effort,
                                    the Navy implemented a pilot project consolidating the management,
                                    operations, and funding of the Pearl Harbor Naval Shipyard and the Naval
                                    Intermediate Maintenance Facility in Hawaii. The Navy expects that the
                                    pilot project, commonly called the Pearl Harbor Pilot, will serve as a model
                                    for other potential consolidations by confirming that integrating shipyard
                                    and intermediate activities can result in greater efficiency and lower overall
                                    unit costs.

                                    As you requested, we examined the Navy's progress in implementing this
                                    pilot project. This report discusses (1) the preliminary results of the Pearl
                                    Harbor Pilot on improving performance of maintenance activities, (2) the
                                    usefulness of the pilot as a model for future consolidations, and (3) issues
                                    related to financial and organizational structures for such consolidations.
                                    Our scope and methodology are described in appendix I.



Results in Brief                    Although the Pearl Harbor Pilot is not complete, preliminary results have
                                    been mixed, showing either improvements, no improvements, or
                                    insufficient data to determine results. Where data are available, overall
                                    indications are that the pilot has the potential to improve maintenance
                                    activities in Hawaii. While the Navy's pilot test plan calls for evaluating
                                    performance using nine metrics, data has been gathered for only five to
                                    compare performance under the consolidated operation with performance
                                    under the preconsolidation organizations. Preliminary results for two
                                    metrics indicate improvements that meet or exceed the Navy's
                                    expectations, and two others indicate improvements that fall slightly short



                                    Page 3                                       GAO/NSIAD-99-199 Depot Maintenance
             B-280524




             of the Navy's expectations, while one metric indicates no improvement.
             Other positive results include increasing workforce flexibility by
             integrating nearly 4,000 workers from two work centers into a single
             workforce. Consequently, the number of workers assigned daily to the
             excess labor shop has dropped from about 200 to below 10. Furthermore,
             the Navy has reduced the maintenance infrastructure by 11 buildings
             (114,131 square feet) and plans to eliminate another 6 buildings (24,907
             square feet). At the same time, other anticipated efficiencies have yet to be
             realized. For example, not all the industrial plant equipment has been
             consolidated and the projected number of overhead workers has not been
             moved to direct maintenance positions to improve productivity.

             The Pearl Harbor Pilot is likely to serve only as a general model for future
             consolidation efforts because of unique aspects of ship maintenance
             activities in Hawaii, such as the close proximity of facilities and the large
             portion of fleet-funded work than in other locations. Likewise, because of
             this uniqueness, the pilot provides only a general indication that future
             consolidations elsewhere will result in similar efficiencies. Nevertheless, it
             does provide general information on such issues as combining workforces
             from separate activities and consolidating equipment and facilities.

             The Pearl Harbor Pilot has sharpened the debate over the most appropriate
             financial and organizational structures for such consolidated activities.
             Conflicting views continue to exist within the Departments of Defense and
             the Navy over whether these consolidated activities should operate under
             direct appropriations, the Navy Working Capital Fund, or a combination of
             the two financial structures, particularly in terms of maintaining visibility
             over total operational costs. Similarly, little progress has been made toward
             resolving the departments' differences over the most appropriate
             organizational structure to manage these types of activities.

             This report recommends that the Navy take steps to address unresolved
             issues related to financial and organizational structures as it proceeds with
             similar consolidations.



Background   In March 1994, the Chief of Naval Operations (CNO) announced a Regional
             Maintenance Program to streamline the Navy ship repair and maintenance
             processes, reduce infrastructure and costs, and maximize outputs. The
             program consisted of three phases: (1) optimizing intermediate-level
             maintenance through consolidation of intermediate activities,
             (2) integrating intermediate and depot activities with management by the



             Page 4                                      GAO/NSIAD-99-199 Depot Maintenance
                         B-280524




                         fleet commanders, and (3) conducting fleet maintenance using a single
                         process. The first phase, optimizing intermediate-level maintenance by
                         minimizing redundant intermediate ship maintenance capacity and
                         capability, is nearing completion. To ensure the validity of the second phase
                         of the program and to provide a model for other potential consolidations,
                         the Navy implemented the Pearl Harbor Pilot in 1998. Its completion date
                         has not been established. The third phase, performing a single maintenance
                         process by using common business and production processes, is under way
                         and scheduled to be completed in fiscal year 2001.


The Pearl Harbor Pilot   Prior to the consolidation, the Navy recognized that organizational and
                         funding structures inhibited the ability of the Pearl Harbor Naval Shipyard
                         (PHNSY) and Naval Intermediate Maintenance Facility (NIMF) to readily
                         share workloads and resources. The shipyard was managed by the Naval
                         Sea Systems Command (NAVSEA) and funded through the Navy Working
                         Capital Fund (NWCF),1 while the intermediate maintenance facility was
                         managed by the Pacific Fleet (PACFLT) and financed through direct
                         appropriations.2 During the pilot, PACFLT assumed ownership and overall
                         management and financial responsibility for the consolidated activity,
                         while NAVSEA continued to be the technical and operating authority. The
                         Navy named the combined activity the Pearl Harbor Naval Shipyard &
                         Intermediate Maintenance Facility (NSY & IMF). To achieve a fully
                         integrated organization, the Navy decided the Pearl Harbor Pilot should use
                         a single financial structure and selected direct appropriations rather than
                         the working capital fund as the funding approach. This latter decision was
                         based on several factors, but Navy officials believed that direct
                         appropriations were better suited to achieve the pilot goals and matched
                         the financial structure of the largest shipyard customer−PACFLT.


                         1
                           Under the working capital fund arrangement, activities such as a shipyard sell goods and
                         services to customers based on predetermined rates designed to recoup the full cost of
                         goods and services, including any military personnel costs. Customers pay for the goods and
                         services with their operation and maintenance funds. The fund's accounting practices and
                         related requirements for disclosure of information are intended to help managers and
                         workers focus on production costs and performance. However, the Department of Defense
                         (DOD) has long-standing problems in accumulating and reporting the full costs associated
                         with its working capital fund operations.
                         2
                          Direct appropriations are sometimes referred to as mission funding by DOD and Navy
                         officials. They are funds appropriated directly by the Congress that authorize agencies to
                         incur obligations for designated purposes. An agency may not spend more than the amount
                         appropriated.




                         Page 5                                             GAO/NSIAD-99-199 Depot Maintenance
                              B-280524




                              Furthermore, in recent years there has been increasing Navy
                              dissatisfaction with the costs associated with the working capital fund.
                              Given the nature of the fund and its emphasis on fully recovering costs of
                              the services provided, the perception sometimes exists that the fund
                              included fees and charges that inflated ship maintenance costs compared
                              to mission funding, which does not reflect the full cost of operations. In
                              accordance with one of the basic tenets of a working capital fund, the costs
                              associated with ship maintenance are more visible when financed using
                              this mechanism. While the level of resources required to carry out ship
                              maintenance activities is likely to be similar regardless of whether financed
                              using mission funding or through a working capital fund, under the
                              working capital fund the customer is more likely to be directly responsible
                              for a larger portion of those costs. Use of a working capital fund better
                              enables DOD components to fully account for their share of the program
                              costs. In commenting on a draft of this report, the Navy said that it chose
                              the Shipyard Management Information System as the cost accounting
                              system for the Pearl Harbor Pilot. According to the Navy, this system
                              (which is used by other naval shipyards under the NWCF) provides the
                              pilot the same degree of cost visibility as a working capital fund activity.

                              Full accounting for the costs of federal programs is required by federal
                              accounting standards. Specifically, Statement of Federal Financial
                              Accounting Standards No. 4 requires federal agencies to accumulate the
                              full cost of outputs through appropriate costing methodologies or cost
                              finding techniques (the full cost of an output is the sum of (1) the cost of
                              resources consumed that directly or indirectly contributes to the output
                              and (2) the cost of identifiable supporting services provided by other
                              entities).


After Expressing Concerns,    In 1998, as Navy officials were formulating plans for the Pearl Harbor Pilot,
Department of Defense         DOD officials expressed concern that the Navy's test plan and baseline data
                              would not adequately measure the pilot's impact on productivity and
Officials Approved the Test   performance. Consequently, the Navy revised the test plan and tried to
Plan in September 1998        minimize the data limitations in the baseline. Ultimately, as shown in table
                              1, the Navy selected nine test metrics to evaluate the pilot's effectiveness.




                              Page 6                                      GAO/NSIAD-99-199 Depot Maintenance
B-280524




Table 1: The Navy's Nine Test Metrics for Evaluating the Pearl Harbor Pilot's
Effectiveness

Metric
1. Total cost of a production shop, direct labor hour of work delivered to the customer
(indicator of the pilot's efficiency in terms of the cost per direct maintenance hour)
2. Total labor hours expended to deliver a production shop, direct labor hour to the
customer (indicator of the pilot's productivity in terms of personnel utilization)
3. Total Current Ship Maintenance Program work items completed (indicator of the pilot's
productivity in terms of the number of work items completed)
4. Total Current Ship Maintenance Program work items in the backlog (indicator of the
material conditions of PACFLT ships in terms of work items not yet completed)
5. Schedule adherence of CNO maintenance projects (indicator of customer satisfaction
in terms of completing projects on schedule)
6. Rework index for CNO maintenance projects (indicator of the pilot's quality of work in
terms of hours required to correct work deficiencies)
7. Activity work schedule integrity index (indicator of customer satisfaction in terms of
completing projects on schedule)
8. Casualty reports caused by activity work (indicator of the pilot's quality of work in terms
of casualty reports—ship reports indicating equipment failure)
9. Earned value (indicator of the pilot's productivity in terms of hours to complete similar
work items)


Source: The Pearl Harbor Pilot Test Plan, as amended, Department of the Navy (June 30, 1998).


In June 1998, the DOD Inspector General had recommended the Navy delay
the pilot and restart the data collection process, applying the same data
collection methodologies and controls for the baseline that would be used
to collect data during the pilot period. According to DOD Comptroller and
Inspector General officials, the most severe of the metrics' limitations
affected the baseline costs for the first two test metrics shown in table 1.
For example, military personnel costs were not traditionally collected by
intermediate maintenance facilities because these facilities were largely
funded out of operation and maintenance appropriations while military
personnel costs were already paid for separately out of military personnel
appropriations. Consequently, military personnel costs were also
accounted for separately. In addition, some labor hours appeared to be
inaccurate. One DOD contractor reported the NIMF database contained
zero labor hours for approximately 6 percent of the maintenance tasks
completed by the facility. Further, some Navy officials expressed concern
that some recorded labor hours appeared to reflect the expected work
standards and not actual production hours.




Page 7                                                 GAO/NSIAD-99-199 Depot Maintenance
                             B-280524




                             In response to the DOD Inspector General recommendation, Navy officials
                             said the pilot was too far along in the integration process in June 1998 to
                             reestablish separate maintenance activities without incurring significant
                             cost and operational turmoil. The expected cost and turmoil of such a
                             reversal, according to these officials, would outweigh any potential benefit
                             obtained from developing a better baseline. Consequently, the Navy revised
                             the test plan to minimize the data limitations. For example, as CNO and
                             NAVSEA officials requested, the Naval Audit Service (NAS) agreed to
                             validate the baseline and pilot results for six of the nine test metrics.3 In
                             several instances, baseline data were routinely collected by
                             preconsolidation activities, and they are available to provide a basis for
                             comparison with postconsolidation performance data. In addition, as
                             requested by CNO and NAVSEA officials, the NAS developed a
                             methodology to estimate baseline values for the first two metrics. The
                             methodology is based on the Navy's personnel wages and standards, as
                             well as, on the best available data from the former NIMF. Although the
                             resulting baseline estimates for the two metrics may not be exact, Navy
                             officials believe they are comparable to historical trends for intermediate
                             maintenance facilities and, consequently, provide a reasonable baseline on
                             which to indicate the pilot's impact on productivity and performance. On
                             September 14, 1998, the Principal Deputy Under Secretary of Defense
                             (Comptroller) approved the Navy's test plan. Appendix II provides more
                             detail concerning the evolution of the test plan and the nine metrics.


Congressional                The Conference Report for the DOD Appropriations Act for Fiscal Year
Requirements for the Pearl   19984 recognized that the Navy planned to initiate a pilot to study whether
                             combining fleet intermediate maintenance facilities with Navy shipyards
Harbor Pilot
                             might yield economies of scale and allow maintenance managers to better
                             balance workloads. The conferees concluded that it would take at least
                             2 years before the Navy could determine whether this new arrangement
                             was effective and should be made permanent or expanded to other
                             locations. Further, the report directed the Navy not to expand the pilot until
                             6 months after it reported to the Committees on Appropriations on its
                             findings, and that such report be made on or after April 1, 1999. The
                             conferees also directed the Navy not to make any permanent changes to


                             3
                               Previously, the NAS had agreed to collect and validate the baseline and pilot results for the
                             first two test metrics.
                             4
                                 House Report 105-265, September 23, 1997.




                             Page 8                                                GAO/NSIAD-99-199 Depot Maintenance
                              B-280524




                              the workforce in terms of total number of employees or any other
                              permanent changes until the pilot had been completed and evaluated by the
                              Congress. Currently, the Navy plans to issue an interim report on the pilot
                              shortly after October 1999.



While the Pilot Has           Preliminary results of the Pearl Harbor Pilot have been mixed, showing
                              either improvements, no improvements, or insufficient data to determine
Potential to Improve          results. For most of the measures where data are available, indications are
Maintenance Activities,       that the pilot has the potential to improve maintenance activities in Hawaii.
                              While the test plan calls for using nine metrics, the data has been gathered
Preliminary Data Show         for only five of them. For these five metrics, preliminary results for two
Mixed Results                 metrics indicate improvements that meet or exceed the Navy's
                              expectations; two others indicate improvements that fall slightly short of
                              the Navy's expectations; and one metric indicates no improvement. Other
                              positive results include the fact that the Navy has integrated the
                              workforces from two work centers into a common pool, increased
                              management flexibility in assigning workers to maintenance projects based
                              on evolving priorities, and reduced the maintenance infrastructure at Pearl
                              Harbor. At the same time, other planned efficiencies are yet to be realized.
                              For example, the pilot has not resulted in the number of overhead workers
                              projected to move to direct maintenance positions or in the consolidation
                              of all industrial plant equipment.5 According to Navy officials, these actions
                              would have increased the number of people working directly on
                              maintenance projects and the utilization of existing plant equipment.


The Pilot Has the Potential   The nine metrics chosen for the Pearl Harbor Pilot Test Plan represent a
to Improve Maintenance        variety of issues and performance indicators. Although data available for
                              some of the metrics are limited, it is sufficient to provide a general, if
Activities in Hawaii          imprecise, indication of the pilot's impact on productivity and
                              performance. In this case, as indicated in table 2, while test results have not
                              yet been fully developed, the preliminary results have been mixed, showing
                              either improvements, no improvements, or insufficient data to determine



                              5
                               In general, overhead workers are those employees who are not working directly on
                              maintenance projects and are assigned to administrative and support positions such as
                              personnel and financial departments. Direct maintenance workers are those employees who
                              accomplish work directly identifiable to a maintenance project such as repair, maintenance,
                              and installation of equipment and components.




                              Page 9                                             GAO/NSIAD-99-199 Depot Maintenance
                                                    B-280524




                                                    results. Where data are available, overall indications are that the pilot has
                                                    the potential to improve maintenance activities in Hawaii.



Table 2: Preliminary Results, Strengths, and Weaknesses of the Pearl Harbor Pilot Test Metrics

Metric (number) and
anticipated goal                         Preliminary results                  Strengths (S) and weaknesses (W)
Improvement indicated
Total cost of a production shop,         Indicates improvement because        Variant of a traditional indicator of shipyard efficiency (S)
direct labor hour of work delivered      preliminary data show that it has    Shipyard baseline data are considered reliable (S)
to the customer (1)                      cost slightly less to deliver one    NAS developed baseline estimates and plans to validate and
Indicator of the pilot's efficiency in   direct maintenance hour in fiscal    generate the test data (S)
terms of the cost per direct             year 1998, but falls short of the    NAS developed baseline estimates of the NIMF military
maintenance hour                         Navy's expectation                   personnel costs because the facility did not maintain this type of
                                                                              data (W)
                                                                              NIMF baseline data on production hours have limitations, but
                                                                              are considered reliable enough by Navy officials to provide a
                                                                              rough, if imprecise, estimate of the baseline (W)
Total labor hours expended to            Indicates improvement because        Variant of a traditional indicator of shipyard productivity (S)
deliver a production shop, direct        preliminary data show that it has    Shipyard baseline data are considered reliable (S)
labor hour to the customer (2)           taken fewer hours to deliver one     NAS developed baseline estimates and plans to validate and
Indicator of the pilot's productivity    direct maintenance hour in fiscal    generate the test data (S)
in terms of personnel utilization        year 1998                            NIMF baseline data on production hours have limitations but are
                                                                              considered reliable enough by Navy officials to provide a rough,
                                                                              if imprecise, estimate of the baseline (W)
Total Current Ship Maintenance           Indicates improvement because        Traditionally considered an indicator of the material condition of
Program work items in the                preliminary data show that there     ships (S)
backlog (4)                              are fewer backlogged work items      Fleet and ship commanders maintain baseline and test data,
Indicator of the material                since the consolidation, but falls   which are considered reliable (S)
conditions of PACFLT ships in            short of the Navy's expectation      NAS plans to validate the baseline and test data (S)
terms of work items not yet                                                   Is an indicator of the NIMF and the combined facility's
completed                                                                     productivity but not the former shipyard (W)
Schedule adherence of CNO ship           Indicates improvement because        Traditionally considered an indicator of the shipyard customers'
maintenance projects (5)                 preliminary data show that more      satisfaction (S)
Indicator of customer satisfaction       CNO projects are being completed     NAVSEA maintains baseline and test data, which are
in terms of completing projects on       on time or ahead of schedule since   considered reliable (S)
schedule                                 the consolidation                    NAS plans to validate the baseline and test data (S)
                                                                              Is an indicator of shipyard and the combined facility's
                                                                              performance but not the former NIMF (W)
No improvement indicated
Total Current Ship Maintenance           Indicates no improvement because     Fleet and ship commanders maintain baseline and test data,
Program work items completed             preliminary data show that there     which are considered reliable (S)
(3)                                      are fewer work items being           NAS plans to validate the baseline and test data (S)
Indicator of the pilot's productivity    completed since the consolidation    Is an indicator of the NIMF and the combined facility's
in terms of the number of work                                                productivity, but not the former shipyard (W)
items completed                                                               Is affected by other factors not related to the pilot (W)
                                                                                                                                      (continued)




                                                    Page 10                                             GAO/NSIAD-99-199 Depot Maintenance
                                                   B-280524




Metric (number) and
anticipated goal                        Preliminary results                     Strengths (S) and weaknesses (W)
Not yet determined
Rework index for CNO                Not yet determined because the              Traditionally considered an indicator of the shipyard quality of
maintenance projects (6)            Navy requires additional time to            work (S)
Indicator of the pilot's quality of develop the test data                       NAS plans to validate the baseline and test data (S)
work in terms of hours required to                                              Is an indicator of shipyard and the combined facility's
correct work deficiencies                                                       performance, but not the former NIMF (W)
Activity work schedule integrity        Not yet determined because the          Traditionally considered an indicator of the shipyard customers'
index (7)                               Navy does not plan to gather data       satisfaction (S)
Indicator of customer satisfaction      needed to develop this metric           Shipyard maintains baseline and test data, which are
in terms of completing projects on      unless the results of the first six     considered reliable (S)
schedule                                metrics are inconclusive                Is an indicator of shipyard and the combined facility's
                                                                                performance, but not the former NIMF (W)
                                                                                Is similar to the fifth metric measuring schedules adherence
                                                                                and, as a result, Navy officials question the usefulness of
                                                                                developing this metric (W)
Casualty reports caused by              Not yet determined because the          Fleet and ship commanders maintain baseline and test data,
activity work (8)                       Navy does not plan to gather data       which are considered reliable (S)
Indicator of the pilot's quality of     needed to develop this metric           Is an indicator of shipyard and the combined facility's
work in terms of casualty reports       unless the results of the first six     performance, but not the former NIMF (W)
                                        metrics are inconclusive                Is similar to the sixth metric measuring rework and, as a result,
                                                                                Navy officials question the usefulness of developing this metric
                                                                                (W)
Earned value (9)                        Not yet determined because the          Attempts to measure the pilot's effect on maintenance outputs
Indicator of the pilot's productivity   Navy does not plan to gather data       (S)
in terms of hours to complete           needed to develop this metric           Should require significant resources to develop results (W)
similar work items                      unless the results of the first six     Is based on a limited universe of work items (W)
                                        metrics are inconclusive


                                                   Note(s): Results as of June 22, 1999.

                                                   Appendix II provides more detail concerning the strengths and weaknesses of the nine metrics.

                                                   Source: Interviews with Navy officials and our analysis of the Pearl HarborPilot Test Plan dated
                                                   June 30, 1998; NAS capacity evaluation reports for fiscal years 1997 and 1998; and pilot status
                                                   reports.


                                                   As shown in table 2, two test metrics indicate improvements that meet or
                                                   exceed the Navy's expectations and two indicate improvements that fall
                                                   slightly short of the Navy's expectations; only one metric indicates no
                                                   improvement. Because of the operational turbulence experienced during
                                                   the consolidation, we believe that it is reasonable to expect that some
                                                   productivity and performance indicators might show diminished
                                                   improvement during the transition year. In addition, as indicated in the
                                                   table, the Navy has not gathered performance data for the remaining four
                                                   metrics. The Navy is in the process of collecting test data on the rework
                                                   index for CNO maintenance projects. However, Navy officials do not plan



                                                   Page 11                                                   GAO/NSIAD-99-199 Depot Maintenance
                                B-280524




                                to gather data needed to develop the last three metrics unless the test
                                results are inconclusive, even though the three metrics were added to the
                                test plan at the request of the Principle Deputy Under Secretary of Defense
                                (Comptroller).


Other Efficiencies Identified   In August 1997, the Pearl Harbor Regional Maintenance Pilot Executive
During the Planning Process     Steering Committee issued the Pilot Study Report for the integration of the
                                PHNSY and NIMF.6 The purpose of the report was to outline the approach
Have Been Achieved              for this integration. Among other improvements, the report envisioned
                                increasing productivity and cost-effectiveness by integrating the workers of
                                former PHNSY and NIMF into a common workforce and reducing the ship
                                maintenance infrastructure in Hawaii. As planned in the Pilot Study Report,
                                the Navy has integrated nearly 4,000 workers from two separate work
                                centers into a common pool, increased management flexibility in assigning
                                workers to maintenance projects, and reduced the ship maintenance
                                infrastructure.

                                Prior to the pilot, it was difficult to shift work or personnel between
                                maintenance activities due to multiple financial structures. More
                                specifically, the Navy stated that NWCF rules prohibited assigning workers
                                to a job without a funding document, which made it difficult to shift work
                                between maintenance activities. Accordingly, when administrative and
                                financial requirements restricted the movement of workers, shipyard
                                production personnel not working on a specific maintenance project were
                                sent to the excess labor shop to wait assignment. During this time, they
                                performed indirect work such as facility maintenance or grounds-keeping.7
                                Before the pilot, the number of workers assigned daily to this shop ranged
                                between 100 to 200. By integrating the workforce under a single financial
                                and command structure, production shops now have more flexibility in
                                assigning excess workers to other projects, including projects historically
                                completed by the former intermediate maintenance facility. Consequently,
                                the number of workers assigned daily to the excess labor shop has dropped
                                to below 10.



                                6
                                 Pilot Study Report for an Integration of Pearl Harbor Naval Shipyard and Naval
                                Intermediate Maintenance Facility, Pearl Harbor, Pearl Harbor Pilot Executive Steering
                                Committee (Aug. 26, 1997).
                                7
                                 Limited-duty personnel with medical injuries or health problems and personnel whose
                                security clearance levels were under review were also assigned to the excess labor shop.




                                Page 12                                            GAO/NSIAD-99-199 Depot Maintenance
                             B-280524




                             In addition, as a result of the consolidation, the Navy has reduced the
                             maintenance infrastructure at Pearl Harbor. For example, 11 (114,131
                             square feet) of the 27 buildings previously used by the former intermediate
                             maintenance facility were vacated and turned over to Pearl Harbor Naval
                             Station.8 While the consolidated facility will retain 7 of the remaining
                             16 NIMF buildings, it plans to vacate another 6 (24,907 square feet) and is
                             reviewing the disposition of 3 buildings.9 As of June 1999, the Pearl Harbor
                             Naval Station demolished or plans to demolish eight former NIMF buildings
                             and was able to demolish two of its buildings after tenants moved into
                             vacated NIMF buildings. The demolition costs for all 10 buildings is
                             estimated at $1.9 million with a projected annual cost avoidance of
                             $307,000 or a payback period of a little over 6 years.



Some Efficiencies Have Yet   The Pilot Study Report envisioned increasing productivity and
to Be Fully Achieved         cost-effectiveness by moving overhead workers to direct maintenance
                             positions and consolidating industrial plant equipment. However, these
                             efficiencies have not yet been fully achieved.

                             For example, the consolidation did not result in the projected number of
                             overhead workers moving to direct maintenance work−an important
                             element in increasing the combined facility's productivity. To increase
                             productivity, one of the stated goals of the pilot was to increase the number
                             of direct production workers relative to the number of supervisors and
                             overhead personnel without increasing costs. Logically, increasing the
                             number of production workers should result in increased production. To
                             accomplish this goal, the Pilot Study Report proposed that 95 civilian
                             overhead workers be moved to positions in direct maintenance work.
                             However, only four workers moved. According to Pearl Harbor NSY & IMF
                             officials, while they have tried, it has been impracticable for them to move
                             civilians to such positions because of (1) the time required for overhead
                             workers to become skilled, usually several years; (2) regulations restricting
                             the process for downgrades; and (3) potentially negative action by workers




                             8
                              In general, the Pearl Harbor Naval Station has ownership and is responsible for the
                             management and maintenance of Navy buildings in Hawaii.
                             9
                              According to Navy officials, no former shipyard buildings will be vacated as a result of the
                             Pearl Harbor Pilot.




                             Page 13                                             GAO/NSIAD-99-199 Depot Maintenance
                              B-280524




                              and employee representatives.10 In addition, several department directors
                              and overhead supervisors said they were unwilling to release any personnel
                              because of the increased workload due to changes in administrative and
                              financial systems since the consolidation. Furthermore, other Navy
                              officials said that efforts to move supervisors to direct production resulted
                              in too few supervisors, which led to problems in planning and coordinating
                              work on maintenance projects. According to CNO and NAVSEA officials,
                              more overhead workers will be moved to direct maintenance positions as
                              the pilot progresses.

                              In addition, industrial plant equipment has not been completely
                              consolidated at integrated production shops. The Pilot Study Report
                              suggested the in-shop industrial equipment be efficiently consolidated and
                              redistributed to improve maintenance operations. Of the 271 items of
                              industrial equipment at the former intermediate maintenance facility,
                              114 items were relocated, 132 items were mothballed, and 25 items were
                              kept operational at their original location. According to officials formerly
                              associated with the Pearl Harbor NSY & IMF, the 114 related items were
                              equipment that required little or no removal and installation costs.
                              Although many of the 132 mothballed items are in better condition and
                              newer than the same type of equipment located currently in the
                              consolidated shops, the funding required to relocate the NIMF equipment
                              has not been available. Most mothballed items are semipermanently
                              attached heavy equipment that requires funding to remove, transport, and
                              install elsewhere. According to Pearl Harbor officials, congressional
                              guidance directing the Navy not to make any permanent changes until after
                              the pilot period has to a lesser degree influenced the decision not to
                              relocate the mothballed items. They plan to request funding to move
                              equipment after the pilot period is completed.


The Pearl Harbor Pilot Will   The Pearl Harbor Pilot is likely to serve only as a general model for future
Serve Only as a General       consolidation efforts because of the unique aspects of ship maintenance
                              activities in Hawaii, such as the close proximity of facilities and the
Model for Similar
                              significant portion of fleet-funded work. Likewise, because of this
Consolidations


                              10
                                Our prior work examining DOD reform initiatives has noted the importance of top
                              management commitment and sustained support for reform initiatives, as well as
                              overcoming culture barriers and resistance to change. See Defense Reform Initiatives:
                              Organization, Status, and Challenges (GAO/NSIAD-99-87, Apr. 21, 1999).




                              Page 14                                            GAO/NSIAD-99-199 Depot Maintenance
                                               B-280524




                                               uniqueness, the pilot provides only a general indication that future
                                               consolidations will result in similar efficiencies.

                                               While there can be clear benefits to regionalization, available statistical
                                               data suggest conditions may have been more favorable to such efforts in
                                               Pearl Harbor than may exist at other Navy locations. For example, during
                                               fiscal years 1996 through 1998, the Navy reportedly replaced 698 military
                                               personnel at the former NIMF with 504 experienced civilian workers,
                                               mostly from the shipyard. Although civilian workers were periodically
                                               transferred between intermediate facilities and shipyards at other
                                               locations, Pearl Harbor is the only location where a significant number of
                                               workers has recently been transferred. Consequently, according to Navy
                                               officials, this large cadre of civilian workers with recent shipyard
                                               experience should have expedited the integration of the workforce by
                                               requiring less training and fewer trade skill certifications. This condition
                                               does not exist to such a large extent at other potential consolidation
                                               locations, except at the Puget Sound area. Table 3 compares some
                                               additional differences between Pearl Harbor and other Navy locations.



Table 3: Comparison of Ship Maintenance Activities in the Pearl Harbor, Puget Sound, Portsmouth, and Norfolk Areas

Dollars in millions
Category                              Pearl Harbor             Puget Sound                         Portsmouth              Norfolk
Time required to travel between the • 5 to 10 minutes          • 20 to 30 minutes to               • 2 to 3 hours         •   30 to 45 minutes to
intermediate maintenance facilities                              the Trident Refit Facility          to Groton,               the Shore
and the shipyards                                              • 2 to 4 hours to Everett,            Connecticut,             Intermediate
                                                                 Washington, maintenance             maintenance facility     Maintenance
                                                                 facility                                                     Activity
Reported shipyard revenues            $379.4                   $1,041.8                            $394.1                  $822.1
Number of military personnel
• Shipyard                            43                       47                                  53                      64
• Intermediate maintenance facility   641                      1,105                               1,134                   2,516
  or activity
Number of civilian personnel
• Shipyard                            2,742                    8,823                               3,286                   7,127
• Intermediate maintenance facility   554                      967                                 149                     23
  or activity


                                               Note: Fiscal year 1998.

                                               Source: Our analysis of interviews and documents obtained from Navy officials.




                                               Page 15                                                 GAO/NSIAD-99-199 Depot Maintenance
B-280524




As indicated in table 3, Pearl Harbor facilities are located closer to each
other than in other Navy locations. Consequently, consolidation of
activities separated by greater distances at other Navy locations could be
more difficult to manage and, after the consolidation is completed, could
possibly reduce the combined facility's flexibility in the management of the
workforce and workload. For example, given the close proximity of
facilities in Hawaii, workers with multiple and specialized skills from both
the former PHNSY and NIMF can move between ship maintenance projects
and emergent work in a few minutes compared to other Navy locations
where transportation time could range up to 4 hours. In addition, the
consolidated production shops in Hawaii are able within a few minutes to
pick up and deliver replacement parts for maintenance projects compared
to other locations, where delivery time could range up to 4 hours once the
shops are consolidated. Consequently, the longer transportation time
between facilities at other Navy locations could extend the time frames for
maintenance projects compared to projects completed in Hawaii. Further,
as indicated in table 3, Pearl Harbor facilities differ in terms of revenues
and staffing than in some of the other areas. For example, except for the
Portsmouth Naval Shipyard, reported revenues and staffing for the other
two shipyards were more than double the amount of the PHNSY. The
smaller mission at Pearl Harbor should have made the consolidation easier.

In addition during fiscal year 1997, as indicated in figure 1, more of the
PHNSY workload was for the fleet compared with the other three naval
shipyards that completed more work for non-fleet activities. According to
Navy officials, a single major customer should simplify the integration and
management of ship maintenance activities. They indicated that it is much
easier to schedule and budget for maintenance projects on behalf of one
primary and a few secondary customers compared to meeting the varying
requirements of several different customers with equal workloads.




Page 16                                    GAO/NSIAD-99-199 Depot Maintenance
                                          B-280524




Figure 1: Breakdown of the Naval Shipyards' Fiscal Year 1997 Workloads by Major Customer
60 Percentage of total




50




40




30




20




10




 0


        Norfolk Naval Shipyard           Puget Sound                         Portsmouth Naval                Pearl Harbor Naval
                                         Naval Shipyard                          Shipyard                         Shipyard

                                                 Pacific Fleet

                                                 Atlantic Fleet

                                                 Naval Sea Systems Command

                                                 Other activities


                                          Source: Our analysis of NAVSEA data.


                                          In the future, Navy officials estimated that nearly 90 percent of the Pearl
                                          Harbor NSY & IMF workload would be for PACFLT. The remaining 10
                                          percent will be completed under a reimbursable basis for other activities.
                                          According to Navy officials, the large share of PACFLT workload in Hawaii
                                          should reduce the need for the combined maintenance activity to broker
                                          large workloads between different Navy activities. At the same time, having
                                          one common financial system should eliminate the need for potentially
                                          unwieldy financial processes and procedures to manage and budget for
                                          different work and funding sources. In other locations, NAVSEA and the




                                          Page 17                                               GAO/NSIAD-99-199 Depot Maintenance
                           B-280524




                           respective fleet are estimated to share the ship maintenance workload on a
                           more equitable basis.



Issues Related to          The Pearl Harbor Pilot has sharpened the debate over the most appropriate
                           financial and organizational structures for such consolidated activities.
Financial and              Conflicting views continue to exist within the Departments of Defense and
Organizational             the Navy over whether these consolidated activities should operate under
                           direct appropriations, the NWCF, or a combination of the two financial
Structures Need to Be      structures, particularly in terms of maintaining visibility over total
Resolved                   operational costs. Similarly, little progress has been made toward resolving
                           the departments' differences over the most appropriate organizational
                           structure to manage these types of activities. Consequently, we believe
                           these issues need to be resolved to facilitate effective and efficient
                           consolidations.


Issues Remain Unresolved   The Navy's proposal recommending direct appropriations for the Pearl
Regarding Financial        Harbor Pilot stated that the pilot provided an actual demonstration of using
                           direct appropriations for a “major maintenance activity,” which would help
Structure
                           to determine the ultimate financial structure for the Navy's regionalization
                           actions. The proposal further stated that a common financial structure
                           needed to be tested by means of the pilot. However, the Navy's test plan
                           does not attempt to measure the impact of using a single financial structure
                           or the benefit of using direct appropriations instead of the working capital
                           fund. At the same time, DOD and Navy officials continue to have different
                           views on the potential impact of using direct appropriations for the pilot on
                           the productivity and performance of the Pearl Harbor Pilot and future
                           regionalization actions. The same is true about the command structure to
                           be used for such consolidated operations. Consequently, we believe these
                           issues need to be resolved to facilitate effective and efficient
                           consolidations.

Navy Selected Direct       Prior to the consolidation, the Navy recognized that the differences
Appropriations             between activity management and financial structure inhibited the
                           PHNSY's and NIMF's ability to readily share workloads and resources.
                           According to Navy officials, the multi-financial and command structures
                           required them to use cumbersome, workaround procedures to share
                           workloads and workers between the former PHNSY and NIMF. Therefore,
                           the shipyard and intermediate maintenance facility did not readily share
                           workloads and resources because of the time and cost required to carry-out



                           Page 18                                    GAO/NSIAD-99-199 Depot Maintenance
                                   B-280524




                                   the process. To achieve a fully integrated organization, the Navy decided
                                   the Pearl Harbor Pilot should use a single financial structure and selected
                                   direct appropriations instead of the NWCF. This decision was based on
                                   several factors, but primarily because PACFLT was the larger customer of
                                   the two organizations and direct appropriations matched the fleet's
                                   financial structure. Consequently, Navy officials expected fewer financial
                                   issues under direct appropriations than NWCF because PACFLT could
                                   integrate the pilot into its existing financial structure without establishing
                                   another system.

Views on the Financial Structure   DOD and Navy officials have different views regarding the financial impact
Differ                             of removing the PHNSY from the NWCF. There are several centrally
                                   managed functions in support of all shipyards for which the costs are
                                   recovered in the shipyard labor rates under the working capital fund
                                   structure, which are not normally captured under direct appropriations.
                                   According to DOD officials, these functions include NAVSEA personnel
                                   directly supporting shipyards, the Navy's Crane Center that manages all
                                   crane procurements and repair projects, centralized automated
                                   information systems, programming support, and other centrally managed
                                   items. In effect, removing the PHNSY from the fund without addressing the
                                   contribution (approximately 14 percent) it made to cover centrally funded
                                   costs for naval shipyards would increase their rates. However, CNO and
                                   NAVSEA officials believe the PHNSY's share of the overhead cost was not
                                   large enough to have a significant effect on the other shipyards' rates.

                                   As shown in figure 2, the former PHNSY generated 1.7 percent of the fund's
                                   $21.9 billion in total revenues during fiscal year 1998 and the other
                                   shipyards contributed another 10.6 percent, or $2.3 billion.




                                   Page 19                                      GAO/NSIAD-99-199 Depot Maintenance
B-280524




Figure 2: Comparison of Reported NWCF Revenues by Activity Group for Fiscal
Year 1998

                       1.7%
                                                         Pearl Harbor Naval Shipyard
                            7.8%
                                                      Other maintenance
                                                         Base support
           31.7%                  8.7%

                                 10.1%                   Other



                                    10.6%
                                                      Other shipyards
                   29.2%
                                                      Supply management

                                                      Research and development




Note: The “other maintenance” category includes aviation and U.S. Marine Corps depot maintenance
activities and “other” includes ordnance, information services, and transportation activities.

Percentages do not add to 100 percent due to rounding.

Source: Our analysis of the DOD Inspector General report entitled Inspector General, DOD, Oversight
of the Naval Audit Service Audit of the Navy Working Capital Fund Financial Statements for FY 1998
(99-099, Mar. 1, 1999).


DOD officials are concerned that the Navy has not adequately addressed a
potentially more significant issue regarding the impact of removing all
naval shipyards from the NWCF. They believe the Navy decision to remove
the shipyards from the fund would cause other Navy activities remaining in
the fund to absorb a larger share of overhead costs. However, CNO and
NAVSEA officials believe that the shipyards' share is not large enough to
have a significant effect on the activities remaining in the fund.

Further, DOD and Navy officials have different views on other financial
issues:

• Working capital funds are not directly subject to the annual
  appropriation cycle and can continue operations without interruption
  between fiscal years. Consequently, the NWCF provided former
  shipyard operators with significant financial flexibility because they
  were freed from reprogramming limitations and restrictions applicable



Page 20                                                   GAO/NSIAD-99-199 Depot Maintenance
B-280524




  to regular appropriations and allowed to incur costs without waiting for
  enactment of an appropriation. Because this financial flexibility is
  considered critical to shipyard operations, DOD officials are concerned
  that the Navy has not addressed the impact of eliminating this flexibility
  on the pilot operations or future consolidations.
• The Congress established working capital funds, among other purposes,
  to provide a more flexible funding mechanism that would allow DOD
  industrial, commercial, and service activities to operate on the same
  basis as the private sector, including the use of standard cost accounting
  practices and widely-accepted management techniques. According to
  DOD officials, applying these practices and techniques to naval
  shipyards has provided cost visibility and improved the efficiency of
  operations. Although we found that the Navy has retained the cost
  visibility of the former shipyard under the pilot, the cost information is
  not routinely reported to the major commands as it was for the NWCF.
  Further, the visibility is maintained by continuing to operate the former
  shipyard accounting and management system, which could be
  eliminated after the pilot is completed. For example, DOD officials are
  concerned that the visibility will not be maintained after the pilot period
  is completed. Navy officials said they plan to maintain cost visibility of
  the maintenance operations in Hawaii even after the pilot period
  because the visibility is useful in the management of those operations.
  Whichever financing method is adopted after the completion of the
  pilot, the Statement of Federal Financial Accounting Standards No. 4
  requires DOD and the Navy to continue to measure and report the full
  cost of such ship maintenance operations by responsibility segment.
• The buyer/seller relationship is a fundamental aspect of working capital
  funds. According to DOD officials, this relationship established dynamic
  interplay between the buyer and the provider that encouraged cost
  control and responsiveness to customer requirements. In addition, it
  provided the opportunity for shipyard customers to focus their attention
  on accomplishing their respective missions and shipyard operators to be
  accountable and responsive to customers' needs. Absent
  reimbursements for most maintenance transactions at Pearl Harbor
  under direct appropriations, CNO and NAVSEA officials said that a
  buyer/seller relationship has largely been eliminated. They estimated
  that only 10 percent of the combined facility's future workload, such as
  ship alterations and inactivations, will continue to be on a reimbursable
  basis after the pilot period.
• According to DOD officials, another important aspect of working capital
  funds is their capital investment program. In the case of the NWCF, the
  former shipyard depreciated its capital assets and collected this



Page 21                                    GAO/NSIAD-99-199 Depot Maintenance
                                 B-280524




                                      expense from customers through its rates. Therefore, the fund had a
                                      ready reserve to finance capital investments for the shipyard. According
                                      to CNO and NAVSEA officials, NAVSEA will budget for future capital
                                      improvements for the Pearl Harbor NSY & IMF. However, DOD officials
                                      are concerned that future funding levels may be insufficient because of
                                      uncertainties in the DOD budgeting and appropriation process.

Discussions of Other Financial   Although other similar regionalization efforts are likely, according to Navy
Structures Continue              officials, they may not look like the Pearl Harbor Pilot with direct
                                 appropriations as the single financial structure. For example, Navy officials
                                 are currently discussing using either direct appropriations or the working
                                 capital fund to finance the combined activity of future consolidations, or a
                                 combination of the two financial structures. 11 One combination being
                                 discussed involves using direct appropriations to finance the overhead and
                                 military personnel of the combined facility while using the NWCF to
                                 finance direct maintenance work and materials. Because they believe that
                                 the working capital fund is more costly and less flexible than direct
                                 appropriations, several Navy officials are reluctant to use the fund to
                                 finance the cost of intermediate maintenance activities, which are
                                 currently financed with direct appropriations.


Differing Views Remain           The Navy has made little progress toward resolving questions over the
Regarding Organizational         appropriate organizational structure for such consolidations. While the
                                 proposal recommending the pilot's financial structure stated that common
Structure
                                 ownership was believed to be beneficial to the Navy's regional
                                 maintenance initiative, it needed to be tested by the pilot. However, the
                                 Pearl Harbor Pilot Test Plan does not attempt to measure the impact of
                                 providing PACFLT ownership and management responsibilities for the pilot
                                 and Navy officials are uncertain which command structure should be used
                                 for similar consolidations at other locations.

                                 Although the pilot was expected to streamline the maintenance process
                                 and eliminate stovepipe organizations, DOD and Navy officials do not
                                 know the extent to which the pilot's current structure is meeting these
                                 objectives or how they might have been met under other command
                                 structures. Prior to the pilot, the NIMF operated as a PACFLT activity while
                                 the former shipyard operated as a NAVSEA activity. Under the pilot,

                                 11
                                  In its comments on a draft of this report, the Navy said that NAVSEA and the Fleet
                                 Commanders had agreed now to operate future consolidations under direct appropriations.




                                 Page 22                                          GAO/NSIAD-99-199 Depot Maintenance
                  B-280524




                  PACFLT assumed ownership and overall management and financial
                  responsibility for the consolidated activity while NAVSEA continued to be
                  the technical and operating authority. Although CNO and NAVSEA officials
                  viewed the duality of fleet ownership and NAVSEA operation as not adding
                  another management layer, according to several DOD and other Navy
                  officials, this appeared to add a level of responsibility and oversight in lieu
                  of streamlining the former maintenance structure and eliminating
                  organizations. For example, several Pearl Harbor NSY & IMF officials said
                  that the current ownership and command structure has resulted in the
                  practice of reporting and coordinating simultaneously with both
                  commands. In addition, there has been some confusion on which command
                  was ultimately responsible for collecting test data and analyzing the results
                  of the pilot. Navy officials are still undecided about the appropriate
                  organizational structure for these types of activities and are considering
                  different structures for other regionalization efforts.



Conclusions       Although the Pearl Harbor Pilot is not complete, preliminary results have
                  been mixed, showing either performance improvements, no improvements,
                  or insufficient data to determine results. For most of the measures where
                  data are available, indications are that the pilot has the potential to improve
                  maintenance activities in Hawaii. However, because of unique aspects of
                  ship maintenance activities in Hawaii, the pilot is likely to serve only as a
                  general model for future consolidation efforts and provides only a general
                  indication that future consolidations will result in similar efficiencies. At
                  the same time, DOD and Navy officials continue to have different views on
                  the potential impact of using direct appropriations to finance the pilot on
                  (1) the Navy shipyards and activities remaining in the working capital fund,
                  (2) ship maintenance activities during periods without appropriations,
                  (3) cost visibility of ship maintenance activities, (4) incentives inherent
                  under NWCF's buyer/seller relationship for improving productivity and
                  performance, and (5) the capital investment program for ship maintenance
                  activities. Other issues that are still unresolved include determining
                  whether the pilot's organization structure under PACFLT ownership has
                  helped streamline the ship maintenance process and improve operations in
                  Hawaii.



Recommendations   As the Navy proceeds with other similar consolidations, we recommend
                  that the Secretary of Defense require the Secretary of the Navy to resolve
                  issues related to the appropriate mechanism to finance and manage these



                  Page 23                                      GAO/NSIAD-99-199 Depot Maintenance
                      B-280524




                      types of activities. Specific financial questions that need to be resolved
                      include the impact of using the direct appropriations to finance the pilot
                      and other potential regionalization actions with regard to (1) the Navy
                      shipyards and activities remaining in the working capital fund, (2) ship
                      maintenance activities during periods without appropriations, (3) cost
                      visibility of ship maintenance activities, (4) incentives inherent under
                      NWCF's buyer/seller relationship for improving productivity and
                      performance, and (5) the capital investment program for ship maintenance
                      activities. In addition, other questions that need to be resolved include
                      determining whether the pilot's command structure under the Fleet's
                      ownership has helped streamline the ship maintenance process and
                      improve operations in Hawaii.



Agency Comments and   On August 5, 1999, we requested comments on a draft of this report from
                      the Secretary of Defense. On August 31, 1999, officials of the Office of the
Our Evaluation        Assistant Deputy Under Secretary of Defense for Maintenance Policy,
                      Programs, and Resources said that they concurred with the intent of our
                      recommendations. However, DOD Comptroller officials expressed concern
                      that the draft of this report did not adequately address the Navy's
                      underlying assertions and assumptions for the pilot or whether the stated
                      objectives of the pilot were met. By recommending that the Navy take steps
                      to address unresolved issues related to financial and organizational
                      structures as it proceeds with similar consolidations, we believe our report
                      addresses the most significant issues for such consolidations. Because the
                      pilot is not yet complete, we did not attempt to determine whether the
                      stated objectives were fully met but, instead, reported the preliminary
                      results of the pilot on improving performance and maintenance activities in
                      Hawaii. We believe that it could take several years for the full effects of the
                      pilot to develop and, as requested, we are reporting the preliminary results.

                      In addition, Navy officials said that they generally concurred with the draft
                      of this report except for our use of fiscal year 1998 data to indicate the
                      preliminary results of the Pearl Harbor Pilot, and they advised that fiscal
                      year 1999 data would provide the most valid assessment of the pilot. They
                      considered fiscal year 1998 as a transition year when the maintenance
                      activities, financial systems, and workforce were not fully integrated and,
                      consequently, did not indicate the pilot's full performance. As discussed in
                      this report, we also considered fiscal year 1998 as a transition year for the
                      pilot and reported that the pilot was unlikely to result in significant
                      improvements during the transition year because of the operational
                      turbulence experienced during the consolidation. However, we used



                      Page 24                                     GAO/NSIAD-99-199 Depot Maintenance
B-280524




available data from both the transition year and fiscal year 1999 to provide
the preliminary results of the pilot. We continue to believe that, in either
year, overall indications are that the pilot has the potential to improve
maintenance activities in Hawaii.

DOD and the Navy also provided technical corrections and clarifications
and, where appropriate, we incorporated them in the report.


We are sending copies of this report to Senator Pete V. Domenici, Senator
Daniel K. Inouye, Senator Frank R. Lautenberg, Senator Carl Levin, Senator
Joseph I. Lieberman, Senator Fred Thompson, Senator Ted Stevens,
Senator Charles S. Robb, and Senator John W. Warner, and to
Representative John R. Kasich, Representative Jerry Lewis, Representative
John P. Murtha, Representative Ike Skelton, Representative Floyd Spence,
and Representative John M. Spratt, Jr., in their capacities as Chair or
Ranking Minority Member of cognizant Senate and House Committees and
Subcommittees. We are also sending copies of this report to the Honorable
William S. Cohen, Secretary of Defense; the Honorable William J. Lynn,
Under Secretary of Defense (Comptroller); the Honorable Richard Danzig,
Secretary of the Navy; and the Honorable Jacob Lew, Director, Office of
Management and Budget. Copies will also be made available to others upon
request.

GAO contacts and key contributors to this report are listed in appendix III.

Sincerely yours,




David R. Warren, Director
Defense Management Issues




Page 25                                    GAO/NSIAD-99-199 Depot Maintenance
Appendix I

Scope and Methodology                                                                    Appendx
                                                                                               Ii




             During our review, we obtained data and interviewed Departments of
             Defense (DOD) and Navy officials, including those from the offices of the
             Under Secretary of Defense (Comptroller), Deputy Under Secretary of
             Defense for Logistics, DOD Inspector General, Assistant Secretary of the
             Navy for Financial Management and Comptroller, Chief of Naval
             Operations (CNO), Naval Sea Systems Command (NAVSEA), the Pacific
             Fleet (PACFLT), Pearl Harbor Naval Shipyard & Intermediate Maintenance
             Facility (NSY & IMF), San Diego Supervisor of Ship Building Conversion
             and Repair Pearl Harbor Detachment, and the Naval Audit Service (NAS).
             In addition, we interviewed officials of the Hawaii Federal Employees
             Metal Trades Council, the Ship Repair Association of Hawaii, and
             contractors involved in the management and assessment of the Pearl
             Harbor Pilot.

             To identify the preliminary results of the Pearl Harbor Pilot, we obtained
             and reviewed status reports and briefings, financial and human resources
             documents, infrastructure data, budget documents, NAS capacity
             evaluation reports and documents, the Pearl Harbor Pilot Test Plan and
             related guidance, and PACFLT regional maintenance and business plans. In
             addition, we interviewed DOD and Navy officials to discuss the preliminary
             results of the pilot and challenges that must be resolved to obtain the full
             benefits of the consolidation. We observed the operations of the Pearl
             Harbor NSY & IMF and interviewed superintendents, first-line supervisors,
             and workers to identify operational conditions and determine the pilot's
             impacts on their maintenance shops and activities. We reviewed the Pearl
             Harbor Pilot Study Report and other planning documents to identify
             maintenance procedures and processes expected to improve under the
             pilot and compared these expectations with actual operational results for
             fiscal year 1998 and the beginning of fiscal year 1999. When these results
             did not meet expectations, we discussed the differences with Navy officials
             to determine the causes and identify expectations for future operations. In
             addition, we interviewed officials of various PACFLT commands
             (customers of the Pearl Harbor NSY & IMF) to determine whether the
             combined facility's maintenance activities and services were comparable to
             those provided prior to the consolidation. Additionally, we interviewed
             Navy officials and collected data on the apprentice program, safety and
             quality assurance programs, and issues related to the morale of Pearl
             Harbor NSY & IMF.

             To assess the adequacy of the Pearl Harbor Pilot Test Plan, we reviewed the
             DOD Inspector General's preliminary assessment of the test plan and
             baseline data. We also interviewed DOD, Navy, and contractor officials



             Page 26                                    GAO/NSIAD-99-199 Depot Maintenance
Appendix I
Scope and Methodology




about the methodology for the test plan, reliability of the data, and
strengths and weaknesses of each test metric. To obtain an understanding
of the methodology and data reliability, we reviewed the process and data
used to develop the fiscal year 1997 baseline, the fiscal year 1998 transition
results, and the fiscal year 1999 preliminary results and, on a selected basis,
we traced the data to their original source documents. To identify potential
areas for improving the test plan, we discussed the strengths and
weaknesses of the test plan with DOD, Navy, and contractor officials. We
also discussed several key elements of the pilot that were omitted from the
test plan, including the reasons for omission or the potential need for
inclusion.

To identify issues that remain for future consolidations at other Navy
locations, we obtained and reviewed status reports and briefings, financial
and human resources documents, the Pearl Harbor Pilot Test Plan and
related guidance, and PACFLT regional maintenance and business plans. In
addition, we interviewed DOD and Navy officials to discuss issues that
must be resolved to obtain the full benefits of the Pearl Harbor
consolidation and other similar consolidations at other Navy locations. In
addition, during meetings with DOD and Navy officials, we discussed the
advantages and disadvantages of whether future consolidations should use
direct appropriations, the Naval Working Capital Fund (NWCF), or a
combination of the two financial systems. To help assess the potential
impact of removing the former Pearl Harbor Naval Shipyard (PHNSY) and
other naval shipyards from the NWCF, we identified and analyzed the
percentage of revenues the former PHNSY and the other shipyards
contributed to the fund during fiscal years 1997 and 1998. We also
discussed the potential impact of their removal from the fund with DOD
and Navy officials. Additionally, during meetings with DOD and Navy
officials, we discussed the advantages and disadvantages of whether
NAVSEA or the fleets should be responsible for the ownership and
management of future consolidations. To determine whether the pilot
could serve as a model for similar consolidations at other locations, we
identified and analyzed reasons why the Navy selected Pearl Harbor
maintenance activities for the pilot. We compared these conditions with
those found at the Puget Sound, Portsmouth, and Norfolk areas and
discussed the potential impact of these differences on other consolidations
with DOD and Navy officials.

Furthermore, we documented the reasons the Navy selected direct
appropriations, and the pilot's experience of operating under direct
appropriations. We also interviewed DOD and Navy officials to discuss the



Page 27                                      GAO/NSIAD-99-199 Depot Maintenance
Appendix I
Scope and Methodology




actual level of cost visibility retained under the pilot and the various levels
considered adequate for ship maintenance activities. In addition, we
determined the level of cost visibility retained for four ongoing
maintenance projects for the U.S.S. Charlotte, U.S.S. Russell, U.S.S.
Tucson, and U.S.S. Crommelin. Specifically, we reviewed the start date,
estimated date of completion, estimated cost, direct labor hours of military
and civilian personnel, overtime hours, material and supply cost, and
contract services for the projects. In addition, to determine whether the
same level of detail was being maintained by the Pearl Harbor NSY & IMF
under direct appropriations, we reviewed material expenditures and labor-
hours for military and civilian personnel by job order number for the
projects.

In performing this review, we used the same budget and accounting
systems, reports, and statistics DOD and the Navy use to manage and
monitor their ship maintenance program. We did not independently
determine the reliability of the reported financial information. However,
our recent audit of the federal government’s financial statements, including
DOD’s and the Navy’s statements, questioned the reliability of reported
financial information because not all obligations and expenditures are
recorded to specific budgetary accounts.

We conducted our review from May 1998 to July 1999 in accordance with
generally accepted government auditing standards.




Page 28                                      GAO/NSIAD-99-199 Depot Maintenance
Appendix II

Description of the Pearl Harbor Pilot Test Plan
and Its Strengths and Weaknesses                                                                                                                     Appendx
                                                                                                                                                           Ii




                                                 The Navy's test plan for assessing the effectiveness of the Pearl Harbor
                                                 Pilot evolved between December 1997, when the Deputy Secretary of
                                                 Defense first required a test plan, and September 1998, when the last three
                                                 test metrics were added to the plan. During this evolution, DOD, Navy, and
                                                 audit agency officials expressed various concerns with the scope and
                                                 validity of the test plan. Ultimately, the Navy selected nine test metrics to
                                                 evaluate the productivity and performance of the pilot.



Metrics to Be Used to                            After several revisions and additions, the current test plan contains nine
                                                 metrics that capture unit cost per output, production efficiency, material
Evaluate the Pilot                               readiness of ships home ported at Pearl Harbor, and customer satisfaction.
                                                 During the initial planning period in late 1997, Navy officials discussed
                                                 several metrics to measure the impact of the pilot and initially agreed upon
                                                 the two shown in table 4.



Table 4: Navy's Initial Two Metrics to Test Whether the Consolidated Pearl Harbor NSY & IMF Increased Manpower Utilization
and Lowered Unit Cost of Production

Metric                                       Data elements                                               Calculation
1. Total cost of a production shop, direct • Total costs of the ship maintenance activity                • Total costs of the ship maintenance
labor hour of work delivered to the        • Total production shop, direct labor hours delivered           activity or activities divided by the
customer                                                                                                   total production shop, direct labor
                                                                                                           hours delivered
2. Total available labor hours expended      • Total available labor hours, including direct and         • Total available labor hours divided by
to deliver a production shop, direct labor     indirect hours, expended by the ship maintenance            the total production shop, direct labor
hour to the customer                           activity                                                    hours delivered
                                             • Total production shop, direct labor hours delivered


                                                 Source: Interviews with Navy officials and our analysis of the Pearl HarborPilot Test Plan dated
                                                 June 30, 1998.


                                                 In December 1997, recognizing that the Navy had not developed a plan or
                                                 established a baseline for measuring success or failure for the pilot, the
                                                 Deputy Secretary of Defense required the Navy to develop a test plan to
                                                 measure the benefits of the pilot.1 In addition, the Deputy Secretary of
                                                 Defense suggested that “productive man-days delivered per dollar outlay”
                                                 and “number of civilian end strengths and military on-board it took to


                                                 1
                                                     Program Budget Decision Number 404, Deputy Secretary of Defense (Dec. 11, 1997).




                                                 Page 29                                                   GAO/NSIAD-99-199 Depot Maintenance
Appendix II
Description of the Pearl Harbor Pilot Test
Plan and Its Strengths and Weaknesses




produce the productive man-days” should be used as key assessment
metrics. Subsequently, the Chief of Naval Operations (CNO) and the Naval
Sea Systems Command (NAVSEA) requested the Naval Audit Service
(NAS) determine the combined cost of operations for the former Pearl
Harbor Naval Shipyard (PHNSY) and the Naval Intermediate Maintenance
Facility (NIMF) for fiscal year 1997.2 In an April 1998 report, the NAS
provided a basis to compare the cost effectiveness of combining the two
maintenance activities and metric calculations for the two assessment
metrics defined at that time, which are the first two metrics of the current
test plan.3 In the report, the NAS recognized that the consolidation was an
evolving process and revisions to its report may be necessary to reflect
changes in consolidation decisions. In May 1999, after making changes to
the baseline data and fine-tuning its methodology, the NAS revised its
baseline cost and productivity estimates. 4 The revised estimates did not
change significantly from the initial estimates.

In April 1998, the Navy issued a draft of the Pearl Harbor Pilot Test Plan
that included five metrics to capture cost per unit output, production
efficiency and resource utilization, material readiness of PACFLT ships,
customer satisfaction, and quality. However, in June 1998, the DOD
Assistant Inspector General for Auditing stated that the draft test plan
would not provide sufficient or supportable data to draw conclusions about
the effectiveness of consolidating the operations of the former shipyard
and intermediate maintenance facility. Consequently in June 1998, a joint
DOD and Navy team issued a revised Pearl Harbor Pilot Test Plan that
changed the basis for the quality metric from formal customer surveys to
an analysis of rework items and added a metric measuring the number of




2
 The Navy selected fiscal year 1997 as the baseline year because this was the last full year
the former PHNSY and NIMF operated as independent activities. Fiscal year 1998 was
eliminated because of the operational turbulence expected by the consolidation of activities
during the year. Fiscal year 1996 was eliminated because there were separate surface ship
and submarine intermediate maintenance facilities operating during the fiscal year.
3
  Capacity Evaluation Report: Baseline Costs of Operating Naval Intermediate Maintenance
Facility, Pearl Harbor and Pearl Harbor Naval Shipyard, Auditor General of the Navy (98-
0400, Apr. 1998).
4
  Capacity Evaluation Report: Revised Baseline Costs of Operations and Metrics for Naval
Intermediate Maintenance Facility, Pearl Harbor and Pearl Harbor Naval Shipyard
Consolidation, Auditor General of the Navy (99-0408, May 1999).




Page 30                                             GAO/NSIAD-99-199 Depot Maintenance
                                          Appendix II
                                          Description of the Pearl Harbor Pilot Test
                                          Plan and Its Strengths and Weaknesses




                                          work-items completed.5 Consequently, the Navy included four metrics in
                                          addition to those two metrics listed in the previous table to its final test
                                          plan. (See table 5.)



Table 5: Metrics Three Through Six to Test Whether the Consolidated Pearl Harbor NSY & IMF Increased Manpower Utilization
and Lowered Unit Cost of Production

Metric                                Data elements                                               Calculation
3. Total Current Ship Maintenance     • Total number of CSMP work items completed on        • None required, absolute number of
Program (CSMP) work items completed     ships home ported at Pearl Harbor during the fiscal   completed CSMP work items used
                                        year
4. Total Current Ship Maintenance     • Total number of CSMP work items in the backlog for • None required, absolute number of
Program work items in the backlog       ships home ported at Pearl Harbor                    CSMP work items in the backlog used
5. Schedule adherence of CNO          • Sum of the differences in the actual and scheduled        • Sum of the differences in the actual
maintenance projects                    completion dates for each CNO ship maintenance              and scheduled completion dates
                                        project completed during the fiscal year                    divided by the total scheduled
                                      • Sum of the scheduled duration (number of days) for          duration (number of days) for each
                                        each CNO maintenance project completed during               CNO scheduled ship maintenance
                                        the fiscal year                                             project completed during the fiscal
                                                                                                    year
6. Rework index for CNO maintenance   • Sum of the labor hours expended to correct work           • Sum of the labor hours expended to
projects                                deficiencies for each CNO scheduled ship                    correct work deficiencies divided by
                                        maintenance project completed during the fiscal             the total number of direct labor hours
                                        year                                                        delivered for each CNO scheduled
                                      • Total production shop, direct labor hours delivered         ship maintenance project completed
                                        for each CNO scheduled ship maintenance project             during the fiscal year
                                        completed during the fiscal year


                                          Source: Interviews with Navy officials and our analysis of the Pearl HarborPilot Test Plan dated
                                          June 30, 1998.


                                          In September 1998, the Principal Deputy Under Secretary of Defense
                                          (Comptroller) suggested the Navy incorporate three additional test metrics
                                          developed under contract by the Anteon Corporation into the revised June
                                          1998 test plan.6 (See table 6.) In response, the Acting Deputy Chief of Naval



                                          5
                                           The Pearl Harbor NSY & IMF has developed three additional internal metrics:
                                          production-shop hours versus total available hours, direct production worker versus non-
                                          production worker, and civilian labor cost per labor hour. While they are not formally part of
                                          the test plan, Pearl Harbor NSY & IMF officials believe the metrics will assist them in
                                          assessing the effectiveness of the pilot.
                                          6
                                              The Pearl Harbor Pilot Study, Anteon Corporation (Sept. 10, 1998).




                                          Page 31                                                   GAO/NSIAD-99-199 Depot Maintenance
                                             Appendix II
                                             Description of the Pearl Harbor Pilot Test
                                             Plan and Its Strengths and Weaknesses




                                             Operations (Logistics) attached them in an appendix E to the plan. At this
                                             time, Principal Deputy Under Secretary of Defense approved the test plan.



Table 6: Metrics Seven Through Nine to Test Whether the Consolidated Pearl Harbor NSY & IMF Increased Manpower Utilization
and Lowered Unit Cost of Production

Metric                         Data elements                                              Calculation
7. Activity work schedule      • Budgeted quantity of work scheduled (labor days)         • Budgeted allowance of work scheduled (labor
integrity index                • Actual quantity of work performed (labor days)             days) divided by the actual amount of work
                                                                                            performed (labor days)
8. Casualty reportsa caused by • Casualty reports                                         • Analysis of reports occurring within 6 months
activity work                                                                               following the completion of maintenance
9. Earned value                • Actual quantity of work performed (labor days) for  • Statistical analysis of actual quantity of work
                                 selected Ship Work Line Item Numbers for the fiscal   performed (labor days) for selected Ship Work
                                 year                                                  Line Item Numbers for fiscal years 1997 and
                                                                                       1999


                                             a
                                              Reports filed by the ship of equipment failures that may indicate faulty work performed during more
                                             recent maintenance projects.

                                             Source: Interviews with Navy and Anteon Corporation officials and our analysis of the Pearl Harbor
                                             Pilot Test Plan dated June 30, 1998.


                                             In November 1998, the Director, Industrial Capability, Maintenance Policy
                                             and Acquisition Logistics Division, CNO, requested the NAS to assume the
                                             overall lead and control in the test plan assessment process. The Director
                                             requested that NAS involvement be expanded to include data collection for
                                             all of the test plan metrics, metric calculation and assessment, and pilot
                                             metric assessment reporting. In response, the auditors agreed to continue
                                             to develop and report data for the first two metrics in the test plan and
                                             verify the data for metrics number 3 through 6. Additionally, NAS
                                             considered the responsibility for data collection and reporting the results
                                             of the pilot to be a management function. It did not assume responsibility
                                             for collecting and validating the data for the contractor-generated metrics.



Baseline, Performance                        Given the time and data constraints in developing the Pearl Harbor Pilot
                                             Test Plan, the test plan and assessment metrics provide the Navy a
Expectations, and                            mechanism for evaluating the pilot's productivity and performance. As
Preliminary Values for                       shown in table 7, the Navy has developed the values for the fiscal year 1997
                                             baseline and performance expectations for six of the nine metrics and
the Pearl Harbor Pilot


                                             Page 32                                                  GAO/NSIAD-99-199 Depot Maintenance
                                                 Appendix II
                                                 Description of the Pearl Harbor Pilot Test
                                                 Plan and Its Strengths and Weaknesses




                                                 preliminary values for five metrics. However, the details for collecting and
                                                 analyzing the contractor-generated metrics have not yet been determined.



Table 7: Baseline, Performance Expectations, and Preliminary Results of the Test Plan

Metric                                Fiscal year 1997 baseline              Expectation                     Preliminary result
1. Total cost of a production shop,   • $135.36 in current year      • 3-5 percent reduction                 • $135.59 per hour in fiscal year 1998
direct labor hour of work delivered     dollars
to the customera                      • $138.13 in adjusted dollarsb
2. Total available labor hours        • 3.15 hours                            • 3-5 percent reduction        • 3.04 hours in fiscal year 1998
expended to deliver a production
shop, direct labor hour to the
customera
3. Total Current Ship Maintenance     • 19,777                                • Maintain at least the        • 14,568 items in fiscal year 1998
Program work items completedc                                                   same completion rate         • 7,786 items as of May 1999
4. Total Current Ship Maintenance • 17,733                                    • 10 percent reduction         • 16,462 items as of September 30, 1998
Program work items in the backlogc                                                                           • 16,538 items as of May 1999
5. Schedule adherence of CNO          • 11.4 percent late                     • To improve                   • 18.6 percent late in fiscal year 1998
maintenance projectsc                                                                                        • 3.7 percent early in fiscal year 1999
                                                                                                               through May 1999
6. Rework index for CNO               • 0.086                                • No degradation of the         • Not yet determined
maintenance projectsc                                                          index
7. Activity work schedule integrity   • Not yet determined                    • Unclear                      • Not yet determined
index
8. Casualty reports caused by         • Not yet determined                    • Unclear                      • Not yet determined
activity work
9. Earned value                       • Varies by work item                   • Unclear                      • Not yet determined


                                                 a
                                                     Reported by the NAS.
                                                 b
                                                     Adjusted for increases in military and civilian pay rates due to inflation.
                                                 c
                                                  Reported by the Pearl Harbor NSY & IMF, but not yet verified by the NAS.

                                                 Source: Interviews with Navy and Anteon Corporation officials and our analysis of the Pearl Harbor
                                                 Pilot Test Plan dated June 30, 1998; NAS capacity evaluation reports for fiscal years 1997 and 1998;
                                                 and pilot status reports.


                                                 Although the Navy has established nine metrics that provide indicators of
                                                 productivity and performance, the metrics have data limitations and
                                                 weaknesses that preclude their use for precise measurements.
                                                 Nevertheless, they provide a basis for an order of magnitude comparison. A
                                                 discussion of the nine metrics follows.




                                                 Page 33                                                         GAO/NSIAD-99-199 Depot Maintenance
                             Appendix II
                             Description of the Pearl Harbor Pilot Test
                             Plan and Its Strengths and Weaknesses




Cost of a Production Shop,   Measuring the pilot's efficiency in terms of the cost per direct maintenance
Direct Labor Hour of Work    hour, the Navy's first metric is determined by dividing the total activity cost
                             by the number of production shop, direct labor hours delivered to the
Delivered                    customer.7 The NAS estimated that it cost $135.36 to deliver one production
                             shop, direct labor hour in fiscal year 1997—the baseline year for measuring
                             the pilot's success. For the pilot to show improvement, this index should
                             decrease, indicating that since the consolidation, it cost less to deliver a
                             direct maintenance hour. The NAS estimated that the index was $135.59
                             during the fiscal year 1998 transition, indicating some improvement when
                             compared with the adjusted baseline of $138.13, but falling short of Navy's
                             performance expectation of 3- to 5-percent reduction in the index.8 The
                             NAS adjusted the baseline to account for differences in military and civilian
                             wages and inflation between fiscal years 1997 and 1998. Because of the
                             operational turbulence experienced during the consolidation, we believe
                             this indicator was unlikely to show a significant improvement during the
                             transition year.

Strengths and Weaknesses     In analyzing this metric, we noted strengths and weaknesses. The strengths
                             of this metric are that the baseline is partly based on shipyard data, which
                             are considered reliable, and the NAS developed the baseline estimates and
                             plans to validate and generate the test data. However, DOD and Navy
                             officials have expressed concern with the accuracy of the intermediate
                             maintenance data collected during the baseline year, which is considered a
                             weakness. For example, military personnel costs were not collected
                             because intermediate maintenance facilities traditionally did not collect
                             these costs. In addition, some production hours appeared to be inaccurate.
                             For example, a DOD contractor reported the former NIMF database
                             contained zero labor hours for approximately 6 percent of the maintenance
                             tasks completed by the facility. Consequently, to strengthen the metric, the
                             NAS developed a methodology that used Navy personnel wages and
                             standards and the best available NIMF data to estimate a baseline for this
                             metric. Although the NIMF baseline data on military personnel costs and



                             7
                              The production shops consist of the crane, ship fitter, sheet metal, welding, inside
                             machine, outside machine, boilermaker, electrical, pipe fitter, wood/fabric, cryptographic
                             shore electronics, electronics, shipwright/rigging, and temporary services shops and the
                             Navy calibration and nuclear regional maintenance departments.
                             8
                               Fiscal Year 1998 Interim Costs for the Consolidated Pearl Harbor Naval Shipyard and
                             Intermediate Maintenance Facility, Pearl Harbor, Auditor General of the Navy (99-0400,
                             June 1, 1999).




                             Page 34                                             GAO/NSIAD-99-199 Depot Maintenance
                             Appendix II
                             Description of the Pearl Harbor Pilot Test
                             Plan and Its Strengths and Weaknesses




                             production hours have limitations, they are considered reliable enough by
                             Navy officials to provide a rough, if imprecise, estimate of the baseline.


Labor Hours Expended to      Measuring the pilot's productivity in terms of personnel utilization, the
Deliver a Production Shop,   second metric is determined by dividing the total number of available labor
                             hours by the number of production shop, direct labor hours delivered to
Direct Labor Hour            the customers. The NAS estimated that it took 3.15 activity labor hours
                             (overhead and direct maintenance hours) to deliver one production shop,
                             direct labor hour in fiscal year 1997. For the pilot to show improvement,
                             this index should decrease indicating that it has taken fewer overhead
                             hours to generate 1 direct maintenance hour since the consolidation. The
                             NAS reported that the index decreased to 3.04 hours in fiscal year 1998,
                             indicating a 3.5-percent improvement during the transition year. This met
                             the low end of the Navy's performance expectation of 3- to 5-percent
                             reduction in value.

Strengths and Weaknesses     In our analysis, we noted the strengths and weaknesses of this metric. The
                             strengths of this metric are that the baseline is partly based on shipyard
                             data, which are considered reliable, and the NAS developed the baseline
                             estimates and plans to validate and generate the test data. However, as
                             discussed previously, DOD and Navy officials have expressed concern with
                             the intermediate maintenance data collected during the baseline year.
                             Consequently, to strengthen the metric, the NAS developed a methodology
                             that used best available NIMF data to estimate a baseline for this metric.
                             Although the NIMF baseline data on production hours have weaknesses,
                             they are considered reliable enough by Navy officials to provide a rough, if
                             imprecise, estimate of the baseline.


Current Ship Maintenance     Measuring the pilot's productivity in terms of output, the third metric is the
Program Work Items           number of Current Ship Maintenance Program (CSMP) work items
                             completed by the Pearl Harbor NSY & IMF. Although this metric is not an
Completed                    indicator of the former shipyard's productivity, Navy officials have
                             considered the number of CSMP work items completed by their
                             intermediate maintenance facilities as a measurement of their
                             productivity.9 The CSMP is the central database for the maintenance
                             history of each ship and contains corrective, preventative, and alteration


                             9
                                 Traditionally, naval shipyards completed a minimal number of CSMP items.




                             Page 35                                             GAO/NSIAD-99-199 Depot Maintenance
                            Appendix II
                            Description of the Pearl Harbor Pilot Test
                            Plan and Its Strengths and Weaknesses




                            maintenance items requiring work. The Pearl Harbor NSY & IMF estimated
                            that the former intermediate maintenance facility completed 19,777 CSMP
                            work items in fiscal year 1997. However, it further estimated that the
                            consolidated activity completed 14,568 CSMP work items during the fiscal
                            year 1998 transition and 7,786 work items as of May in fiscal year 1999,
                            indicating no improvement in performance since the consolidation.
                            Although the combined facility was expected to maintain the same
                            completion rate as the former NIMF, several Navy officials believe that this
                            expectation was unreasonable because the number of military enlisted
                            personnel decreased from 1,275 in October 1996 to 616 in April 1999.
                            Traditionally, enlisted personnel completed the majority of CSMP work
                            items. Consequently, Navy officials have discussed lowering the
                            performance expectation for this metric.

Strengths and Weaknesses    During our analysis, we found that this metric had both strengths and
                            weaknesses. The strengths of this metric are that the fleet and ship
                            commanders maintain the baseline and test data, which are considered
                            reliable, and the NAS plans to validate the baseline and pilot results.
                            However, other factors not related to the pilot affect the metric that
                            weaken its usefulness. For example, Navy officials have expressed concern
                            with using CSMP work items to measure the benefits of the Pearl Harbor
                            Pilot because one work item does not equal another item in terms of the
                            labor hours, work skills, time frame, and material cost required. CSMP
                            items range from replacing a light bulb in a control panel, which may take a
                            worker less than an hour to replace, to overhauling a pump, which may
                            take several workers days to complete. In addition, the requirements
                            needed to overhaul one pump differ from other overhauls depending on the
                            problem and the type of pump. Although this metric has weaknesses that
                            preclude its use for precise measurements, Navy officials believe that it
                            provides a basis for an order-of-magnitude comparison.


Current Ship Maintenance    Historically, Navy officials have measured the material condition of their
Program Work Items in the   ships based on the number of backlogged CSMP items. Fewer backlogged
                            items imply that the ships are in better condition. The Navy's performance
Backlog                     expectation for the combined facility was to reduce the estimated
                            17,733-baseline backlog by 10 percent to 15,960 work items. Recently, the
                            Pearl Harbor NSY & IMF estimated that the backlog decreased to 16,538
                            items as of May 1999. Although this decrease indicates improvement, it falls
                            short of the Navy's expectation for the pilot. However, it was unlikely that
                            the CSMP backlog would decrease significantly during this period because




                            Page 36                                      GAO/NSIAD-99-199 Depot Maintenance
                            Appendix II
                            Description of the Pearl Harbor Pilot Test
                            Plan and Its Strengths and Weaknesses




                            of the reduction in the number of military enlisted personnel since October
                            1996.

Strengths and Weaknesses    As discussed previously, using CSMPs as a test metric had both strengths
                            and weaknesses. The strengths of this metric are that it traditionally has
                            been considered an indicator of the material condition of fleet ships and
                            that the fleet and ship commanders maintain the baseline and test data,
                            which are considered reliable. In addition, the NAS plans to validate the
                            baseline and pilot results. For many of the same reasons that CSMP work
                            items are not a precise measurement of output, the backlog is not a precise
                            measurement of the ship's material condition, which makes the backlog a
                            weakness in regard to measurement. For example, one work item related
                            to the safety of a nuclear submarine could potentially restrict the
                            submarine operations, while a similar item on a surface ship could have
                            little or no effect on its operations. Additionally, other factors affecting the
                            backlog include non-pilot related items such as

                            • decommissions of PACFLT ships homeported at Pearl Harbor, which
                              decrease the backlog by the number of CSMP items recorded for the
                              decommissioned ship;
                            • visits by the U.S.S. McKee (a maintenance tender), which decrease the
                              backlog by the number of CSMP items completed by the tender;
                            • maintenance inspections, which increase the backlog by the number of
                              unrecorded CSMP items identified by the inspection team; and
                            • procedural changes in identifying and recording CSMP items, which
                              may increase or decrease the backlog depending on whether the
                              changes weaken or strengthen the process.

                            Although this metric has weaknesses that preclude its use for precise
                            measurements, Navy officials believe that it provides a basis for an order-
                            of-magnitude comparison.


Schedule Adherence of CNO   According to Navy officials, adherence to completing CNO maintenance
Maintenance Projects        projects on schedule should provide an indication of customer satisfaction
                            because, from the fleet and ship commanders' perspective, these
                            maintenance projects should be completed on schedule.10 The performance


                            10
                              CNO maintenance projects include depot-level maintenance that require skills or facilities
                            beyond those of the ship and intermediate maintenance activities and are completed by
                            public and private shipyards.




                            Page 37                                             GAO/NSIAD-99-199 Depot Maintenance
                           Appendix II
                           Description of the Pearl Harbor Pilot Test
                           Plan and Its Strengths and Weaknesses




                           expectation during the pilot test was to reduce the 11.4 percent schedule
                           adherence index achieved during fiscal year 1997, since a lower percentile
                           would indicate that projects were being completed closer to their
                           scheduled completion dates.11 According to the Pearl Harbor NSY & IMF,
                           the schedule adherence index increased to 18.6 percent during fiscal year
                           1998—indicating a worsening performance during the transition year.
                           However, Pearl Harbor NSY & IMF data showed that the schedule
                           adherence index for the CNO maintenance projects completed in fiscal
                           year 1999 has exceeded the Navy's performance expectations for the pilot.
                           For example, although the first CNO project completed in fiscal year 1999
                           was 20 percent late, the four subsequent CNO projects have been either on
                           time or early, decreasing the cumulative index to 3.7 percent early.

Strengths and Weaknesses   The strengths of this metric are that it traditionally has been considered an
                           indicator of the shipyard customers' satisfaction and that NAVSEA
                           maintains the baseline and test data, which are considered reliable. In
                           addition, the NAS plans to validate the baseline and pilot results. The
                           weaknesses include that the metric is based solely on the former shipyard's
                           performance in meeting the completion dates for CNO maintenance
                           projects and, consequently, does not include all types of maintenance
                           projects. Pearl Harbor NSY & IMF officials believe this metric should
                           include fleet maintenance projects because there are more fleet
                           maintenance projects than CNO projects, thus affecting more ship
                           commanders. However, according to CNO and NAVSEA officials, they
                           included only CNO projects for this metric because the schedule adherence
                           data for fleet maintenance projects were considered of minimal value. For
                           example, the former intermediate maintenance process was to defer work
                           rather than to extend the duration of fleet maintenance projects and,
                           consequently, these projects were generally completed on time. Because
                           the pilot has not changed this process for fleet maintenance projects, CNO
                           and NAVSEA officials believe an analysis of schedule adherence data for
                           these projects would be of minimal value in assessing the pilot's
                           effectiveness.




                           11
                             The schedule adherence index is the sum of the differences in the actual and scheduled
                           completion dates divided by the total scheduled duration (number of days) for each CNO
                           ship maintenance project completed during the fiscal year.




                           Page 38                                            GAO/NSIAD-99-199 Depot Maintenance
                           Appendix II
                           Description of the Pearl Harbor Pilot Test
                           Plan and Its Strengths and Weaknesses




Rework Index for CNO       In analyzing this metric, we found strengths and weaknesses. Measuring
Maintenance Projects       the quality of work completed by the Pearl Harbor NSY & IMF, the sixth
                           metric is determined by dividing the total number of direct rework hours
                           required to correct work deficiencies by the number of direct labor hours
                           delivered for CNO maintenance projects completed during the fiscal year.
                           Rework hours are the hours spent to correct work deficiencies after the
                           maintenance project is completed.

Strengths and Weaknesses   The strengths of the metric are that it traditionally has been considered an
                           indicator of the shipyard quality of work and the NAS plans to validate the
                           baseline and test data. However, its weaknesses include that the baseline
                           for this metric is limited to the shipyard's CNO projects because the former
                           intermediate maintenance facility did not collect rework data for its
                           projects. In addition, the metric has not been computed for fiscal years
                           1998 or 1999 because no one has assumed responsibility for tracking this
                           information since the inception of the pilot.


Activity Work Schedule     Measuring the pilot's degree of schedule integrity, the seventh metric is
Integrity Index            calculated by dividing the budget quantity of work (direct labor hours)
                           scheduled by the actual amount of work (direct labor hours) delivered to
                           complete the maintenance project. According to the test plan, actual work
                           exceeding budgeted work indicates inefficiencies in the facility's planning
                           or production processes. A metric value of more than one indicates the
                           maintenance project was completed within budget.

Strengths and Weaknesses   The strengths of this metric are that it traditionally has been considered an
                           indicator of the shipyard customers' satisfaction and the shipyard
                           maintains the baseline and test data, which are considered reliable.
                           However, one weakness of metric is that it does not include data from the
                           former intermediate maintenance facility. In addition, Navy officials believe
                           this metric is similar to the fifth metric measuring schedule adherence and,
                           therefore, is not significantly useful in determining the benefits of the pilot.
                           Consequently, they have not expended resources to compute the metric
                           value for the baseline year or established performance expectations for this
                           metric. Furthermore, the details of collecting and analyzing the data have
                           not yet been determined. According to CNO and NAVSEA officials, the
                           Navy does not plan to gather the data needed to develop and analyze this
                           metric unless the results of the first six metrics are inconclusive. For
                           example, if the pilot results for those six metrics are mixed, then it may be
                           necessary to measure other pilot results, such as this metric.



                           Page 39                                      GAO/NSIAD-99-199 Depot Maintenance
                             Appendix II
                             Description of the Pearl Harbor Pilot Test
                             Plan and Its Strengths and Weaknesses




Casualty Reports Caused by   Measuring the quality of work completed by the Pearl Harbor NSY & IMF,
Activity Work                the eighth metric is based on analysis of casualty reports. These reports are
                             filed by the ship after any equipment failure and may help identify work
                             improperly performed by the Pearl Harbor NSY & IMF. Those reports
                             indicating improperly performed work may offer insight to potential
                             corrective actions to the facility's methods and procedures.

Strengths and Weaknesses     We noted strengths and weaknesses in our analysis of this metric. The
                             primary strength of the metric is that the fleet and ship commanders
                             maintain the baseline and test data, which are considered reliable.
                             However, considered a weakness, the metric does not include data from
                             the former intermediate maintenance facility. In addition, Navy officials
                             believe this metric is similar to the sixth metric measuring rework items
                             and, therefore, is not significantly useful in identifying the benefits of the
                             pilot. Consequently, they have not expended resources to determine the
                             details of collecting and analyzing the data for this metric. Similar to the
                             previous metric, the Navy does not plan to gather the data needed to
                             develop and analyze this metric unless the results of the first six metrics
                             are inconclusive.


Earned Value                 For the earned value metric, the process determines the value of a
                             completed task by comparing actual labor hours it took to complete a unit
                             of ship maintenance work in fiscal year 1997 with actual labor hours it took
                             to complete the same unit of work in fiscal year 1999 and subsequent test
                             years. This metric attempts to measure the pilot's effect on maintenance
                             outputs. Based on an analysis of the former shipyard and NIMF production
                             and labor data for fiscal year 1997, the Anteon Corporation identified
                             14 surface ship and submarine systems that were considered cost drivers,
                             consuming the most labor hours during the fiscal year.12 On the basis of the
                             distribution of labor hours for each of these cost drivers, the Anteon
                             Corporation determined that, while the distributions are nearly identical in
                             form, they were not distributed normally at the 99.9 percent confidence
                             level. The most common statistics used to describe data normally
                             distributed are the mean and standard deviation. In this case, however, the
                             mean and standard deviations do not adequately represent the data
                             distribution. Consequently, non-parametric statistics−which are generally


                             12
                               The Anteon Corporation defined a cost driver as a quantity of ship maintenance work that
                             represented a major or controlling portion of work on a particular ship maintenance project.




                             Page 40                                             GAO/NSIAD-99-199 Depot Maintenance
                           Appendix II
                           Description of the Pearl Harbor Pilot Test
                           Plan and Its Strengths and Weaknesses




                           based on the median (as opposed to the mean)−and percentile ranges (as
                           opposed to the standard deviation) of the distribution of labor hours will be
                           used to describe the labor hour distribution for each of the cost drivers.

                           If the distribution of labor hours for the 14 cost drivers in fiscal year 1999 is
                           less than the fiscal year 1997 baseline, the earned value metric indicates
                           that the productivity of the work performed at the combined facility has
                           improved compared to when the facilities operated separately. Conversely,
                           if distribution of labor hours in fiscal year 1999 is greater than the fiscal
                           year 1997 baseline, the productivity of the combined facility has decreased
                           compared to when the facilities operated separately. Although it is
                           expected that the cost drivers from fiscal year 1997 will be similar to those
                           in fiscal year 1999, a statistical method will be used to verify that the data
                           for the different fiscal years are similar. If the data are not statistically
                           similar, additional analysis of data will be conducted to develop the earned
                           value metric.

Strengths and Weaknesses   A strength of this metric is that it attempts to measure the pilot's effect on
                           maintenance outputs. However, the primary weakness of the metric,
                           according to Navy officials, is that it affects such a small population that
                           the metric is not significantly useful in measuring the efficiency of the pilot.
                           Another stated weakness is that the metric requires significant resources to
                           develop the results. Consequently, Navy officials have not expended
                           resources to determine the details of collecting and analyzing the data for
                           the metric. Similar to the previous two metrics, the Navy does not plan to
                           gather the data needed to develop and analyze this metric unless the results
                           of the first six metrics are inconclusive.




                           Page 41                                       GAO/NSIAD-99-199 Depot Maintenance
Appendix III

GAO Contacts and Staff Acknowledgments                                                      AppendxIi




GAO Contacts      David Warren (202) 512-8412
                  Julia Denman (202) 512-4290



Acknowledgments   In addition to those named above, Mark Little,
                  Dennis De Hart, Jean Orland, Barbara Wooten, and
                  John Brosnan made key contributions to this report.




                  Page 42                                  GAO/NSIAD-99-199 Depot Maintenance
Related GAO Products


             Navy Ship Maintenance: Allocation of Ship Maintenance Work in the
             Norfolk, Virginia, Area (GAO/NSIAD-99-54, Feb. 24, 1999).

             Defense Depot Maintenance: Public and Private Sector Workload
             Distribution Reporting Can Be Further Improved (GAO/NSIAD-98-175,
             July 23, 1998).

             Defense Depot Maintenance: Contracting Approaches Should Address
             Workload Characteristics (GAO/NSIAD-98-130, June 15, 1998).

             Defense Depot Maintenance: Use of Public-Private Partnering
             Arrangements (GAO/NSIAD-98-91, May 7, 1998).

             Navy Ship Maintenance: Temporary Duty Assignments of Temporarily
             Excess Shipyard Personnel Are Reasonable (GAO/NSIAD-98-93, Apr. 21,
             1998).

             Public-Private Competitions: DOD's Additional Support for Combining
             Depot Workloads Contains Weaknesses (GAO/NSIAD-98-143, Apr. 17,
             1998).

             Defense Depot Maintenance: DOD Shifting More Workload for New
             Weapon Systems to the Private Sector (GAO/NSIAD-98-8, Mar. 31, 1998).

             Public-Private Competitions: DOD's Determination to Combine Depot
             Workloads Is Not Adequately Supported (GAO/NSIAD-98-76, Jan. 20, 1998).

             Defense Depot Maintenance: Information on Public and Private Sector
             Workload Allocations (GAO/NSIAD-98-41, Jan. 20, 1998).

             Navy Regional Maintenance: Substantial Opportunities Exist to Build on
             Infrastructure Streamlining Progress (GAO/NSIAD-98-4, Nov. 13, 1997).

             Navy Depot Maintenance: Privatizing Louisville Operations in Place Is Not
             Cost-Effective (GAO/NSIAD-97-52, July 31, 1997).

             Defense Depot Maintenance: Challenges Facing DOD in Managing Working
             Capital Funds (GAO/T-NSIAD/AIMD-97-152, May 7, 1997).

             Defense Depot Maintenance: Uncertainties and Challenges DOD Faces in
             Restructuring Its Depot Maintenance Program (GAO/T-NSIAD-97-112,
             May 1,1997) and (GAO/T-NSIAD-97-111, Mar. 18, 1997).



             Page 43                                   GAO/NSIAD-99-199 Depot Maintenance
                    Related GAO Products




                    Defense Outsourcing: Challenges Facing DOD as It Attempts to Save
                    Billions in Infrastructure Costs (GAO/T-NSIAD-97-110, Mar. 12, 1997).

                    High-Risk Series: Defense Infrastructure (GAO/HR-97-7, Feb. 1997).




(709349)   Letter   Page 44                                   GAO/NSIAD-99-199 Depot Maintenance
Ordering Information

The first copy of each GAO report and testimony is free.
Additional copies are $2 each. Orders should be sent to the
following address, accompanied by a check or money order made
out to the Superintendent of Documents, when necessary, VISA and
MasterCard credit cards are accepted, also.

Orders for 100 or more copies to be mailed to a single address are
discounted 25 percent.

Orders by mail:

U.S. General Accounting Office
P.O. Box 37050
Washington, DC 20013

or visit:

Room 1100
700 4th St. NW (corner of 4th and G Sts. NW)
U.S. General Accounting Office
Washington, DC

Orders may also be placed by calling (202) 512-6000
or by using fax number (202) 512-6061, or TDD (202) 512-2537.

Each day, GAO issues a list of newly available reports and
testimony. To receive facsimile copies of the daily list or any list
from the past 30 days, please call (202) 512-6000 using a touchtone
phone. A recorded menu will provide information on how to obtain
these lists.

For information on how to access GAO reports on the INTERNET,
send an e-mail message with “info” in the body to:

info@www.gao.gov

or visit GAO’s World Wide Web Home Page at:

http://www.gao.gov
United States                       Bulk Rate
General Accounting Office      Postage & Fees Paid
Washington, D.C. 20548-0001           GAO
                                 Permit No. GI00
Official Business
Penalty for Private Use $300

Address Correction Requested