Defense Transportation: Plan Needed for Evaluating the Navy Personal Property Pilot

Published by the Government Accountability Office on 1999-06-23.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                   United States General Accounting Office

GAO                Report to Congressional Committees

June 1999

                   Plan Needed for
                   Evaluating the Navy
                   Personal Property

United States General Accounting Office                                                             National Security and
Washington, D.C. 20548                                                                       International Affairs Division

                                    B-282502                                                                                  Letter

                                    June 23, 1999

                                    Congressional Committees

                                    Since 1994, the Department of Defense (DOD) has been engaged in
                                    initiatives to reengineer the personal property program to simplify current
                                    processes, control program costs, ensure quality of service by adopting
                                    commercial business processes characteristic of world-class businesses,
                                    and relieve carriers of DOD-unique terms and conditions. We recently
                                    testified before the Military Readiness Subcommittee, House Armed
                                    Services Committee, on the results of the Army’s pilot and DOD’s plans to
                                    evaluate the other three pilots that are planned or under way.1 The
                                    statement of managers in the conference report on the 1997 DOD
                                    Appropriations Act conference report directed us to validate the results
                                    and savings achieved from any personal property pilot program before
                                    DOD proposes further expansion of such programs.2 In January 1998, the
                                    Navy began a servicemember arranged move pilot that allows
                                    servicemembers to select a carrier and arrange their own moves. This
                                    report provides status information on the Navy pilot and our assessment of
                                    plans for evaluating the pilot’s results.

Results in Brief                    Between January 1998 and March 1999, 223 servicemembers have used the
                                    pilot program to arrange their own move rather than use the current
                                    program. Participation has been relatively low compared to the other three
                                    pilots under way or planned, which involve substantially more shipments—
                                    approximately 1,400 to 45,000 shipments annually. Survey data indicate
                                    that participants are satisfied with the pilot and would use the option again.
                                    Because the pilot offers servicemembers a choice between the current
                                    program and arranging their own move, implementing the option increases
                                    the workload for local personal property officials.

                                    The member arranged move option is not featured in any of the other
                                    pilots, which are broader in scope and are intended to replace the current

                                    1 Defense Transportation: Efforts to Improve DOD’s Personal Property Program (GAO-T/NSIAD-99-106,
                                    Mar. 18, 1999).

                                    2 House Report   104-863 (Sept. 28, 1996) p. 865.

                                    Page 1                                              GAO/NSIAD-99-138 Defense Transportation

             program. While the U.S. Transportation Command is responsible for
             evaluating all of the pilots to determine which one could provide better
             long-term results, its plan for doing so has not been finalized. In addition,
             the U.S. Transportation Command’s draft evaluation plan proposes to
             collect information on the extent the Navy pilot works as an option within
             the current program at a few Navy sites, which may not provide adequate
             data to assess the Navy pilot’s feasibility or its compatibility with the other
             pilots’ results.

             Consistent with the recommendation in our report on the Army’s Hunter
             pilot (that DOD should develop a comprehensive strategy for testing each
             of the approaches), we are recommending that, in developing this strategy,
             the Secretary of Defense consider testing the Navy pilot and/or its unique
             features in conjunction with one or more of the other pilots.3

Background   The Navy pilot allows servicemembers the option of either participating in
             the current program or of selecting their own carrier from a list of
             approved small business carriers.4 Key features of the servicemember
             arranged move include shipment location information, direct claims
             settlement with the carrier, full replacement value for lost or damaged
             household goods, and payment via the government charge card. Initiated
             in January 1998, the pilot included only shipments originating at Puget
             Sound, Washington, headed to San Diego, California; Norfolk, Virginia; New
             London, Connecticut; Pensacola, Florida; and Jacksonville, Florida. To
             increase participation, the pilot’s origin site was expanded from Puget
             Sound (including Whidbey Island and Everett, Washington) to include San
             Diego, Norfolk, and New London in July 1998. However, due to potential
             competition with another pilot that will also serve the Florida area,
             Pensacola and Jacksonville remain as destination sites only. The pilot does
             not have a specific end date; however, DOD established a target in the
             November 1997 Defense Reform Initiative Report to offer the Navy option
             to every servicemember by January 1, 2000.

             The Navy pilot is one of four quality-of-life initiatives DOD is pursuing to
             change the way it is currently doing business, adopt commercial business

              Defense Transportation: Army’s Hunter Pilot Project Is Inconclusive but Provides Lessons Learned
             (GAO/NSIAD-99-129, June 23, 1999).

             4 Inthe motor freight and transportation industry, firms with annual gross revenues of $18.5 million or
             less are classified as small businesses. See Federal Acquisition Regulation 19.102.

             Page 2                                                GAO/NSIAD-99-138 Defense Transportation

                           practices, and achieve quality moving services for military families. The
                           other three pilots are described below.

                           • The Army’s pilot at Hunter Army Airfield, Savannah, Georgia, which was
                             initiated in February 1996, is a quality-of-life effort to improve the
                             relocation process and to test commercial practices in a military
                             environment. Services provided by the contractor include point-to-point
                             move management with a single point of contact for the member,
                             assistance in buying/selling a residence, full replacement value for lost
                             or damaged household goods, direct claims settlement with the
                             servicemember, and visibility of the shipment for approximately 1,400
                             annual moves. The contract will end on September 30, 1999.
                           • In January 1999, the Military Traffic Management Command (MTMC),
                             which manages the current personal property program, began a pilot
                             involving 50 percent of the moves in North Carolina, South Carolina, and
                             Florida, which will total approximately 18,500 moves annually. Key
                             features of the pilot include the selection of carriers based on
                             servicemember satisfaction and past performance rather than on price
                             alone; achieving stronger carrier commitment through long-term
                             contracts; offering full-value replacement protection and direct claims
                             settlement to users. The pilot is planned to run for 3 years, ending
                             September 2002.
                           • DOD announced on February 12, 1999, that it intended to begin the Full
                             Service Moving Project as a fourth test. This pilot would be similar to
                             the pilot at Hunter Army Airfield, with modifications based on the
                             Army’s lessons learned, and it would involve a larger number of moves
                             (approximately 45,000 annually). The pilot would be tested in the
                             National Capital (Washington, D.C.) Region, Georgia, and North Dakota.
                             Like the Army pilot, it is intended to replace the existing program by
                             using a contractor or contractors to provide both transportation and
                             move management services. No official start date has been set for this
                             pilot program.

Status of the Navy Pilot   As of March 1999, 223 servicemembers have participated in the pilot. In its
                           second year of operation, the pilot has been expanded from one to four
                           shipment sites. Program results thus far show (1) participation has been
                           relatively low compared to the other pilots that are under way or planned,
                           (2) servicemembers are satisfied arranging their own move, according to
                           limited survey data, and (3) workload would increase for personal property
                           officials because this option adds an alternative to the current program.

                           Page 3                                 GAO/NSIAD-99-138 Defense Transportation

Servicemember          As of March 26, 1999, a total of 223 members have participated in the pilot
Participation Is Low   and 132 moves have been completed. Participation has been low, in part,
                       because of the pilot’s short duration and eligibility restrictions. Although
                       the Puget Sound office began to offer this option in January 1998, the other
                       three sites did not offer it until late July 1998, and then only for Navy
                       military members moving to one of six destinations. The pilot’s eligibility
                       restrictions exclude civilians and members of the other services. Also, only
                       certain types of domestic household goods shipments are eligible. The pilot
                       excludes shipments of boats and mobile homes as well as shipments from
                       non-temporary storage or from a mini-warehouse. Shipments must weigh
                       at least 3,000 pounds and are expected to cost between $2,500 and $25,000.

                       As of the end of calendar year 1998, the participating personal property
                       offices reported interviewing 1,083 Navy members that were moving to or
                       from the six pilot sites to determine whether their shipments met pilot
                       eligibility criteria. Some 573, or 53 percent, of them had eligible shipments
                       and 133, or 23 percent, selected the option. Over 90 percent of those
                       determined as ineligible had household goods not weighing at least 3,000
                       pounds. Site officials reported that about 70 percent of eligible members
                       cited the effort involved in selecting a carrier as the primary reason that
                       they did not use the option (see table 1 for participant information by site).5

                       Table 1: Number of Participants by Site and Program Status, 1998

                                                       Puget Soundd        Norfolk     New London           San Diego        Total
                       Interviews                                  420         145                247              271       1,083
                       Eligible                                    149           93               115              216           573
                       Select pilot c                               78           15                19                21          133
                       Move completed                               54            5                  9               10           78
                       Move in-process                              24           10                10                11           55
                       a“Interviews”       includes only those servicemembers who planned to move to a participating site.
                           “Eligible” includes servicemembers who had shipments that met the pilot’s eligibility restrictions.
                       c “Select   pilot” includes servicemembers that participated in the pilot program.
                       dThese figures include shipments from the Whidbey Island Naval Air Station. Also, Puget Sound
                       began offering the option in January 1998, whereas the other sites began in July 1998.

                       5 AfterOctober 1998, pilot officials discontinued the requirement for sites to report the reasons for
                       ineligibility and for members not selecting the pilot because data showed the same dominant reasons.
                       However, sites still report the overall number of Navy members interviewed, eligible, and participating.

                       Page 4                                                    GAO/NSIAD-99-138 Defense Transportation

                                Navy officials have recently modified eligibility requirements to increase
                                participation in the pilot. The major changes would increase eligibility by
                                (1) reducing the weight minimum from 3,000 pounds to 1,000 pounds, (2)
                                allowing boats, and (3) eliminating current restrictions to allow shipments
                                to any domestic destination.

Limited Customer Survey         To measure customer satisfaction with the pilot, the Navy uses a
Results Indicate Satisfaction   nine-question customer survey that servicemembers are asked to fill out
                                and return after the move is completed. The survey includes questions
With Pilot
                                pertaining to pickup and delivery time, loss, damage, and overall
                                satisfaction. The survey asks, among other things, if the move was of a
                                better quality than the servicemembers’ prior move and if the customer
                                would (1) choose the pilot again, (2) recommend it to someone else, and
                                (3) use the same carrier again.

                                As of February 1999, the Navy had received 30 customer satisfaction
                                surveys. These surveys are predominantly from the Puget Sound personal
                                property office because Puget Sound was the first site to offer the pilot. Of
                                the 30 surveys, 23 indicate that the pilot was a better quality move than
                                other military moves. Eighteen of the 30 shipments (60 percent) had no
                                claims, and 12 had a claim. Of the 12 claims made, only 2 exceeded $500.
                                One of these claims was for $1,900 for broken china. Despite this damage,
                                the customer responded that this was a better quality move than prior ones
                                and stated that they would use the same carrier and the pilot again (see
                                table 2 for selected survey responses).6

                                Table 2: Summary of Survey Responses to Selected Questions

                                Question                                                   Yes               No         No difference
                                Better quality move?                                        23                 4                   1
                                Use pilot again?                                            27                 3
                                Any claims?                                                 12               18
                                Recommend the pilot?                                        27                 3
                                aThe numbers in this row do not add to 30 because two surveys did not contain the question regarding
                                whether the servicemember arranged move was a better quality move than other military moves.

                                6 The   carriers, not the government, settle the claims directly with servicemembers.

                                Page 5                                                  GAO/NSIAD-99-138 Defense Transportation

Increased Workload at       While the pilot offers advantages to the servicemember, it would add to the
Personal Property Offices   workload of the personal property officials who are responsible for the
                            bulk of day-to-day program management. Among their additional duties, or
                            duties that were previously handled by MTMC, are negotiating agreements
                            with participating small business carriers, providing individual counseling
                            to potential participants, maintaining up-to-date carrier information and
                            performance data, and tracking customer satisfaction survey results.
                            These are additional duties not previously handled by the personal
                            property office.

                            Local contracting officers at each participating installation enter into
                            agreements with companies that offer acceptable discounts off of
                            commercial rates. The agreements are entered into with companies that
                            are self-certified as small businesses and on MTMC’s approved list. These
                            agreements provide for the use of a government charge card for simplified
                            payment and for direct claims settlement with the carrier at full repair or
                            replacement value at no additional cost to the government. In addition,
                            carriers are required to provide information on a shipment’s location
                            through the use of a toll-free help line and a pager for direct delivery
                            notification, which are designed to improve service and reduce storage

                            The personal property office provides servicemembers with the names of
                            participating movers that have been determined to have reasonable prices.
                            The property offices also maintain carrier quality books that contain a
                            carrier’s performance history, the returned customer satisfaction surveys,
                            and the carrier’s marketing materials. Participating servicemembers are
                            required to contact at least three moving companies and document the
                            basis for their preference of one of the carriers. The contracting officer can
                            then make the selection considering both price and the servicemember’s
                            recommendation concerning non-price factors such as quality. The award
                            is made to the firm selected under the simplified acquisition procedures
                            contained in Federal Acquisition Regulation part 13.

Plan for Evaluating the     The Navy does not plan to conduct an evaluation of the pilot program
                            separate from its own evaluation. The U.S. Transportation Command
Pilot Is Incomplete         (USTRANSCOM) is currently in the process of developing and coordinating
                            an evaluation plan with the services to evaluate the personal property pilot
                            tests, and it will make a recommendation as to the follow-on course of
                            action. In this regard, the Navy has modified its customer survey questions

                            Page 6                                  GAO/NSIAD-99-138 Defense Transportation

              so that the survey questions match the USTRANSCOM survey questions.
              Navy officials are also accumulating pilot project transportation costs and
              developing a methodology to compare these costs with those that the
              government would have otherwise paid.

              Presently, the pilot has only been tested as an option to the current
              program at a few Navy sites and not as an option at the other pilot sites.
              The other pilots are designed to test an approach to replace the current
              program. However, the USTRANSCOM draft evaluation plan does not
              address the feasibility or potential benefits of incorporating the pilot option
              into the pilots that may replace the current program. Further, the
              evaluation plan does not directly assess the unique aspects of the pilot,
              which include limiting carrier selection to small business carriers and using
              the government charge card for payment. Consequently, DOD may not
              have the information it needs to craft a DOD-wide personal property

              Navy officials are concerned about several aspects of the USTRANSCOM
              evaluation plan. In March 1999, they stated that the data elements to be
              collected should be better defined, that a consistent evaluation time period
              should be established, and that expert advice should be sought. The
              officials also believe that cost comparisons of overhead will be difficult
              because the same personnel who administer the current program are
              implementing the Navy pilot. In our recent testimony and report on the
              Army’s Hunter pilot and in our comments on several draft evaluation plans,
              we have stated similar concerns about the current evaluation plan. These
              concerns include the need to (1) develop a comprehensive strategy for
              testing the unique characteristics and/or processes of each pilot and (2) use
              expert advice in finalizing a methodologically-sound evaluation plan.

Conclusions   The Navy pilot program differs from other ongoing or proposed pilot
              programs because it adds an option to the current program; it is not
              intended to replace the current program. The pilot participation levels and
              results thus far provide general information about the program’s potential
              benefits and customer satisfaction compared to the current program—
              which DOD is proposing to replace. Since the Navy pilot option is not to be
              integrated and tested with the other pilots, information on the viability of
              providing this option will only be available in comparison to the current
              program. Further, the draft evaluation plan does not identify a specific
              method for assessing the pilot’s unique features.

              Page 7                                   GAO/NSIAD-99-138 Defense Transportation

Recommendation    Consistent with the recommendation in our report on the Army’s Hunter
                  pilot that the Secretary of Defense develop a comprehensive strategy for
                  testing each of the approaches, we recommend that the Secretary of
                  Defense, in developing this strategy, consider testing the Navy pilot and/or
                  its unique features in conjunction with one or more of the other pilots.
                  Doing so would test the Navy pilot in an environment that is more
                  consistent with the changes being considered and likely to be

Agency Comments   DOD stated that it concurred with the report and its recommendations.
                  Specifically, DOD stated that, as part of its plan to develop a
                  comprehensive strategy for evaluating each of the pilots, it would
                  determine—in concert with the services—how best to incorporate the
                  features of the Navy pilot into the other pilots. DOD’s comments are
                  reprinted in appendix I.

Scope and         To determine how the Navy plans to implement and evaluate the pilot
                  program, we reviewed available program documents and met with program
Methodology       management officials at the Naval Supply Systems Command,
                  Mechanicsburg, Pennsylvania. We discussed how the pilot is being
                  implemented and managed, the contract terms and conditions, and the key
                  characteristics of the pilot. Further, we discussed and reviewed the
                  customer survey results and other data that will be provided to
                  USTRANSCOM for its evaluation of the pilot. We also discussed with Navy
                  Supply Systems Command and Naval Audit Service officials their concerns
                  with the USTRANSCOM evaluation plan.

                  We visited and interviewed officials at the Fleet and Industrial Supply
                  Center, Puget Sound, Bremerton, Washington, to determine how the pilot
                  was implemented at its first location. We discussed the process used to
                  establish and manage the pilot at Puget Sound while continuing to operate
                  the current system. Additionally, we discussed their experiences and
                  observations with the pilot. We also visited and interviewed program,
                  contracting, and government charge card officials at the Fleet and
                  Industrial Supply Center, Norfolk, Virginia, to determine how
                  implementation was progressing and to compare the pilot operations at
                  Puget Sound to those of Norfolk.

                  Page 8                                  GAO/NSIAD-99-138 Defense Transportation

To provide information on the pilot’s progress, we visited and interviewed
Naval Supply Systems Command officials to determine how sites were
selected for expansion after the pilot was established at Puget Sound.
Further, we attended a “lessons learned” conference that included personal
property officials from all of the pilot sites as well as from the Navy
Transportation Support Center, Bureau of Naval Personnel, and the
Defense Finance and Accounting Service to understand the problems these
organizations encountered with the pilot and possible solutions.

To obtain information on pilot participation, we reviewed weekly reports
summarizing the number of military members that were eligible and those
that were selected to participate. We also discussed the reasons for their
participation with site officials.

To provide information on available cost data, we reviewed the cost
comparisons developed by site officials that compare actual pilot
transportation costs to estimated/constructed costs for the same move
under the current system. We did not independently verify any data
reported by pilot sites or Naval Supply Systems Command headquarters.

Our review was conducted between October 1998 and April 1999 in
accordance with generally accepted government auditing standards.

We are sending copies of this report to the Honorable William S. Cohen,
Secretary of Defense; the Honorable Richard Danzig, Secretary of the Navy;
General Charles T. Robertson, Jr., Commander in Chief, USTRANSCOM;
Rear Admiral Donald E. Hickman, Commander, Naval Supply Systems
Command; Major General Mario F. Montero, Jr., Commander, MTMC; and
the Honorable Jacob J. Lew, Director, Office of Management and Budget.
We will also make copies available to others upon request.

Page 9                                 GAO/NSIAD-99-138 Defense Transportation

Please contact me at (202) 512-8412 if you or your staff have any questions
concerning this report. Major contributors to this report are listed in
appendix II.

David R. Warren, Director
Defense Management Issues

Page 10                                GAO/NSIAD-99-138 Defense Transportation

List of Congressional Committees

The Honorable John Warner
The Honorable Carl Levin
Ranking Minority Member
Committee on Armed Services
United States Senate

The Honorable Ted Stevens
The Honorable Daniel K. Inouye
Ranking Minority Member
Subcommittee on Defense
Committee on Appropriations
United States Senate

The Honorable Floyd Spence
The Honorable Ike Skelton
Ranking Minority Member
Committee on Armed Services
House of Representatives

The Honorable Jerry Lewis
The Honorable John P. Murtha
Ranking Minority Member
Subcommittee on Defense
Committee on Appropriations
House of Representatives

Page 11                            GAO/NSIAD-99-138 Defense Transportation

Letter                                                                                            1

Appendix I                                                                                       14
Comments From the
Department of Defense

Appendix II                                                                                      15
Major Contributors to
This Report

Tables                  Table 1: Number of Participants by Site and Program Status, 1998          4
                        Table 2: Summary of Survey Responses to Selected Questions                5


                        DOD        Department of Defense
                        MTMC       Military Traffic Management Command
                        USTRANSCOM U.S. Transportation Command

                        Page 12                              GAO/NSIAD-99-138 Defense Transportation
Page 13   GAO/NSIAD-99-138 Defense Transportation
Appendix I

Comments From the Department of Defense                          ApIenxdi

             Page 14       GAO/NSIAD-99-138 Defense Transportation
Appendix II

Major Contributors to This Report                                                                      ApIpx

National Security and   Charles I. Patton, Jr., Associate Director
                        Nomi R. Taslitt, Assistant Director
International Affairs   Robert L. Self, Senior Evaluator
Division, Washington,

Office of the General   John G. Brosnan, Assistant General Counsel

Counsel, Washington,

Norfolk Field Office    Daniel A. Omahen, Evaluator-in-Charge
                        John R. Beauchamp, Senior Evaluator

(709369)      Leetr     Page 15                                  GAO/NSIAD-99-138 Defense Transportation
Ordering Information

The first copy of each GAO report and testimony is free.
Additional copies are $2 each. Orders should be sent to the
following address, accompanied by a check or money order made
out to the Superintendent of Documents, when necessary, VISA and
MasterCard credit cards are accepted, also.

Orders for 100 or more copies to be mailed to a single address are
discounted 25 percent.

Orders by mail:

U.S. General Accounting Office
P.O. Box 37050
Washington, DC 20013

or visit:

Room 1100
700 4th St. NW (corner of 4th and G Sts. NW)
U.S. General Accounting Office
Washington, DC

Orders may also be placed by calling (202) 512-6000
or by using fax number (202) 512-6061, or TDD (202) 512-2537.

Each day, GAO issues a list of newly available reports and
testimony. To receive facsimile copies of the daily list or any list
from the past 30 days, please call (202) 512-6000 using a touchtone
phone. A recorded menu will provide information on how to obtain
these lists.

For information on how to access GAO reports on the INTERNET,
send an e-mail message with “info” in the body to:


or visit GAO’s World Wide Web Home Page at:

United States                       Bulk Rate
General Accounting Office      Postage & Fees Paid
Washington, D.C. 20548-0001           GAO
                                 Permit No. GI00
Official Business
Penalty for Private Use $300

Address Correction Requested