United States General Accounting Office GAO Report to Congressional Committees June 1999 DEFENSE TRANSPORTATION Plan Needed for Evaluating the Navy Personal Property Pilot GAO/NSIAD-99-138 United States General Accounting Office National Security and Washington, D.C. 20548 International Affairs Division B-282502 Letter June 23, 1999 Congressional Committees Since 1994, the Department of Defense (DOD) has been engaged in initiatives to reengineer the personal property program to simplify current processes, control program costs, ensure quality of service by adopting commercial business processes characteristic of world-class businesses, and relieve carriers of DOD-unique terms and conditions. We recently testified before the Military Readiness Subcommittee, House Armed Services Committee, on the results of the Army’s pilot and DOD’s plans to evaluate the other three pilots that are planned or under way.1 The statement of managers in the conference report on the 1997 DOD Appropriations Act conference report directed us to validate the results and savings achieved from any personal property pilot program before DOD proposes further expansion of such programs.2 In January 1998, the Navy began a servicemember arranged move pilot that allows servicemembers to select a carrier and arrange their own moves. This report provides status information on the Navy pilot and our assessment of plans for evaluating the pilot’s results. Results in Brief Between January 1998 and March 1999, 223 servicemembers have used the pilot program to arrange their own move rather than use the current program. Participation has been relatively low compared to the other three pilots under way or planned, which involve substantially more shipments— approximately 1,400 to 45,000 shipments annually. Survey data indicate that participants are satisfied with the pilot and would use the option again. Because the pilot offers servicemembers a choice between the current program and arranging their own move, implementing the option increases the workload for local personal property officials. The member arranged move option is not featured in any of the other pilots, which are broader in scope and are intended to replace the current 1 Defense Transportation: Efforts to Improve DOD’s Personal Property Program (GAO-T/NSIAD-99-106, Mar. 18, 1999). 2 House Report 104-863 (Sept. 28, 1996) p. 865. Page 1 GAO/NSIAD-99-138 Defense Transportation B-282502 program. While the U.S. Transportation Command is responsible for evaluating all of the pilots to determine which one could provide better long-term results, its plan for doing so has not been finalized. In addition, the U.S. Transportation Command’s draft evaluation plan proposes to collect information on the extent the Navy pilot works as an option within the current program at a few Navy sites, which may not provide adequate data to assess the Navy pilot’s feasibility or its compatibility with the other pilots’ results. Consistent with the recommendation in our report on the Army’s Hunter pilot (that DOD should develop a comprehensive strategy for testing each of the approaches), we are recommending that, in developing this strategy, the Secretary of Defense consider testing the Navy pilot and/or its unique features in conjunction with one or more of the other pilots.3 Background The Navy pilot allows servicemembers the option of either participating in the current program or of selecting their own carrier from a list of approved small business carriers.4 Key features of the servicemember arranged move include shipment location information, direct claims settlement with the carrier, full replacement value for lost or damaged household goods, and payment via the government charge card. Initiated in January 1998, the pilot included only shipments originating at Puget Sound, Washington, headed to San Diego, California; Norfolk, Virginia; New London, Connecticut; Pensacola, Florida; and Jacksonville, Florida. To increase participation, the pilot’s origin site was expanded from Puget Sound (including Whidbey Island and Everett, Washington) to include San Diego, Norfolk, and New London in July 1998. However, due to potential competition with another pilot that will also serve the Florida area, Pensacola and Jacksonville remain as destination sites only. The pilot does not have a specific end date; however, DOD established a target in the November 1997 Defense Reform Initiative Report to offer the Navy option to every servicemember by January 1, 2000. The Navy pilot is one of four quality-of-life initiatives DOD is pursuing to change the way it is currently doing business, adopt commercial business 3 Defense Transportation: Army’s Hunter Pilot Project Is Inconclusive but Provides Lessons Learned (GAO/NSIAD-99-129, June 23, 1999). 4 Inthe motor freight and transportation industry, firms with annual gross revenues of $18.5 million or less are classified as small businesses. See Federal Acquisition Regulation 19.102. Page 2 GAO/NSIAD-99-138 Defense Transportation B-282502 practices, and achieve quality moving services for military families. The other three pilots are described below. • The Army’s pilot at Hunter Army Airfield, Savannah, Georgia, which was initiated in February 1996, is a quality-of-life effort to improve the relocation process and to test commercial practices in a military environment. Services provided by the contractor include point-to-point move management with a single point of contact for the member, assistance in buying/selling a residence, full replacement value for lost or damaged household goods, direct claims settlement with the servicemember, and visibility of the shipment for approximately 1,400 annual moves. The contract will end on September 30, 1999. • In January 1999, the Military Traffic Management Command (MTMC), which manages the current personal property program, began a pilot involving 50 percent of the moves in North Carolina, South Carolina, and Florida, which will total approximately 18,500 moves annually. Key features of the pilot include the selection of carriers based on servicemember satisfaction and past performance rather than on price alone; achieving stronger carrier commitment through long-term contracts; offering full-value replacement protection and direct claims settlement to users. The pilot is planned to run for 3 years, ending September 2002. • DOD announced on February 12, 1999, that it intended to begin the Full Service Moving Project as a fourth test. This pilot would be similar to the pilot at Hunter Army Airfield, with modifications based on the Army’s lessons learned, and it would involve a larger number of moves (approximately 45,000 annually). The pilot would be tested in the National Capital (Washington, D.C.) Region, Georgia, and North Dakota. Like the Army pilot, it is intended to replace the existing program by using a contractor or contractors to provide both transportation and move management services. No official start date has been set for this pilot program. Status of the Navy Pilot As of March 1999, 223 servicemembers have participated in the pilot. In its second year of operation, the pilot has been expanded from one to four shipment sites. Program results thus far show (1) participation has been relatively low compared to the other pilots that are under way or planned, (2) servicemembers are satisfied arranging their own move, according to limited survey data, and (3) workload would increase for personal property officials because this option adds an alternative to the current program. Page 3 GAO/NSIAD-99-138 Defense Transportation B-282502 Servicemember As of March 26, 1999, a total of 223 members have participated in the pilot Participation Is Low and 132 moves have been completed. Participation has been low, in part, because of the pilot’s short duration and eligibility restrictions. Although the Puget Sound office began to offer this option in January 1998, the other three sites did not offer it until late July 1998, and then only for Navy military members moving to one of six destinations. The pilot’s eligibility restrictions exclude civilians and members of the other services. Also, only certain types of domestic household goods shipments are eligible. The pilot excludes shipments of boats and mobile homes as well as shipments from non-temporary storage or from a mini-warehouse. Shipments must weigh at least 3,000 pounds and are expected to cost between $2,500 and $25,000. As of the end of calendar year 1998, the participating personal property offices reported interviewing 1,083 Navy members that were moving to or from the six pilot sites to determine whether their shipments met pilot eligibility criteria. Some 573, or 53 percent, of them had eligible shipments and 133, or 23 percent, selected the option. Over 90 percent of those determined as ineligible had household goods not weighing at least 3,000 pounds. Site officials reported that about 70 percent of eligible members cited the effort involved in selecting a carrier as the primary reason that they did not use the option (see table 1 for participant information by site).5 Table 1: Number of Participants by Site and Program Status, 1998 Puget Soundd Norfolk New London San Diego Total a Interviews 420 145 247 271 1,083 b Eligible 149 93 115 216 573 Select pilot c 78 15 19 21 133 Move completed 54 5 9 10 78 Move in-process 24 10 10 11 55 a“Interviews” includes only those servicemembers who planned to move to a participating site. b “Eligible” includes servicemembers who had shipments that met the pilot’s eligibility restrictions. c “Select pilot” includes servicemembers that participated in the pilot program. dThese figures include shipments from the Whidbey Island Naval Air Station. Also, Puget Sound began offering the option in January 1998, whereas the other sites began in July 1998. 5 AfterOctober 1998, pilot officials discontinued the requirement for sites to report the reasons for ineligibility and for members not selecting the pilot because data showed the same dominant reasons. However, sites still report the overall number of Navy members interviewed, eligible, and participating. Page 4 GAO/NSIAD-99-138 Defense Transportation B-282502 Navy officials have recently modified eligibility requirements to increase participation in the pilot. The major changes would increase eligibility by (1) reducing the weight minimum from 3,000 pounds to 1,000 pounds, (2) allowing boats, and (3) eliminating current restrictions to allow shipments to any domestic destination. Limited Customer Survey To measure customer satisfaction with the pilot, the Navy uses a Results Indicate Satisfaction nine-question customer survey that servicemembers are asked to fill out and return after the move is completed. The survey includes questions With Pilot pertaining to pickup and delivery time, loss, damage, and overall satisfaction. The survey asks, among other things, if the move was of a better quality than the servicemembers’ prior move and if the customer would (1) choose the pilot again, (2) recommend it to someone else, and (3) use the same carrier again. As of February 1999, the Navy had received 30 customer satisfaction surveys. These surveys are predominantly from the Puget Sound personal property office because Puget Sound was the first site to offer the pilot. Of the 30 surveys, 23 indicate that the pilot was a better quality move than other military moves. Eighteen of the 30 shipments (60 percent) had no claims, and 12 had a claim. Of the 12 claims made, only 2 exceeded $500. One of these claims was for $1,900 for broken china. Despite this damage, the customer responded that this was a better quality move than prior ones and stated that they would use the same carrier and the pilot again (see table 2 for selected survey responses).6 Table 2: Summary of Survey Responses to Selected Questions Question Yes No No difference a Better quality move? 23 4 1 Use pilot again? 27 3 Any claims? 12 18 Recommend the pilot? 27 3 aThe numbers in this row do not add to 30 because two surveys did not contain the question regarding whether the servicemember arranged move was a better quality move than other military moves. 6 The carriers, not the government, settle the claims directly with servicemembers. Page 5 GAO/NSIAD-99-138 Defense Transportation B-282502 Increased Workload at While the pilot offers advantages to the servicemember, it would add to the Personal Property Offices workload of the personal property officials who are responsible for the bulk of day-to-day program management. Among their additional duties, or duties that were previously handled by MTMC, are negotiating agreements with participating small business carriers, providing individual counseling to potential participants, maintaining up-to-date carrier information and performance data, and tracking customer satisfaction survey results. These are additional duties not previously handled by the personal property office. Local contracting officers at each participating installation enter into agreements with companies that offer acceptable discounts off of commercial rates. The agreements are entered into with companies that are self-certified as small businesses and on MTMC’s approved list. These agreements provide for the use of a government charge card for simplified payment and for direct claims settlement with the carrier at full repair or replacement value at no additional cost to the government. In addition, carriers are required to provide information on a shipment’s location through the use of a toll-free help line and a pager for direct delivery notification, which are designed to improve service and reduce storage costs. The personal property office provides servicemembers with the names of participating movers that have been determined to have reasonable prices. The property offices also maintain carrier quality books that contain a carrier’s performance history, the returned customer satisfaction surveys, and the carrier’s marketing materials. Participating servicemembers are required to contact at least three moving companies and document the basis for their preference of one of the carriers. The contracting officer can then make the selection considering both price and the servicemember’s recommendation concerning non-price factors such as quality. The award is made to the firm selected under the simplified acquisition procedures contained in Federal Acquisition Regulation part 13. Plan for Evaluating the The Navy does not plan to conduct an evaluation of the pilot program separate from its own evaluation. The U.S. Transportation Command Pilot Is Incomplete (USTRANSCOM) is currently in the process of developing and coordinating an evaluation plan with the services to evaluate the personal property pilot tests, and it will make a recommendation as to the follow-on course of action. In this regard, the Navy has modified its customer survey questions Page 6 GAO/NSIAD-99-138 Defense Transportation B-282502 so that the survey questions match the USTRANSCOM survey questions. Navy officials are also accumulating pilot project transportation costs and developing a methodology to compare these costs with those that the government would have otherwise paid. Presently, the pilot has only been tested as an option to the current program at a few Navy sites and not as an option at the other pilot sites. The other pilots are designed to test an approach to replace the current program. However, the USTRANSCOM draft evaluation plan does not address the feasibility or potential benefits of incorporating the pilot option into the pilots that may replace the current program. Further, the evaluation plan does not directly assess the unique aspects of the pilot, which include limiting carrier selection to small business carriers and using the government charge card for payment. Consequently, DOD may not have the information it needs to craft a DOD-wide personal property program. Navy officials are concerned about several aspects of the USTRANSCOM evaluation plan. In March 1999, they stated that the data elements to be collected should be better defined, that a consistent evaluation time period should be established, and that expert advice should be sought. The officials also believe that cost comparisons of overhead will be difficult because the same personnel who administer the current program are implementing the Navy pilot. In our recent testimony and report on the Army’s Hunter pilot and in our comments on several draft evaluation plans, we have stated similar concerns about the current evaluation plan. These concerns include the need to (1) develop a comprehensive strategy for testing the unique characteristics and/or processes of each pilot and (2) use expert advice in finalizing a methodologically-sound evaluation plan. Conclusions The Navy pilot program differs from other ongoing or proposed pilot programs because it adds an option to the current program; it is not intended to replace the current program. The pilot participation levels and results thus far provide general information about the program’s potential benefits and customer satisfaction compared to the current program— which DOD is proposing to replace. Since the Navy pilot option is not to be integrated and tested with the other pilots, information on the viability of providing this option will only be available in comparison to the current program. Further, the draft evaluation plan does not identify a specific method for assessing the pilot’s unique features. Page 7 GAO/NSIAD-99-138 Defense Transportation B-282502 Recommendation Consistent with the recommendation in our report on the Army’s Hunter pilot that the Secretary of Defense develop a comprehensive strategy for testing each of the approaches, we recommend that the Secretary of Defense, in developing this strategy, consider testing the Navy pilot and/or its unique features in conjunction with one or more of the other pilots. Doing so would test the Navy pilot in an environment that is more consistent with the changes being considered and likely to be implemented. Agency Comments DOD stated that it concurred with the report and its recommendations. Specifically, DOD stated that, as part of its plan to develop a comprehensive strategy for evaluating each of the pilots, it would determine—in concert with the services—how best to incorporate the features of the Navy pilot into the other pilots. DOD’s comments are reprinted in appendix I. Scope and To determine how the Navy plans to implement and evaluate the pilot program, we reviewed available program documents and met with program Methodology management officials at the Naval Supply Systems Command, Mechanicsburg, Pennsylvania. We discussed how the pilot is being implemented and managed, the contract terms and conditions, and the key characteristics of the pilot. Further, we discussed and reviewed the customer survey results and other data that will be provided to USTRANSCOM for its evaluation of the pilot. We also discussed with Navy Supply Systems Command and Naval Audit Service officials their concerns with the USTRANSCOM evaluation plan. We visited and interviewed officials at the Fleet and Industrial Supply Center, Puget Sound, Bremerton, Washington, to determine how the pilot was implemented at its first location. We discussed the process used to establish and manage the pilot at Puget Sound while continuing to operate the current system. Additionally, we discussed their experiences and observations with the pilot. We also visited and interviewed program, contracting, and government charge card officials at the Fleet and Industrial Supply Center, Norfolk, Virginia, to determine how implementation was progressing and to compare the pilot operations at Puget Sound to those of Norfolk. Page 8 GAO/NSIAD-99-138 Defense Transportation B-282502 To provide information on the pilot’s progress, we visited and interviewed Naval Supply Systems Command officials to determine how sites were selected for expansion after the pilot was established at Puget Sound. Further, we attended a “lessons learned” conference that included personal property officials from all of the pilot sites as well as from the Navy Transportation Support Center, Bureau of Naval Personnel, and the Defense Finance and Accounting Service to understand the problems these organizations encountered with the pilot and possible solutions. To obtain information on pilot participation, we reviewed weekly reports summarizing the number of military members that were eligible and those that were selected to participate. We also discussed the reasons for their participation with site officials. To provide information on available cost data, we reviewed the cost comparisons developed by site officials that compare actual pilot transportation costs to estimated/constructed costs for the same move under the current system. We did not independently verify any data reported by pilot sites or Naval Supply Systems Command headquarters. Our review was conducted between October 1998 and April 1999 in accordance with generally accepted government auditing standards. We are sending copies of this report to the Honorable William S. Cohen, Secretary of Defense; the Honorable Richard Danzig, Secretary of the Navy; General Charles T. Robertson, Jr., Commander in Chief, USTRANSCOM; Rear Admiral Donald E. Hickman, Commander, Naval Supply Systems Command; Major General Mario F. Montero, Jr., Commander, MTMC; and the Honorable Jacob J. Lew, Director, Office of Management and Budget. We will also make copies available to others upon request. Page 9 GAO/NSIAD-99-138 Defense Transportation B-282502 Please contact me at (202) 512-8412 if you or your staff have any questions concerning this report. Major contributors to this report are listed in appendix II. David R. Warren, Director Defense Management Issues Page 10 GAO/NSIAD-99-138 Defense Transportation B-282502 List of Congressional Committees The Honorable John Warner Chairman The Honorable Carl Levin Ranking Minority Member Committee on Armed Services United States Senate The Honorable Ted Stevens Chairman The Honorable Daniel K. Inouye Ranking Minority Member Subcommittee on Defense Committee on Appropriations United States Senate The Honorable Floyd Spence Chairman The Honorable Ike Skelton Ranking Minority Member Committee on Armed Services House of Representatives The Honorable Jerry Lewis Chairman The Honorable John P. Murtha Ranking Minority Member Subcommittee on Defense Committee on Appropriations House of Representatives Page 11 GAO/NSIAD-99-138 Defense Transportation Contents Letter 1 Appendix I 14 Comments From the Department of Defense Appendix II 15 Major Contributors to This Report Tables Table 1: Number of Participants by Site and Program Status, 1998 4 Table 2: Summary of Survey Responses to Selected Questions 5 Abbreviations DOD Department of Defense MTMC Military Traffic Management Command USTRANSCOM U.S. Transportation Command Page 12 GAO/NSIAD-99-138 Defense Transportation Page 13 GAO/NSIAD-99-138 Defense Transportation Appendix I Comments From the Department of Defense ApIenxdi Page 14 GAO/NSIAD-99-138 Defense Transportation Appendix II Major Contributors to This Report ApIpx Ien di National Security and Charles I. Patton, Jr., Associate Director Nomi R. Taslitt, Assistant Director International Affairs Robert L. Self, Senior Evaluator Division, Washington, D.C. Office of the General John G. Brosnan, Assistant General Counsel Counsel, Washington, D.C. Norfolk Field Office Daniel A. Omahen, Evaluator-in-Charge John R. Beauchamp, Senior Evaluator (709369) Leetr Page 15 GAO/NSIAD-99-138 Defense Transportation Ordering Information The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary, VISA and MasterCard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent. Orders by mail: U.S. General Accounting Office P.O. Box 37050 Washington, DC 20013 or visit: Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000 or by using fax number (202) 512-6061, or TDD (202) 512-2537. Each day, GAO issues a list of newly available reports and testimony. To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone. A recorded menu will provide information on how to obtain these lists. For information on how to access GAO reports on the INTERNET, send an e-mail message with “info” in the body to: email@example.com or visit GAO’s World Wide Web Home Page at: http://www.gao.gov United States Bulk Rate General Accounting Office Postage & Fees Paid Washington, D.C. 20548-0001 GAO Permit No. GI00 Official Business Penalty for Private Use $300 Address Correction Requested
Defense Transportation: Plan Needed for Evaluating the Navy Personal Property Pilot
Published by the Government Accountability Office on 1999-06-23.
Below is a raw (and likely hideous) rendition of the original report. (PDF)