United States GAO General Accounting Washington, Office D.C. 20548 -.. National Security and International Affairs Division B-284102 December 21, 1999 The Honorable Herbert H. Bateman Chairman The Honorable Solomon P. Ortiz Ranking Minority Member Subcommittee on Military Readiness Committee on Armed Services House of Representatives Subject: Defense Transportation: Preliminary Personal Property Pilot Results Are Inconclusive This report responds to your request for information on the Department of Defense’s report to Congress entitled DOD Demonstration Program to ImDrove the Qualitv of Personal Proper-@ ShiDments of Members of the Armed Forces, dated August 31, 1999. This is the Department’s second report on the progress of the demonstration (pilot) program. The pilot is intended to correct many of the problems in shipping personal property that exist with the current program, including poor service from movers, excessive loss and damage to servicemembers’ property, and high claims costs to the government. The Department of Defense has been concerned about the quality servicemembers receive from its nearly $3 billion annual program to transport, store, and manage the personal property of its personnel. In June 1995, the House Committee on National Security’ directed the Department to initiate a pilot program that incorporates commercial business practices and standards in its personal property program.’ Concerned about the pilot’s impact on small businesses, the Committee directed the Secretary of Defense to establish a working group of military and industry representatives to develop a mutually agreeable program to pilot test. During the summer of 1996, the working group reached a consensus on the pilot’s 10 goals. These included providing quality service, improving on-time pickup and delivery, and achieving high customer satisfaction. However, the working group could not agree on the approach the pilot test should take. Consequently, industry prepared an alternative plan that differed from the one favored by the Department in several ways. Differences included who could participate in the pilot, how shipping rates would be set, and how shipments would be allocated among participating moving companies. The Department’s proposal served as the ’ The House Committee on National Security is now called the House Committee on Armed Services. ’ H.Rept. 104-131. p. 164 (1995). GAOflrlSIAD-OO-.52R Defense Transportation framework for the Military Traffic Management Command’s pilot, while the transportation industry’s proposal evolved into a proposal referred to as Commercial-Like Activities for Superior Quality Demonstration Program. The Command’s pilot was phased into operation in North Carolina, South Carolina, and Florida over a 3-week period during January and February 1999. Industry’s proposal will not be tested, but Congress has directed that the Command evaluate and report on the likelihood of it satisfying the goals. As agreed with your offices, our objectives were to provide you with information on the (1) preliminary pilot results, (2) the Department’s assessment of the industry proposal, and (3) the Command’s plan for assessing the pilot’s results. RESVLTS IN BRIEF The preliminary pilot results are inconclusive, but do provide positive indications regarding the pilot program. The Department’s progress report included preliminary results that showed improvement over the current system associated with several of the pilot’s goals. For example, pilot participants indicated a higher level of satisfaction with their moves, and loss and damage claims that resulted from moves under the pilot were settled, on average, seven times faster. At this time, the Department of Defense does not have sufficient data to reach a final conclusion regarding the pilot results. The Department plans to submit a subsequent report during the first quarter of calendar year 2000. This report is expected to discuss whether the pilot is superior to the existing system. The Department completed its evaluation of the industry proposal and based on available information concluded that it was unlikely to achieve several of the pilot goals. These unmet goals are to provide quality service, adopt corporate business processes, simplify the system, and provide best value moving services to the government. Therefore, the Department plans no further evaluation. The methodology being used to evaluate the pilot program should provide a credible analytical basis for determining the success of the pilot. The Military Traffic Management Command’s evaluation plan addresses most of the methodological weaknesses we identified in our earlier report3 For example, the relative weight each of the goals has in determining overall pilot success and the roles and responsibilities of participating organizations are now incorporated in the evaluation plan. One remaining issue is to establish baseline transportation and process costs. According to Command officials, the baseline costs will be finalized, evaluated, and reported in a report to be issued during the first quarter of 2000. BACKGROUND Section 376 of the Strom Thurmond National Defense Authorization Act for Fiscal Year 1999 (P. L. 105261) required the Secretary of Defense to submit interim reports on the Military Traffic Management Command pilot no later than January 15 and April 15, 1999, and a final report to Congress no later than August 31, 1999. The Department issued its first interim ’ Defense Transoortation: Pronress of MTMC Pilot (GAO/NSIAD99-130R. Apr. 15. 1999). Page 2 GAOLWAD-00-BZR Defense Transportation B-284102 report on February 27, 1999. Due to a delay in program start-up, the Department did not issue a report on April 15 because there was insufficient data on which to prepare the report. The Department issued its next report on August 31, which consolidated the April 15 and August 31 reports. Although the August 31 report should, according to the 1999 Defense Authorization Act, include an evaluation of whether the pilot met its goals, the August report reflected only the then-current status of the pilot. This report did not provide the final results of the pilot program. The Department will submit a subsequent report during the first quarter of calendar year 2000, when additional data is expected to be available. During the summer of 1996, the working group reached a consensus on the pilot’s 10 goals (see fig. 1). However, the working group could not agree on how the pilot should be implemented; consequently, the Department and industry prepared separate proposals to Congress. We reviewed both proposals and determined that the Department’s proposal met the agreed upon goals to a greater extent.4 Because the industry proposal is not being tested, the Department retained a contractor to evaluate the extent that the proposal could achieve the 10 goals. The criteria for making this assessment were based on the professional judgment of the contractor and are described in documents we obtained from the Department and the contractor. Figure 1: The 10 Goals for the Personal Property Shipment Pilot @ii+ Provide quality service. @+ Improve on-time pickup. Goal 3 Improve on-time delivery. o+ Goal 4 Achieve high customer satisfaction. o+ Goal 5 Adopt corporate business processes. o+ @iii+ Lower loss and damage. @ii+ Simplify the system. Goal 8 Ensure capacity to meet the Department’s needs. o+ Goal 9 Provide opportunity for small businesses. o+ @iti+ Provide best value moving services to the government. Source: The Military Traffic Management Command. ’ Defense Transwrtation: Reennineeriw? the DOD Personal ROU&V RO~WI (GAOflrlSIAD97-49, NJJ. 27,1996). Page 3 GAO/NSIAD-W-52R Defense Transportation B-284102 The pilot was to include -50percent of the eligible moves originating in North Carolina South Carolina, and Florida to destinations elsewhere in the continental United States and to Europe; the remaining half were to move under the Command’s existing system.’ The pilot was expected to include about 18,500 annual shipments and run for a 3-year test period, which will end in January 2002. However, based on actual shipments to date, officials anticipate that approximately 9,000 shipments will occur under the pilot in the first year. In collaboration with defense transportation officers in the three participating states, Command officials are investigating the reasons that the number of shipments is lower than estimated. PRELIMINARY PILOT RESULTS INCONCLUSIVE BUT PROVIDE POSITIVE INDICATIONS The results reported in the Department’s August 1999 report were based on about 5,500 personal property shipments that were initiated during the first 6 months of the pilot. The results do not cover the entire May through September moving season when the majority of Defense moves are made, and therefore are insufficient to make an overall evaluation of the pilot. The preliminary results indicate improvements related to five of the pilot goals, -- Eleven percent of the shipments resulted in servicemembers’ filing loss and damage claims for which they received payment, and a claim was settled, on average, 17 days after it was filed with the contractor. In contrast, under the existing system, 35 percent of shipments result in servicemembers’ filing claims and settlement takes, on average, 126 days. (Goal l-provide quality service and goal 6-lower loss and damage.) -- Eighty-one percent of the servicemembers who completed the customer satisfaction survey after their moves indicated that they would use their move contractors again. If this satisfaction rate persists, it will exceed the 75percent rating under the current system that serves as the baseline. (Goal 4-achieve high customer satisfaction.) -- The frequency of contractors’ refusing offers to move shipments dropped from the current 24percent peak season rate to less than 2 percent, which included about half of the most recent May to September peak moving season. (Goal 8-ensure capacity to meet the Department’s needs.) -- Large businesses submitted subcontracting plans to allocate almost 60 percent of their shipments (as measured by contract value) to small businesses. If these plans are realized, the pilot’s goal of providing small businesses an opportunity to move more than 39 percent of all shipments would be satisfied. (Goal g-provide opportunity for small businesses.) ’ Some moves are excluded from the pilot such as moves from nontemporary storage and intrastate moves. Page 4 GAOi?4SIAJ3-OOdPR Defense Transportation B-284102 DEPARTMENT CONCLUDES THAT INDUSTRY PROPOSAL WOULD NOT ACHIEVE SEVERAL GOALS The Department completed its evaluation of the industry proposal! and based on available information concluded that it was unlikely to achieve several of the pilot goals. Therefore, the Department plans no further evaluation. Based on its review, the Department concluded that the proposal probably would not meet at least 4 of the 10 pilot goals: (1) provide quality service, (2) adopt corporate business processes, (3) simplify the system, and (4) provide best value moving services to the government. Further, the Department reported that since the pilot will not be tested, there is insufficient information to determine whether the industry proposal would meet the remaining six goals. To evaluate the industry proposal, the Department relied on the professional judgment of a contractor and information contained in two industry documents that described the proposal. We agree with the Department’s conclusion that, based on available information, the industry proposal is unlikely to satisfy at least three of the goals. Specifically, the proposal is unlikely to satisfy the goal to provide quality service because it would not offer servicemembers full replacement value for loss and damages that occur during their moves, nor would it provide free storage for shipments that arrive earlier than scheduled. The proposal also is not likely to satisfy the goals to adopt corporate business processes and simplify the system because it would not increase the use of electronic commerce or web-based applications, nor would it streamline the existing personal property movement process. Regarding the goal to provide best value moving services, while the Department has stated this is not likely to be achieved, we believe there is not enough information at this time to gauge whether it would be satisfied until the baseline transportation cost data is finalized. Moreover, we agree with the Department that there is not enough information to evaluate the proposal on the remaining six goals. EVALUATION PLAN CORRECTS METHODOLOGICAL WEAKNESSES The Command’s evaluation plan, dated September 30,1999, corrects most of the methodological weaknesses we identified in the first progress report, issued February 27, 1999, which discussed the evaluation strategy. Specifically, the evaluation plan discusses (1) the roles and responsibilities of participating organizations; (2) the way data will be collected, analyzed, and evaluated; and (3) the relative weight each goal has in determining the pilot’s success. The evaluation plan describes in sufficient detail what data will be used to evaluate pilot success for each goal and how data will be collected and analyzed. Baseline cost data that had been established for two goals are being reexamined for accuracy. According to Command officials, these cost data-average transportation and process costs to move a personal property shipment-will be finalized before the Department’s next report is issued in the first quarter of 2000. While the Department and industry did not establish priorities among the 10 goals, the Command described criteria in the evaluation plan that gave weight to each of the first 9 goals equally and that gave greater weight to the tenth goal (provide best value moving services to the government). If the pilot meets five or more of the first nine goals, and associated cost increases are deemed reasonable given the quality of improvements that have resulted, it will be considered superior to the existing program. The plan lists data sources Page 5 GAO/NSIAD-OO-SZR Defense Transportation B-284 lo” for each of the goals and identifies who is responsible for collecting and analyzing the pilot results. For example, the plan states that the Department of Defense Inspector General will assess the data collection process and the costing methodology. Obtaining complete and accurate baseline program cost data is critical to evaluating the pilot’s achievement of the seventh and tenth goals.” AGENCY COMMENTS We provided the Secretary of Defense a draft of this report on November 24, 1999, for review and comment. The Department did not provide written comments, but senior-level transportation policy and personal property officials of the Office of the Secretary of Defense and the Military Traffic Management Command provided oral comments on December 7, 1999. The officials agreed with the facts presented in the report. They stated that the next report on the pilot, which was scheduled for December 15, 1999, would be issued during the first quarter of 2000. This delay will allow the Department to finalize the baseline cost data and incorporate it into the pilot evaluation. Department officials also told us that the evaluation of the industry proposal is complete and that they plan no further evaluations. We revised our report to reflect this additional information. SCOPE AND METHODOLOGY We conducted our work and interviewed officials at headquarters, Miiitary Traffic Management Command, Falls Church, Virginia; Parsifal Corporation, Melbourne, Florida; and PriceWaterhouse Coopers, Fairfax, Virginia.. We also attended quarterly progress meetings between the Command and contractors participating in the pilot. To review the preliminary pilot results and verify the accuracy of key statements in the progress report, we obtained relevant documents from Command officials and met with officials from the Command and Parsifal Corporation, a support contractor responsible for compiling and reporting contractor performance data. Although some inaccuracies were noted, corrections were made and incorporated into our analysis. To review the Department’s assessment of the industry proposal, we obtained relevant documents and met with officials from the Command and PriceWaterhouse Coopers, a support contractor responsible for evaluating the industry proposal. To assess the Command’s methodology for evaluating the final pilot results and the industry proposal, we met with Command and PriceWaterhouse Coopers officials. We reviewed versions of the Command’s evaluation plan and interim progress report to ensure that they addressed methodological weaknesses we identified in our April 15, 1999 report to Congress. We conducted our review from September through November 1999 in accordance with generally accepted government auditing standards. Our most recent financial statement audit results highlighted the Department’s current systems’ inability to capture the full cost of its activities and programs (Deomment of Defense: Status of Financial Management Weaknesses and ActIons Needed to Correct Continuiw Challenges (GAO/T-AIMDNSIAD-99-171. May 4.1999). Page A GAONSIAD-OO-5PR Defense Transportation B-234102 We are sending copies of this report to the Honorable William S. Cohen, Secretary of Defense; General Charles T. Robertson, Jr., Commander in Chief of the U. S. Transportation Command; and Major General Kenneth L. Privratsky, Commander of the Military Traffic Management, Command. We will also make copies available to others upon request. Major contributors to this report were Charles Patton, Jr., Nomi Taslitt, and Marc Schwartz. Please contact me at (20‘2) 512-8412 if you or your staff have any questions. 2&d- Defense Management Issues (iO9413) Page i GAO/NSIAD-OO-62R Defense Transportation Ordering Information The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. VISA and Mastercard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent. Orders by mail: U.S. General Accounting Office P.O. Box 37050 Washington, DC 20013 or visit: Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calIing (202) 512-6000 or by using fax number (202) 512-6061, or TDD (202) 512-2537. Each day, GAO issues a list of newly available reports and testimony. To receive facsimile copies of the daily list or any list from the past 30 days, please caII (202) 512-6000 using a touchtone phone. A recorded menu will provide information on how to obtain these lists. For information on how to access GAO reports on the INTERNET, send an e-mail message with “info” in the body to: email@example.com or visit GAO’s World Wide Web Home Page at: httpti/www.gao.gov United States General Accounting Office Washington, D.C. 20548-0001 Offkial Business Penalty for Private Use $300 Address Correction Requested
Defense Transportation: Preliminary Personal Property Pilot Results Are Inconclusive
Published by the Government Accountability Office on 1999-12-21.
Below is a raw (and likely hideous) rendition of the original report. (PDF)