United States GAO General Accounting Office Washington, D.C. 20548 National Security and International AlTairs Division B-28245 1 April 15,1999 The Honorable Herbert H. Bateman Chairman, Subcommittee on Military Readiness Committee on Armed Services House of Representatives The Honorable Solomon P. Ortiz Ranking Minority Member, Subcommittee on Military Readiness Committee on Armed Services House of, Representatives Subject: Defense Transportation: Progress of MTMC Pilot This letter responds to your request for comments on the Military Traffic Management Command’s (MTMC) first interim progress report to Congress entitled “Current DOD Demonstration Program to Improve the Quality of Personal Property Shipments of the Armed Forces,” dated February 27,199g.l In the March 18,1999, hearing before your Subcommittee, we testified that methodological weaknesses in how data were collected and evaluated precluded us from validating all of the results of the Army’s Hunter personal property pilot.’ Concerned that similar weaknesses may exist with the Department of Defense (DOD) Demonstration Program (the MTMC pilot), you asked us to review the February 1999 progress report. Specifically, we focused on MTMC’s plan for evaluating the (1) pilot results and (2) the proposed industry alternative called the Commercial-Like Activities for Superior Quality (CLASS) proposal. The CLASS proposal will not be tested, but Congress directed that MTMC evaluate and report on the proposal. ‘Section 376 of the Strom Thurmond National Defense Authorization Act for Fiscal Year 1999 (p. L. 105-261)requires the Secretary of Defense to submit interim reports on the MTMC pilot no later than January 15 and April 15,1999, and a final report to Congress no later than August 31,1999. *Defense Transaortation: Efforts to Imorove DOD’s Personal Proaertv Program (GAO/T-NSLAD-99-106,Mar. l&1999). /~a-o% GAO/WUD-99-130R Progress of MTMC Pilot B-282451 RESULTS IN BRIEF The first interim progress report provides limited information on the status of the pilot program because it only recently began. The report generally describes the methodology for evaluating the program based on 10 goals that were mutually agreed upon by industry and MTMC in 1996 and the definition and measurement of each goal. Overall, this approach appears to have merit; however, we noted some weaknesses in the methodology that could diminish the quality and credibility of M.TMC’s evaluation. Absent from the current methodology is a delineation of roles and responsibtities of participating organizations and information on how the data will be collected, analyzed, and validated. MTMC also does not specify what will constitute success for the pilot, particularly, how much weight each goal will have in determinin g overall success. The interim progress report also does not provide any detail on how the CLASS proposal, that MTMC was directed by Congress to report on, will be evaluated. Such advance determinations would enhance the credibility of the results and avoid any perception of bias. In recent meetings, MTMC officials have indicated they are making changes to address these weaknesses. We are recommending that, as MTMC finalizes its methodology, it include information on how the data will be collected, validated, and analyzed and how success will be determined for both the pilot and the CLASS proposal. We encourage the MTMC Commander to seek expert methodological advice if a MTMC support contractor is not providing such assistance to ensure a robust, defensible methodology. MTMC PERSONAL PROPERTY PILOT The MTMC pilot is one of four pilots DOD has proposed or underway to improve its personal property program.3 In June 1994, the Deputy Commander in Chief of the U. S. Transportation Command (TRANSCOM) tasked MTMC with reengineering DOD’s personal property program. A year later, the House Committee on National Security4 directed DOD to initiate a pilot program that incorporated commercial business practices and standards in its personal property program. Concerned about the impact such a pilot would have on small businesses, the Committee directed the Secretary of Defense to establish a work@ group of military and industry representatives to develop a mutually agreeable program to pilot test. During the summer of 1996, the working group met and reached a consensus on many issues, including the pilot’s 10 goals. The working group could not agree on how the pilot should be implemented, consequently, the two sides presented separate proposals to Congress. The military’s proposal served as the framework for the MTMC pilot, while the transportation industry’s proposal evolved into CLASS. As stated previously, the CLASS proposal will not be tested, but MTMC plans to include the evaluation based on the likelihood that it might satisfy the 10 goals in the final report to Congress. ’ A description of these other pilot programs is included in our March 18,1999, testimony. aThe House Committee on National Security is now called the House Committee on Armed Services. Page 2 GAOLNSIAD-99-130RProgress of MTMC Pilot B-282451 The MTMC pilot, delayed by numerous’bid protests, was phased into operation over a 3-week period during January and February 1999. The pilot began in North Carolina on January 11, in South Carolina on January 18, and in Florida on February 1. The pilot includes 50 percent of the eligible moves originating in these three states to destinations elsewhere in the continental United States and to Europe; the remaining half will move under the existing MTMC system.’ The pilot is expected to encompass about 18,500 annual shipments and to run for a 3-year test period (1 year with two l-year options), which will end in December 2002. In August and September 1998, MTMC awarded contracts to 41 companies to acquire personal property services on the basis of best value rather than the lowest price. MTMC will evaluate company performance quarterly and compliance with the terms and conditions of the contracts continually. Performance reviews will be conducted based on customer satisfaction survey results and claims data Once companies receive their minimum guarantee of business, future awards will be placed with the best performers. UNCLEAR HOW MTMC WILL EVALUATE PILOT RESULTS AND INDUSTRY PROPOSAL MTMC submitted the first of three mandated reports on its personal property pilot program to Congress on February 27, 1999. The first progress report provides limited information on the status of the program because the pilot has just begun. Overall, MTMc’s methodology appears to have merit; however, we noted some weaknesses in the methodology that could diminish the quality and credibility of MTMC’s evaluation. While the report indicates that MTMC will evaluate the pilot based on the 10 goals, it is not clear how MTMC will assess the goals to reach definitive conclusions about the results of the pilot. Absent from the methodology is a delineation of roles and responsibilities of participating organizations and information on how the data will be collected, analyzed, and validated. For example, the report does not state whether the services’ audit agencies will validate the data. It is also not clear whether each goal will be assessed equally to determine the overall success of the pilot. For example, one of the pilot’s goals is to adopt corporate business practices that lead to world-class customer service. The interim progress report identifies six such business practices but does not describe how these practices were selected, how they will be measured and scored, and their relative weight to each other and to the remaining nine pilot goals. (See enclosure I for more details on our concerns for each of the goals.) An evaluation plan for the pilot separate from the interim report has not been developed. MTMC’s interim progress report also does not discuss in any detail how an independent contractor will evaluate the CLASS proposal and a plan for evaluatingthe proposal has not been developed. The report states that the independent contractor will rely on professional judgment to evaluate how well the proposal is likely to have met the 10 goals if it were ’ Some moves are excluded from the MTMC pilot such as moves from non-temporary storage and intrastate moves. Page 3 GAO&WAD-99-130R Progress of hiTMC Pilot B-28245 1 implemented. However, MTMC has not yet developed the criteria or basis for the independent contractor to make those judgments. We discussed our concerns with MTMC officials, and as a result, they plan to address them in their second interim progress report to Congress. In a related issue, TRANSCOM is responsible for overseeing all of the DOD personal property pilots, and in this capacity is developing an evaluation plan to ensure that the pilots use comparable evaluation data. TRANSCOM plans to evaluate the pilots and then recommend a follow-on course of action and time-lines for implementing a new personal property program throughout DOD. As we stated in our March 18,1999, testimony, TRANSCOM asked us to review and comment on its evaluation plan. In our comments, we encouraged TRANSCOM to seek expert methodological advice before finalizing its evaluation plan to enhance the quality of its assessment. CONCLUSIONS AND RECOMMENDATION While MTMC has developed an evaluation methodology that describes how it will evaluate. the pilot results, weaknesses remain. The methodology can be enhanced to ensure a credible and valid assessment. To address the methodology weaknesses, we recommend that the Commander, MTMC take steps to assure that, as the methodology is finalized, it include information on how the data will be collected, validated, and analyzed and how success will be determined for the pilot and likely success for the CLASS proposal. Because the pilot is entering its fourth month of operation, it is important that the evaluation methodology be completed in a timely manner. To ensure a robust, defensible methodology, we encourage the MTMC Commander to seek expert methodological advice if a MTMC support contractor is not already providing such assistance. AGENCY COMMENTS We discussed our observations with DOD officials and provided a draft of this letter for their review and comment. Although we did not obtain written comments, the officials stated that they generally agreed with the information discussed in the letter and concurred with the recommendation. As a result, MTMC is making a number of changes to its evaluation methodology with the assistance of its support contractor. These changes will be reflected in the second interim progress report to Congress. MTMC also provided technical comments that we incorporated as appropriate. To accomplish this work, we met with officials from MTMC and PriceWaterhouse Coopers- a MTMC support contractor-and reviewed key documents these officials provided us. We conducted our review in accordance with generally accepted government auditing standards. Page 4 GAO/NSI.AD-99-130RProgress of MTMC Pilot B-282451 We are sending copies of this report to the Secretary of Defense, the Commander in Chief, USTRANSCOM, and the Commander, MTMC. We will also make copies available to others upon request. Major contributors to this letter were Charles Patton, Jr., Nomi Taslitt, and Marc Schwartz. Please contact me at (202) 512-8412 if you or your staffhave any questions. Sincerely yours, David R. Warren, Director Defense Management Issues Page 5 GAO/NSIAD-99-130RProgress of MTMC Pilot ENCLOSURE I ENCLOSURE I OUR OBSERVATIONS ON THE PILOT’S 10 GOALS GOAL 1: PROVIDE QUALITY SERVICE The Military Traffic Management Command (MTMC) includes six best commercial practices under this goal. The practices are (1) full replacement value for loss and damage, (2) toll-free telephone numbers to allow service members an opportunity to contact contractors directly, (3) direct contact between service members and contractors for arranging personalized moves and settling claims, (4) inconvenience claims for service members when contractors do not honor accepted pickup or delivery times, (5) free storage for service members when shipments arrive ahead of schedule and are placed in storage by the contractor, and (6) faster payment of claims. It is unclear how these commercial practices will be evaluated by “considering their overall impact on customer satisfaction” through the customer survey or what data will be collected for each of the practices. While we recognize that many of these practices are not, measurable individually, MTMC does not state which of the practices will be measured quantitatively and/or qualitatively. In addition, the sixth practice, faster payment of claims, was one of the factors we could not validate in the Army Hunter pilot because start and end dates could not be established with certainty. GOALS 2 AND 3: IMPROVE ON-TIME PICKUP AND ON-TIME DEIJVERY The interim progress report does not state what constitutes “on-time” for pickup and delivery and how the data will be collected and validated. That is, the parameters are not specified on how late or early a pickup or delivery can be made and still constitute “on time” and who is . responsible for making the determination. GOAL 4: ACHIEVE HIGH CUSTOMER SATISFACTION IN RELATIONSHIP TO ENTIRE MOVE PROCESS Customer satisfaction will be evaluated based on pilot participants’ responses to two survey questions. These survey questions correspond with questions that were included in a MTMC 1996 customer satisfaction survey that provided the baseline measure for this goal’. However, the 1996 survey had less than a 20-percent response rate; thus, comparisons between it andthe current survey should be made cautiously. ’ These 10 goals appear in MTMc’s February 27,1999, interim progress report. ’ Customer satisfaction surveys were administered during April and May 1997based on moves that occurred between March and August 1996. Page 6 GAOLWAD-9943OR Progress of MTMC Pilot ENCLOSURE I ENCLOSURE I MTMC has set a pilot goal of 95 percent customer satisfaction to parallel the satisfaction level best commercial companies have experienced; however, like the other nine goals it is unclear how the results will be validated. If the pilot achieves a satisfaction rate of 92 percent-3 percent below the goal-does that indicate failure, even though this represents 21 percent above the baseline? GOAL 5: ADOPT CORPORATE BUSINESS PROCESSES THAT LEAD TO WORLD-CLASS CUSTOMER SERVICE Six business practices are included under this goal. The practices are (1) establish contracts with suppliers, (2) establish long-term relationships with suppliers, (3) use a smaller number of carriers to provide transportation services, (4) use commercial documentation practices to eliminate government unique documents, (5) increase the use of electronic commerce advances, and (6) adopt the growing trend of using relocation companies to perform personal property movement services. The interim progress report does not indicate how these business practices were selected, nor is there a discussion of how these six practices will be measured, evaluated, validated, and scored. GOAL 6: LOWER LOSS/DAMAGE AND LOWER CLAIMS FREQUENCY AND CLAIMS AVERAGES This goal consists of three subgoals: the percentage of shipments that incur damage (loss/damage frequency); the percentage of shipments that generate claims from service members (claims frequency); and the amount of money these claims represent (average claim size). Baseline values are cited for all three subgoals. To evaluate the benefits of the pilot on this goal, MTMC established performance standards for two subgoals, but not for loss/damage frequency. However, no information is provided about how data will be collected, validated, and scored, and the relative weighting of the subgoals. GOAL 7: SIMPLIFY THE SYSTEM. INCLUDING REDUCING ADMINISTRATIVE WORKLOAD According to the interim progress report, the pilot meets this goal if the administrative and operating costs are less than the $370 per shipment under the current system. The $370 figure was developed by a MTMC support contractor and is based on a 1995 study. According to MTMC officials, it has not been validated by a service audit agency. GOAL 8: ENSURE CAPACITY TO MEET NEEDS FOR QUALITY MOVES This goal focuses on how often carriers refuse to move shipments, and the possible effect carrier refusal has on on-time pickup of shipments. MTMC established a baseline refusal rate of 24 percent by excluding the non-peak months (October to April). However, using the 24 percent rate may be misleading since it does not represent the annual rate of refusal. MTMC may want to consider comparing this data by month to get a better indicator of the impact of Page 7 GAO/NSIAD-9%13ORProgress of MTMC Pilot ENCLOSURE I ENCLOSURE I seasonal periods on carrier’s refusal of shipments. In addition, MTMC does not describe how the data will be collected, analyzed, and validated. GOAL 9: PROVIDE OPPORTUNITY FOR SMALL BUSINESSES OFFERING QUALITY SERVICE TO COMPETE FOR BUSINESS AS A PRIME CONTRACTOR Opportunity for small business participation will be measured by comparing the number of shipments and the contract value of these shipments that are moved by small businesses under the pilot with the number and value moved under the current program (the baseline). However, it will be difficult to determine a baseline of small businesses because historically such data have been based on carrier/forwarder self-certification, which MTMC has considered questionable. GOAL 10: PROVIDE BEST VALUE MOVING SERVICES TO THE GOVERNMENT The interim progress report states that this goal will be evaluated by assessing the other nine goals to determine if quality of service has improved, customer satisfaction has increased, and administrative and operating costs have decreased, while considering the change in transportation costs. According to MTMC, the question is “are the [expected] increases in transportation costs reasonable compared to the improvements in quality of life for military members?” The interim progress report does not indicate how this question will be answered. While MTMC states a methodology to derive transportation costs by comparing what it would have cost to move the same shipments under the current system, it is not clear how efficiencies and quality of life improvements will be aggregated, measured, scored, validated, and compared with transportation costs. (709407) Page 8 GAO/i&AD-99-130R Progress of MTMC Pilot Ordering Information The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. VISA and Mastercard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent. Orders by mail: U.S. General Accounting Office P.O. Box 37050 Washington, DC 20013 or visit: Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by caRing (202) 512-6000 or by using fax number (202) 512-6061, or TDD (202) 512-2537. Each day, GAO issues a list of newly available reports and testimony. To receive facsimile copies of the daily list or any list from the past 30 days, please caB (202) 512-6000 using a touchtone phone. A recorded menu wiIl provide information on how to obtain these lists. For information on how to access GAO reports on the INTERNET, send an e-mail message with “info” in the body to: firstname.lastname@example.org or visit GAO’s World Wide Web Home Page at: http://www.gao.gov United States General Accounthg Office Washington, D.C. 20548-0001 Official Business Penalty for Private Use $300 Address Correction Requested
Defense Transportation: Progress of MTMC Pilot
Published by the Government Accountability Office on 1999-04-15.
Below is a raw (and likely hideous) rendition of the original report. (PDF)