Defense Transportation: Progress of MTMC Pilot

Published by the Government Accountability Office on 1999-04-15.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

      United States

GAO   General Accounting Office
      Washington, D.C. 20548

      National Security and
      International AlTairs Division

      B-28245 1

      April 15,1999

      The Honorable Herbert H. Bateman
      Chairman, Subcommittee on
      Military Readiness
      Committee on Armed Services
      House of Representatives

      The Honorable Solomon P. Ortiz
      Ranking Minority Member,
      Subcommittee on Military Readiness
      Committee on Armed Services
      House of, Representatives

      Subject: Defense Transportation: Progress of MTMC Pilot

      This letter responds to your request for comments on the Military Traffic Management
      Command’s (MTMC) first interim progress report to Congress entitled “Current DOD
      Demonstration Program to Improve the Quality of Personal Property Shipments of the Armed
      Forces,” dated February 27,199g.l In the March 18,1999, hearing before your Subcommittee,
      we testified that methodological weaknesses in how data were collected and evaluated
      precluded us from validating all of the results of the Army’s Hunter personal property pilot.’
      Concerned that similar weaknesses may exist with the Department of Defense (DOD)
      Demonstration Program (the MTMC pilot), you asked us to review the February 1999
      progress report. Specifically, we focused on MTMC’s plan for evaluating the (1) pilot results
      and (2) the proposed industry alternative called the Commercial-Like Activities for Superior
      Quality (CLASS) proposal. The CLASS proposal will not be tested, but Congress directed that
      MTMC evaluate and report on the proposal.

      ‘Section 376 of the Strom Thurmond National Defense Authorization Act for Fiscal Year 1999 (p. L. 105-261)requires the
      Secretary of Defense to submit interim reports on the MTMC pilot no later than January 15 and April 15,1999, and a final report
      to Congress no later than August 31,1999.
      *Defense Transaortation: Efforts to Imorove DOD’s Personal Proaertv Program (GAO/T-NSLAD-99-106,Mar. l&1999).

                                                      /~a-o%                             GAO/WUD-99-130R Progress of MTMC Pilot


The first interim progress report provides limited information on the status of the pilot
program because it only recently began. The report generally describes the methodology for
evaluating the program based on 10 goals that were mutually agreed upon by industry and
MTMC in 1996 and the definition and measurement of each goal. Overall, this approach
appears to have merit; however, we noted some weaknesses in the methodology that could
diminish the quality and credibility of M.TMC’s evaluation.

Absent from the current methodology is a delineation of roles and responsibtities of
participating organizations and information on how the data will be collected, analyzed, and
validated. MTMC also does not specify what will constitute success for the pilot, particularly,
how much weight each goal will have in determinin g overall success. The interim progress
report also does not provide any detail on how the CLASS proposal, that MTMC was directed
by Congress to report on, will be evaluated. Such advance determinations would enhance the
credibility of the results and avoid any perception of bias. In recent meetings, MTMC officials
have indicated they are making changes to address these weaknesses.

We are recommending that, as MTMC finalizes its methodology, it include information on
how the data will be collected, validated, and analyzed and how success will be determined
for both the pilot and the CLASS proposal. We encourage the MTMC Commander to seek
expert methodological advice if a MTMC support contractor is not providing such assistance
to ensure a robust, defensible methodology.


The MTMC pilot is one of four pilots DOD has proposed or underway to improve its personal
property program.3 In June 1994, the Deputy Commander in Chief of the U. S. Transportation
Command (TRANSCOM) tasked MTMC with reengineering DOD’s personal property
program. A year later, the House Committee on National Security4 directed DOD to initiate a
pilot program that incorporated commercial business practices and standards in its personal
property program. Concerned about the impact such a pilot would have on small businesses,
the Committee directed the Secretary of Defense to establish a work@ group of military and
industry representatives to develop a mutually agreeable program to pilot test.

During the summer of 1996, the working group met and reached a consensus on many issues,
including the pilot’s 10 goals. The working group could not agree on how the pilot should be
implemented, consequently, the two sides presented separate proposals to Congress. The
military’s proposal served as the framework for the MTMC pilot, while the transportation
industry’s proposal evolved into CLASS. As stated previously, the CLASS proposal will not be
tested, but MTMC plans to include the evaluation based on the likelihood that it might satisfy
the 10 goals in the final report to Congress.

’ A description of these other pilot programs is included in our March 18,1999, testimony.
aThe House Committee on National Security is now called the House Committee on Armed Services.

 Page 2                                                                             GAOLNSIAD-99-130RProgress of MTMC Pilot

The MTMC pilot, delayed by numerous’bid protests, was phased into operation over a 3-week
period during January and February 1999. The pilot began in North Carolina on January 11,
in South Carolina on January 18, and in Florida on February 1. The pilot includes 50 percent
of the eligible moves originating in these three states to destinations elsewhere in the
continental United States and to Europe; the remaining half will move under the existing
MTMC system.’ The pilot is expected to encompass about 18,500 annual shipments and to
run for a 3-year test period (1 year with two l-year options), which will end in December

In August and September 1998, MTMC awarded contracts to 41 companies to acquire
personal property services on the basis of best value rather than the lowest price. MTMC will
evaluate company performance quarterly and compliance with the terms and conditions of
the contracts continually. Performance reviews will be conducted based on customer
satisfaction survey results and claims data Once companies receive their minimum
guarantee of business, future awards will be placed with the best performers.


MTMC submitted the first of three mandated reports on its personal property pilot program
to Congress on February 27, 1999. The first progress report provides limited information on
the status of the program because the pilot has just begun. Overall, MTMc’s methodology
appears to have merit; however, we noted some weaknesses in the methodology that could
diminish the quality and credibility of MTMC’s evaluation. While the report indicates that
MTMC will evaluate the pilot based on the 10 goals, it is not clear how MTMC will assess the
goals to reach definitive conclusions about the results of the pilot.

Absent from the methodology is a delineation of roles and responsibilities of participating
organizations and information on how the data will be collected, analyzed, and validated. For
example, the report does not state whether the services’ audit agencies will validate the data.
It is also not clear whether each goal will be assessed equally to determine the overall
success of the pilot. For example, one of the pilot’s goals is to adopt corporate business
practices that lead to world-class customer service. The interim progress report identifies six
such business practices but does not describe how these practices were selected, how they
will be measured and scored, and their relative weight to each other and to the remaining
nine pilot goals. (See enclosure I for more details on our concerns for each of the goals.) An
evaluation plan for the pilot separate from the interim report has not been developed.

 MTMC’s interim progress report also does not discuss in any detail how an independent
 contractor will evaluate the CLASS proposal and a plan for evaluatingthe proposal has not
been developed. The report states that the independent contractor will rely on professional
judgment to evaluate how well the proposal is likely to have met the 10 goals if it were

’ Some moves are excluded from the MTMC pilot such as moves from non-temporary storage and intrastate moves.

Page 3                                                                        GAO&WAD-99-130R Progress of hiTMC Pilot
B-28245 1

implemented. However, MTMC has not yet developed the criteria or basis for the
independent contractor to make those judgments.

We discussed our concerns with MTMC officials, and as a result, they plan to address them in
their second interim progress report to Congress.

In a related issue, TRANSCOM is responsible for overseeing all of the DOD personal property
pilots, and in this capacity is developing an evaluation plan to ensure that the pilots use
comparable evaluation data. TRANSCOM plans to evaluate the pilots and then recommend a
follow-on course of action and time-lines for implementing a new personal property program
throughout DOD. As we stated in our March 18,1999, testimony, TRANSCOM asked us to
review and comment on its evaluation plan. In our comments, we encouraged TRANSCOM to
seek expert methodological advice before finalizing its evaluation plan to enhance the quality
of its assessment.


While MTMC has developed an evaluation methodology that describes how it will evaluate.
the pilot results, weaknesses remain. The methodology can be enhanced to ensure a credible
and valid assessment.

To address the methodology weaknesses, we recommend that the Commander, MTMC take
steps to assure that, as the methodology is finalized, it include information on how the data
will be collected, validated, and analyzed and how success will be determined for the pilot
and likely success for the CLASS proposal. Because the pilot is entering its fourth month of
operation, it is important that the evaluation methodology be completed in a timely manner.
To ensure a robust, defensible methodology, we encourage the MTMC Commander to seek
expert methodological advice if a MTMC support contractor is not already providing such


We discussed our observations with DOD officials and provided a draft of this letter for their
review and comment. Although we did not obtain written comments, the officials stated that
they generally agreed with the information discussed in the letter and concurred with the
recommendation. As a result, MTMC is making a number of changes to its evaluation
methodology with the assistance of its support contractor. These changes will be reflected in
the second interim progress report to Congress. MTMC also provided technical comments
that we incorporated as appropriate.

To accomplish this work, we met with officials from MTMC and PriceWaterhouse Coopers-
a MTMC support contractor-and reviewed key documents these officials provided us. We
conducted our review in accordance with generally accepted government auditing standards.

 Page 4                                                       GAO/NSI.AD-99-130RProgress of MTMC Pilot

We are sending copies of this report to the Secretary of Defense, the Commander in Chief,
USTRANSCOM, and the Commander, MTMC. We will also make copies available to others
upon request.

Major contributors to this letter were Charles Patton, Jr., Nomi Taslitt, and Marc Schwartz.
Please contact me at (202) 512-8412 if you or your staffhave any questions.

Sincerely yours,

David R. Warren, Director
Defense Management Issues

Page 5                                                       GAO/NSIAD-99-130RProgress of MTMC Pilot
ENCLOSURE I                                                                                     ENCLOSURE I

                            OUR OBSERVATIONS ON THE PILOT’S 10 GOALS


The Military Traffic Management Command (MTMC) includes six best commercial practices
under this goal. The practices are (1) full replacement value for loss and damage, (2) toll-free
telephone numbers to allow service members an opportunity to contact contractors directly,
(3) direct contact between service members and contractors for arranging personalized
moves and settling claims, (4) inconvenience claims for service members when contractors
do not honor accepted pickup or delivery times, (5) free storage for service members when
shipments arrive ahead of schedule and are placed in storage by the contractor, and (6) faster
payment of claims.

It is unclear how these commercial practices will be evaluated by “considering their overall
impact on customer satisfaction” through the customer survey or what data will be collected
for each of the practices. While we recognize that many of these practices are not,
measurable individually, MTMC does not state which of the practices will be measured
quantitatively and/or qualitatively. In addition, the sixth practice, faster payment of claims,
was one of the factors we could not validate in the Army Hunter pilot because start and end
dates could not be established with certainty.


  The interim progress report does not state what constitutes “on-time” for pickup and delivery
  and how the data will be collected and validated. That is, the parameters are not specified on
  how late or early a pickup or delivery can be made and still constitute “on time” and who is
. responsible for making the determination.


 Customer satisfaction will be evaluated based on pilot participants’ responses to two survey
 questions. These survey questions correspond with questions that were included in a MTMC
 1996 customer satisfaction survey that provided the baseline measure for this goal’.
 However, the 1996 survey had less than a 20-percent response rate; thus, comparisons
 between it andthe current survey should be made cautiously.

 ’ These 10 goals appear in MTMc’s February 27,1999, interim progress report.

 ’ Customer satisfaction surveys were administered during April and May 1997based on moves that occurred between March and
 August 1996.

 Page 6                                                                         GAOLWAD-9943OR Progress of MTMC Pilot
ENCLOSURE I                                                                    ENCLOSURE I

MTMC has set a pilot goal of 95 percent customer satisfaction to parallel the satisfaction level
best commercial companies have experienced; however, like the other nine goals it is unclear
how the results will be validated. If the pilot achieves a satisfaction rate of 92 percent-3
percent below the goal-does that indicate failure, even though this represents 21 percent
above the baseline?


Six business practices are included under this goal. The practices are (1) establish contracts
with suppliers, (2) establish long-term relationships with suppliers, (3) use a smaller number
of carriers to provide transportation services, (4) use commercial documentation practices to
eliminate government unique documents, (5) increase the use of electronic commerce
advances, and (6) adopt the growing trend of using relocation companies to perform personal
property movement services.

The interim progress report does not indicate how these business practices were selected,
nor is there a discussion of how these six practices will be measured, evaluated, validated,
and scored.


This goal consists of three subgoals: the percentage of shipments that incur damage
(loss/damage frequency); the percentage of shipments that generate claims from service
members (claims frequency); and the amount of money these claims represent (average claim
size). Baseline values are cited for all three subgoals. To evaluate the benefits of the pilot on
this goal, MTMC established performance standards for two subgoals, but not for
loss/damage frequency. However, no information is provided about how data will be
collected, validated, and scored, and the relative weighting of the subgoals.


According to the interim progress report, the pilot meets this goal if the administrative and
operating costs are less than the $370 per shipment under the current system. The $370 figure
was developed by a MTMC support contractor and is based on a 1995 study. According to
MTMC officials, it has not been validated by a service audit agency.


This goal focuses on how often carriers refuse to move shipments, and the possible effect
carrier refusal has on on-time pickup of shipments. MTMC established a baseline refusal rate
of 24 percent by excluding the non-peak months (October to April). However, using the 24
percent rate may be misleading since it does not represent the annual rate of refusal. MTMC
may want to consider comparing this data by month to get a better indicator of the impact of

Page 7                                                         GAO/NSIAD-9%13ORProgress of MTMC Pilot
ENCLOSURE I                                                                    ENCLOSURE I

seasonal periods on carrier’s refusal of shipments. In addition, MTMC does not describe how
the data will be collected, analyzed, and validated.


Opportunity for small business participation will be measured by comparing the number of
shipments and the contract value of these shipments that are moved by small businesses
under the pilot with the number and value moved under the current program (the baseline).
However, it will be difficult to determine a baseline of small businesses because historically
such data have been based on carrier/forwarder self-certification, which MTMC has
considered questionable.


The interim progress report states that this goal will be evaluated by assessing the other nine
goals to determine if quality of service has improved, customer satisfaction has increased,
and administrative and operating costs have decreased, while considering the change in
transportation costs.

According to MTMC, the question is “are the [expected] increases in transportation costs
reasonable compared to the improvements in quality of life for military members?” The
interim progress report does not indicate how this question will be answered. While MTMC
states a methodology to derive transportation costs by comparing what it would have cost to
move the same shipments under the current system, it is not clear how efficiencies and
quality of life improvements will be aggregated, measured, scored, validated, and compared
with transportation costs.


 Page 8                                                         GAO/i&AD-99-130R Progress of MTMC Pilot
Ordering    Information

The first copy of each GAO report and testimony is free.
Additional   copies are $2 each. Orders should be sent to the
following address, accompanied by a check or money order
made out to the Superintendent     of Documents, when
necessary. VISA and Mastercard      credit cards are accepted, also.
Orders for 100 or more copies to be mailed to a single address
are discounted 25 percent.

Orders by mail:

U.S. General Accounting    Office
P.O. Box 37050
Washington, DC 20013

or visit:

Room 1100
700 4th St. NW (corner of 4th and G Sts. NW)
U.S. General Accounting  Office
Washington, DC

Orders may also be placed by caRing (202) 512-6000
or by using fax number (202) 512-6061, or TDD (202) 512-2537.

Each day, GAO issues a list of newly available reports and
testimony.   To receive facsimile copies of the daily list or any
list from the past 30 days, please caB (202) 512-6000 using a
touchtone phone. A recorded menu wiIl provide information         on
how to obtain these lists.

For information on how to access GAO reports on the INTERNET,
send an e-mail message with “info” in the body to:


or visit GAO’s World Wide Web Home Page at:

United States
General Accounthg     Office
Washington,   D.C. 20548-0001

Official   Business
Penalty    for Private   Use $300

Address    Correction    Requested