oversight

Defense Transportation: Preliminary Personal Property Pilot Results Are Inconclusive

Published by the Government Accountability Office on 1999-12-21.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

      United States

GAO   General Accounting
      Washington,
                           Office
                    D.C. 20548
                                                                                                                       -..
      National     Security and
      International      Affairs Division


      B-284102

      December 21, 1999

      The Honorable Herbert H. Bateman
      Chairman
      The Honorable Solomon P. Ortiz
      Ranking Minority Member
      Subcommittee on Military Readiness
      Committee on Armed Services
      House of Representatives

      Subject: Defense Transportation: Preliminary Personal Property Pilot Results Are
               Inconclusive

      This report responds to your request for information on the Department of Defense’s report
      to Congress entitled DOD Demonstration Program to ImDrove the Qualitv of Personal
      Proper-@ ShiDments of Members of the Armed Forces, dated August 31, 1999. This is the
      Department’s second report on the progress of the demonstration (pilot) program. The pilot
      is intended to correct many of the problems in shipping personal property that exist with the
      current program, including poor service from movers, excessive loss and damage to
      servicemembers’ property, and high claims costs to the government.

      The Department of Defense has been concerned about the quality servicemembers receive
      from its nearly $3 billion annual program to transport, store, and manage the personal
      property of its personnel. In June 1995, the House Committee on National Security’ directed
      the Department to initiate a pilot program that incorporates commercial business practices
      and standards in its personal property program.’ Concerned about the pilot’s impact on small
      businesses, the Committee directed the Secretary of Defense to establish a working group of
      military and industry representatives to develop a mutually agreeable program to pilot test.

      During the summer of 1996, the working group reached a consensus on the pilot’s 10 goals.
      These included providing quality service, improving on-time pickup and delivery, and
      achieving high customer satisfaction. However, the working group could not agree on the
      approach the pilot test should take. Consequently, industry prepared an alternative plan that
      differed from the one favored by the Department in several ways. Differences included who
      could participate in the pilot, how shipping rates would be set, and how shipments would be
      allocated among participating moving companies. The Department’s proposal served as the


      ’ The House Committee on National Security is now called the House Committee on Armed Services.
      ’ H.Rept. 104-131. p. 164 (1995).




                                                                                      GAOflrlSIAD-OO-.52R Defense Transportation
framework for the Military Traffic Management Command’s pilot, while the transportation
industry’s proposal evolved into a proposal referred to as Commercial-Like Activities for
Superior Quality Demonstration Program.

The Command’s pilot was phased into operation in North Carolina, South Carolina, and
Florida over a 3-week period during January and February 1999. Industry’s proposal will not
be tested, but Congress has directed that the Command evaluate and report on the likelihood
of it satisfying the goals. As agreed with your offices, our objectives were to provide you with
information on the (1) preliminary pilot results, (2) the Department’s assessment of the
industry proposal, and (3) the Command’s plan for assessing the pilot’s results.

RESVLTS IN BRIEF

The preliminary pilot results are inconclusive, but do provide positive indications regarding
the pilot program. The Department’s progress report included preliminary results that
showed improvement over the current system associated with several of the pilot’s goals.
For example, pilot participants indicated a higher level of satisfaction with their moves, and
loss and damage claims that resulted from moves under the pilot were settled, on average,
seven times faster. At this time, the Department of Defense does not have sufficient data to
reach a final conclusion regarding the pilot results. The Department plans to submit a
subsequent report during the first quarter of calendar year 2000. This report is expected to
discuss whether the pilot is superior to the existing system.

The Department completed its evaluation of the industry proposal and based on available
information concluded that it was unlikely to achieve several of the pilot goals. These unmet
goals are to provide quality service, adopt corporate business processes, simplify the system,
and provide best value moving services to the government. Therefore, the Department plans
no further evaluation.

The methodology being used to evaluate the pilot program should provide a credible
analytical basis for determining the success of the pilot. The Military Traffic Management
Command’s evaluation plan addresses most of the methodological weaknesses we identified
in our earlier report3 For example, the relative weight each of the goals has in determining
overall pilot success and the roles and responsibilities of participating organizations are now
incorporated in the evaluation plan. One remaining issue is to establish baseline
transportation and process costs. According to Command officials, the baseline costs will be
finalized, evaluated, and reported in a report to be issued during the first quarter of 2000.

BACKGROUND

Section 376 of the Strom Thurmond National Defense Authorization Act for Fiscal Year 1999
(P. L. 105261) required the Secretary of Defense to submit interim reports on the Military
Traffic Management Command pilot no later than January 15 and April 15, 1999, and a final
report to Congress no later than August 31, 1999. The Department issued its first interim


’ Defense   Transoortation:   Pronress   of MTMC   Pilot   (GAO/NSIAD99-130R.   Apr.   15. 1999).




Page    2                                                                                      GAOLWAD-00-BZR   Defense   Transportation
B-284102


report on February 27, 1999. Due to a delay in program start-up, the Department did not issue
a report on April 15 because there was insufficient data on which to prepare the report. The
Department issued its next report on August 31, which consolidated the April 15 and
August 31 reports. Although the August 31 report should, according to the 1999 Defense
Authorization Act, include an evaluation of whether the pilot met its goals, the August report
reflected only the then-current status of the pilot. This report did not provide the final results
of the pilot program. The Department will submit a subsequent report during the first quarter
of calendar year 2000, when additional data is expected to be available.

During the summer of 1996, the working group reached a consensus on the pilot’s 10 goals
(see fig. 1). However, the working group could not agree on how the pilot should be
implemented; consequently, the Department and industry prepared separate proposals to
Congress. We reviewed both proposals and determined that the Department’s proposal met
the agreed upon goals to a greater extent.4 Because the industry proposal is not being tested,
the Department retained a contractor to evaluate the extent that the proposal could achieve
the 10 goals. The criteria for making this assessment were based on the professional
judgment of the contractor and are described in documents we obtained from the
 Department and the contractor.

Figure 1: The 10 Goals for the Personal Property Shipment Pilot

@ii+          Provide      quality     service.

@+            Improve      on-time       pickup.

 Goal 3       Improve      on-time       delivery.
o+
 Goal 4       Achieve      high customer             satisfaction.
o+
 Goal 5       Adopt corporate            business        processes.
o+
@iii+          Lower loss and damage.

@ii+           Simplify     the system.

 Goal 8        Ensure capacity           to meet the Department’s              needs.
o+
 Goal 9        Provide     opportunity        for small businesses.
o+
@iti+          Provide     best value moving              services to the government.
Source:   The Military Traffic       Management       Command.



’ Defense Transwrtation:    Reennineeriw? the DOD Personal           ROU&V   RO~WI   (GAOflrlSIAD97-49,   NJJ. 27,1996).




 Page 3                                                                                     GAO/NSIAD-W-52R Defense Transportation
B-284102


The pilot was to include -50percent of the eligible moves originating in North Carolina South
Carolina, and Florida to destinations elsewhere in the continental United States and to
Europe; the remaining half were to move under the Command’s existing system.’ The pilot
was expected to include about 18,500 annual shipments and run for a 3-year test period,
which will end in January 2002. However, based on actual shipments to date, officials
anticipate that approximately 9,000 shipments will occur under the pilot in the first year. In
collaboration with defense transportation officers in the three participating states, Command
officials are investigating the reasons that the number of shipments is lower than estimated.

PRELIMINARY PILOT RESULTS INCONCLUSIVE
BUT PROVIDE POSITIVE INDICATIONS

The results reported in the Department’s August 1999 report were based on about 5,500
personal property shipments that were initiated during the first 6 months of the pilot. The
results do not cover the entire May through September moving season when the majority of
Defense moves are made, and therefore are insufficient to make an overall evaluation of the
pilot. The preliminary results indicate improvements related to five of the pilot goals,

--       Eleven percent of the shipments resulted in servicemembers’ filing loss and damage
         claims for which they received payment, and a claim was settled, on average, 17 days
         after it was filed with the contractor. In contrast, under the existing system, 35 percent
         of shipments result in servicemembers’ filing claims and settlement takes, on average,
         126 days. (Goal l-provide quality service and goal 6-lower loss and damage.)

--       Eighty-one percent of the servicemembers who completed the customer satisfaction
         survey after their moves indicated that they would use their move contractors again. If
         this satisfaction rate persists, it will exceed the 75percent rating under the current
         system that serves as the baseline. (Goal 4-achieve high customer satisfaction.)

--       The frequency of contractors’ refusing offers to move shipments dropped from the
         current 24percent peak season rate to less than 2 percent, which included about half of
         the most recent May to September peak moving season. (Goal 8-ensure capacity to
         meet the Department’s needs.)

--       Large businesses submitted subcontracting plans to allocate almost 60 percent of their
         shipments (as measured by contract value) to small businesses. If these plans are
         realized, the pilot’s goal of providing small businesses an opportunity to move more than
         39 percent of all shipments would be satisfied. (Goal g-provide opportunity for small
         businesses.)




’ Some   moves   are excluded   from the pilot such as moves   from   nontemporary   storage   and intrastate   moves.




Page 4                                                                                         GAOi?4SIAJ3-OOdPR         Defense   Transportation
B-284102



DEPARTMENT CONCLUDES THAT INDUSTRY
PROPOSAL WOULD NOT ACHIEVE SEVERAL GOALS

The Department completed its evaluation of the industry proposal! and based on available
information concluded that it was unlikely to achieve several of the pilot goals. Therefore,
the Department plans no further evaluation. Based on its review, the Department concluded
that the proposal probably would not meet at least 4 of the 10 pilot goals: (1) provide quality
service, (2) adopt corporate business processes, (3) simplify the system, and (4) provide best
value moving services to the government. Further, the Department reported that since the
pilot will not be tested, there is insufficient information to determine whether the industry
proposal would meet the remaining six goals. To evaluate the industry proposal, the
Department relied on the professional judgment of a contractor and information contained in
two industry documents that described the proposal.

We agree with the Department’s conclusion that, based on available information, the industry
proposal is unlikely to satisfy at least three of the goals. Specifically, the proposal is unlikely
to satisfy the goal to provide quality service because it would not offer servicemembers full
replacement value for loss and damages that occur during their moves, nor would it provide
free storage for shipments that arrive earlier than scheduled. The proposal also is not likely
to satisfy the goals to adopt corporate business processes and simplify the system because it
would not increase the use of electronic commerce or web-based applications, nor would it
streamline the existing personal property movement process. Regarding the goal to provide
best value moving services, while the Department has stated this is not likely to be achieved,
we believe there is not enough information at this time to gauge whether it would be satisfied
until the baseline transportation cost data is finalized. Moreover, we agree with the
Department that there is not enough information to evaluate the proposal on the remaining
six goals.

EVALUATION PLAN CORRECTS
METHODOLOGICAL WEAKNESSES

The Command’s evaluation plan, dated September 30,1999, corrects most of the
methodological weaknesses we identified in the first progress report, issued February 27,
1999, which discussed the evaluation strategy. Specifically, the evaluation plan discusses
(1) the roles and responsibilities         of participating       organizations;     (2) the way data will be
collected, analyzed, and evaluated; and (3) the relative weight each goal has in determining
the pilot’s success. The evaluation plan describes in sufficient detail what data will be used
to evaluate pilot success for each goal and how data will be collected and analyzed. Baseline
cost data that had been established for two goals are being reexamined for accuracy.
According    to Command       officials,   these cost data-average             transportation     and process costs
to move a personal property shipment-will                     be finalized before the Department’s next report
is issued in the first quarter of 2000.

While the Department and industry did not establish priorities among the 10 goals, the
Command described criteria in the evaluation plan that gave weight to each of the first 9
goals equally and that gave greater weight to the tenth goal (provide best value moving
services to the government). If the pilot meets five or more of the first nine goals, and
associated   cost increases     are deemed        reasonable       given the quality     of improvements       that have
resulted, it will be considered superior to the existing program. The plan lists data sources



Page 5                                                                             GAO/NSIAD-OO-SZR Defense Transportation
B-284 lo”


for each of the goals and identifies who is responsible for collecting and analyzing the pilot
results. For example, the plan states that the Department of Defense Inspector General will
assess the data collection process and the costing methodology. Obtaining complete and
accurate baseline program cost data is critical to evaluating the pilot’s achievement  of the
seventh and tenth goals.”

AGENCY COMMENTS

We provided the Secretary of Defense a draft of this report on November 24, 1999, for review
and comment. The Department did not provide written comments, but senior-level
transportation policy and personal property officials of the Office of the Secretary of Defense
and the Military Traffic Management Command provided oral comments on December 7,
1999. The officials agreed with the facts presented in the report. They stated that the next
report on the pilot, which was scheduled for December 15, 1999, would be issued during the
first quarter of 2000. This delay will allow the Department to finalize the baseline cost data
and incorporate it into the pilot evaluation. Department officials also told us that the
evaluation of the industry proposal is complete and that they plan no further evaluations. We
revised our report to reflect this additional information.

SCOPE AND METHODOLOGY

We conducted our work and interviewed officials at headquarters, Miiitary Traffic
Management Command, Falls Church, Virginia; Parsifal Corporation, Melbourne, Florida; and
PriceWaterhouse Coopers, Fairfax, Virginia.. We also attended quarterly progress meetings
between the Command and contractors participating in the pilot.

To review the preliminary pilot results and verify the accuracy of key statements in the
progress report, we obtained relevant documents from Command officials and met with
officials from the Command and Parsifal Corporation, a support contractor responsible for
compiling and reporting contractor performance data. Although some inaccuracies were
noted, corrections were made and incorporated into our analysis. To review the
Department’s assessment of the industry proposal, we obtained relevant documents and met
with officials from the Command and PriceWaterhouse Coopers, a support contractor
responsible for evaluating the industry proposal. To assess the Command’s methodology for
evaluating the final pilot results and the industry proposal, we met with Command and
PriceWaterhouse Coopers officials. We reviewed versions of the Command’s evaluation plan
and interim progress report to ensure that they addressed methodological weaknesses we
identified in our April 15, 1999 report to Congress. We conducted our review from September
through November 1999 in accordance with generally accepted government auditing
standards.




 Our most recent financial statement        audit results highlighted  the Department’s  current   systems’     inability     to capture the full cost
of its activities and programs    (Deomment       of Defense: Status of Financial Management       Weaknesses        and    ActIons Needed to
Correct Continuiw      Challenges   (GAO/T-AIMDNSIAD-99-171.          May 4.1999).




Page   A                                                                                       GAONSIAD-OO-5PR              Defense   Transportation
B-234102


We are sending copies of this report to the Honorable William S. Cohen, Secretary of Defense;
General Charles T. Robertson, Jr., Commander in Chief of the U. S. Transportation Command;
and Major General Kenneth L. Privratsky, Commander of the Military Traffic Management,
Command. We will also make copies available to others upon request.

Major contributors to this report were Charles Patton, Jr., Nomi Taslitt, and Marc Schwartz.
Please contact me at (20‘2) 512-8412 if you or your staff have any questions.




2&d-
Defense Management Issues




(iO9413)




Page i                                                        GAO/NSIAD-OO-62R Defense Transportation
Ordering      Information

The first copy of each GAO report and testimony       is free.
Additional    copies are $2 each. Orders should be sent to the
following   address, accompanied    by a check or money order
made out to the Superintendent      of Documents,   when
necessary.   VISA and Mastercard     credit cards are accepted, also.
Orders for 100 or more copies to be mailed to a single address
are discounted      25 percent.

Orders      by mail:

U.S. General Accounting               Office
P.O. Box 37050
Washington,  DC 20013

or visit:

Room 1100
700 4th St. NW (corner           of 4th and G Sts. NW)
U.S. General Accounting            Office
Washington,  DC

Orders may also be placed by calIing (202) 512-6000
or by using fax number  (202) 512-6061,  or TDD (202)         512-2537.

Each day, GAO issues a list of newly available     reports and
testimony.   To receive facsimile   copies of the daily list or any
list from the past 30 days, please caII (202) 512-6000 using a
touchtone   phone.   A recorded   menu will provide information     on
how to obtain these lists.

For information          on how to access GAO reports on the INTERNET,
send an e-mail         message with “info” in the body to:

info@www.gao.gov

or visit    GAO’s      World   Wide     Web Home   Page at:

httpti/www.gao.gov
United States
General Accounting    Office
Washington,   D.C. 20548-0001

Offkial   Business
Penalty   for Private   Use $300

Address   Correction    Requested