oversight

Defense Transportation: The Army's Hunter Pilot Project Is Inconclusive but Provides Lessons Learned

Published by the Government Accountability Office on 1999-06-23.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                   United States General Accounting Office

GAO                Report to Congressional Committees




June 1999
                   DEFENSE
                   TRANSPORTATION

                   The Army’s Hunter
                   Pilot Project Is
                   Inconclusive but
                   Provides Lessons
                   Learned




GAO/NSIAD-99-129
United States General Accounting Office                                                              National Security and
Washington, D.C. 20548                    Leter
                                                                                              International Affairs Division



                                    B-282406                                                                                  Letter

                                    June 23, 1999

                                    Congressional Committees

                                    The Department of the Army began a pilot project in July 1997 to test an
                                    alternative approach for providing relocation services for its military
                                    personnel stationed at Hunter Army Airfield, Savannah, Georgia. The Army
                                    undertook this effort to address long-standing concerns and problems
                                    associated with the current personal property program. The Department of
                                    Defense (DOD) has two other pilot programs underway to test different
                                    approaches to improving its personal property program and is proposing a
                                    fourth pilot. To determine which pilot, or portions thereof, could provide
                                    better long-term results, the Deputy Under Secretary of Defense (Logistics)
                                    tasked the U.S. Transportation Command (USTRANSCOM) to oversee all
                                    personal property pilot tests and recommend the follow-on course of
                                    action.

                                    The statement of managers in the conference report on the 1997 DOD
                                    Appropriations Act directed us to validate the results and savings achieved
                                    before DOD expands any of its personal property pilot projects.1 In this
                                    regard, we testified in March 1999 on the status of the Army’s pilot and
                                    DOD’s efforts to improve its personal property program.2 This report
                                    supplements our testimony and principally addresses the results of our
                                    review of the 12-month Hunter pilot test and lessons for evaluating other
                                    pilots. Our objectives were to (1) assess the Army’s evaluation
                                    methodology of the Hunter pilot, including the validity of data and reported
                                    results and (2) determine the status of all ongoing and planned pilot
                                    projects and the adequacy of DOD’s plans to evaluate the pilot projects.



Results in Brief                    Although the Army reported that the Hunter pilot was a success, we found
                                    that most of the results of its evaluation were inconclusive. While the
                                    Hunter pilot provides services and benefits that were not previously
                                    available during the moving process, we were unable to validate all



                                    1 House Report   104-863 (Sept. 28, 1996) p. 865.

                                    2 Defense Transportation: Efforts to Improve DOD’s Personal Property Program (GAO/T-NSIAD-99-106,
                                    Mar. 18, 1999).




                   Leter            Page 1                                              GAO/NSIAD-99-129 Defense Transportation
        B-282406




        reported results of the Army’s evaluation of this pilot because of
        weaknesses in the evaluation methodology and data. Specifically, because
        of the methods used in conducting the customer surveys, we could not
        confirm that customer satisfaction improved. Also, due to weaknesses in
        the Army’s methodology and data reliability, we could not validate the
        extent to which pilot costs exceeded baseline costs. However, we were
        able to confirm that 33 percent of the pilot shipments were awarded to
        small business carriers and agents. Further, lessons learned by the Army in
        developing an evaluation plan, conducting the pilot test, and evaluating
        results can provide useful information to DOD as it conducts and assesses
        other pilot efforts.

        The Navy and the Military Traffic Management Command (MTMC) each
        have a personal property pilot project underway, and DOD is proposing a
        fourth pilot to test different approaches to improve its personal property
        program. As a result, DOD will be running multiple pilots concurrently,
        with different goals, objectives, and expected outcomes. USTRANSCOM3 is
        tasked with evaluating the results of the pilots and using that information
        to recommend a redesigned Department-wide relocation program.
        However, DOD has not yet determined how many approaches will
        ultimately be tested and the milestones for completing the pilots’
        evaluation and implementing an improved process, nor has it assured itself
        that a methodologically sound evaluation process is in place to execute this
        process.

        Improving DOD’s personal property program has been a slow, complex
        process. DOD and the services have spent a large amount of time and effort
        to dramatically change the quality of services military personnel receive.
        We support pilots as a tool to test different approaches. However, before
        the Department can make any credible decisions on changing the current
        program, we recommend that the Secretary of Defense direct
        USTRANSCOM to develop a comprehensive strategy for evaluating each of
        the pilots’ attributes in a comparable manner. Further, on the basis of the
        Army’s lessons learned in conducting the Hunter pilot and of our evaluation
        of the pilot, we recommend that the Department seek expert
        methodological advice to enhance the quality of its assessment.


        3 The mission of USTRANSCOM, which is DOD’s single manager of all Defense transportation services,
        is to provide global air, land, and sea transportation to meet national security needs. USTRANSCOM
        executes its mission through three component commands: MTMC for land transportation and port
        operations, the Military Sealift Command for sea transportation, and the Air Mobility Command for air
        transportation.




Leter   Page 2                                               GAO/NSIAD-99-129 Defense Transportation
                            B-282406




Background                  DOD has long been concerned about the quality of service it provides
                            military personnel and their families when they relocate. DOD spends
                            approximately $3 billion annually to transport, store, and manage the
                            household goods and unaccompanied baggage of its servicemembers and
                            families. Past problems included poor service from movers, excessive
                            incidence of loss or damage to servicemembers’ property, and high claims
                            costs to the government. All of these problems contributed to poor quality
                            of service for people using the system.



Most Results of the         Because of weaknesses in the Army’s evaluation methodology and data, we
                            were unable to validate the reported results of the quality of life and cost
Army Hunter Pilot Are       factors of the Hunter pilot program. Moreover, the Army’s evaluation plan
Inconclusive                methodology was not an effective tool for collecting and analyzing the pilot
                            results. However, lessons learned by the Army in conducting the 12-month
                            pilot test do provide useful information to DOD as it conducts and assesses
                            its pilot efforts. Also, the Hunter pilot is providing services and benefits
                            that were not previously available, including point-to-point move
                            management, personal move counseling and coordination, direct claims
                            settlement, assistance in buying/selling a residence, and visibility of the
                            shipment throughout the move.


The Army’s Assessment and   The Army reported that through its first 12 months of operations, the
Evaluation Approach         Hunter pilot successfully demonstrated that commercial practices could be
                            applied to the military relocation process. Specifically, the Army was able
                            to contract with Cendant Mobility, a move management company, to
                            provide services similar to those available in the private sector. In January
                            1997, the Army began developing a test evaluation plan that defined roles
                            and responsibilities, test factors, and processes for capturing data related
                            to each test factor and stated how test data would be compared to that
                            from the existing program (i.e., the baseline). The plan stated that the Army
                            Audit Agency (AAA) would be responsible for validating the baseline and
                            test data and for providing the Army a report on the test results. We
                            reported in June 1998 that the Army needed to further define the various
                            factors and measurements to be included in its evaluation plan.4 The Army
                            clarified how it would measure pilot success and revised its methodology


                            4 Defense
                                   Transportation: The Army’s Hunter Pilot Project to Outsource Relocation Services
                            (GAO/NSIAD-98-149, June 10, 1998).




                            Page 3                                             GAO/NSIAD-99-129 Defense Transportation
B-282406




in the evaluation plan, dated August 12, 1998. This represented the third
such revision, which was not completed until after the 12-month test
period.

The modified plan indicated that the pilot results would be reviewed on the
basis of three factors—quality of life, cost, and impact on small business—
and would use a 5-point scoring system. The Army assigned the greatest
weight to the quality of life factor, which consisted of one point each for
three sub-factors—customer satisfaction, average claims settlement time,
and percentage of direct deliveries. Along with the 3 points for quality of
life, the Army assigned 1 point each for two other factors, total cost and
impact on small business, for a total of 5 points. Appendix I provides
information on the scale and scores used by the Army to evaluate the
results of each factor and sub-factor.

In its October 1998 evaluation report and February and March 1999
supplements, the Army reported to us that Cendant Mobility managed
1,349 shipments involving household goods, unaccompanied baggage,
do-it-yourself (DITY) moves, and mobile homes during the 12-month test.
(See appendix II for the number of shipments in each category, including
domestic and international volumes.) The Army also reported that the pilot
project earned a passing score on each of the test factors that the Army
measured and deemed the Hunter pilot a success because the factor points
exceeded the minimum threshold of 3 based on a 5-point scale. According
to the Army’s analysis, the Hunter pilot test earned 3.75 points as follows:

• customer satisfaction improved by over 11 percent (0.25 point);
• the initial offer to settle claims averaged 9 days (1 point);
• 100 percent of eligible shipments were delivered directly to the
  servicemember without storage in transit (1 point);
• pilot project costs exceeded baseline costs by 18.6 percent (0.50 point);
  and
• 33 percent of the shipments were awarded to small businesses (1 point).

The Army stated that it also considered other factors generally related to
transportation process improvements and did not rely on test scores alone
to determine the pilot a success. These factors included simplifying the
price structure and claims process, providing pre-audit services, and
minimizing government-unique requirements. Each servicemember worked
with one personal move coordinator who integrated the member’s move,
providing point-to-point move management, coordination, and prompt
resolution of all problems. This was in contrast to the current process, in



Page 4                                  GAO/NSIAD-99-129 Defense Transportation
                            B-282406




                            which a servicemember may need to contact four or five different offices
                            over the course of a single move. Neither AAA nor we reviewed these cited
                            benefits because they were not included as measurable factors in the
                            evaluation plan.


Evaluation Weaknesses Led   We identified a number of shortfalls in the Army’s evaluation methodology
to Inconclusive Results     and data. Specifically, the methodology for surveying customer satisfaction
                            and calculating claims settlement time, percentage of direct deliveries
                            made, and overhead costs did not allow the Army to obtain and analyze the
                            data needed to accurately assess pilot results. Furthermore, in several
                            instances the data collected to support the measurements did not
                            demonstrate achievement of the Army’s reported results. Most importantly,
                            we could not confirm that customer satisfaction improved by 11.5 percent.
                            Further, we could not confirm that all eligible shipments were delivered
                            directly without incurring in-transit storage costs, offer of claims
                            settlement time averaged 9 days, and the pilot program cost
                            18.6 percent more than the baseline cost. On the other hand, we were able
                            to confirm that 33 percent of the shipments were awarded by Cendant
                            Mobility to small business carriers and agents—10 percent over the Army’s
                            minimum threshold.


Inconclusive Results for    The Army’s methodology for evaluating customer satisfaction results was
Customer Satisfaction       flawed as it allowed servicemembers to be surveyed multiple times—by
                            Cendant Mobility, two survey contractors, and carriers. In addition to the
                            use of different populations and respondent rates, the various survey
                            instruments also contained different questions used to measure the level of
                            customer satisfaction. Therefore, we could not validate that customer
                            satisfaction improved by 11.5 percent, as reported by the Army, on the basis
                            of the conclusions reached by one of the four survey instruments.
                            Notwithstanding these concerns, other data gathered in these surveys but
                            not scored by the Army in accordance with its evaluation plan, provided
                            indicators of customer satisfaction as well as lessons learned that may be
                            useful to DOD for conducting other pilots.

                            To measure customer satisfaction, the Army directed its contractor
                            (Battelle’s Pacific Northwest National Laboratory) to survey members at
                            Hunter and those making similar moves at Fort Stewart, Georgia (the
                            baseline). Responses from servicemembers at both locations would be
                            compared. Subsequently, the Army decided to use USTRANSCOM’s survey,
                            conducted by Electronic Data Systems (EDS), to evaluate and score the
                            pilot project.


                            Page 5                                  GAO/NSIAD-99-129 Defense Transportation
B-282406




Nonetheless, servicemembers were surveyed multiple times, possibly by as
many as four different entities. EDS and Battelle (the survey contractors),
Cendant Mobility (which was contractually required to survey a sample of
customers), and the carrier (for its own quality control purposes) each may
have contacted the same servicemembers to query them on their move
experiences. As a result, the customer satisfaction results were
compromised and therefore inconclusive. EDS reported that some
servicemembers refused to respond to the survey and that an
undetermined number who did respond gave perfunctory answers. In a
caveat in its report, Battelle stated that it was aware that some individuals
were interviewed multiple times.

The Army informed us that it did not use Battelle’s results because the
Army found the draft report difficult to interpret and inconclusive. The
Army also had concerns about a disclaimer on the survey results.5
Moreover, the contractors’ survey results varied significantly. While EDS
reported that 11.5 percent more of the Hunter respondents were satisfied
with their current relocation experience than the Fort Stewart
respondents, Battelle reported only a 3-percent difference between the two
groups of respondents.6 We cannot explain all the reasons for these
differences. We do know that the questions asked by each contractor to
determine customer satisfaction were not identical and that the population
and number of respondents in the two surveys varied significantly. EDS
also had a much lower response rate (53 percent) than Battelle
(89 percent). Further, EDS used a 75-percent significance level, while
Battelle based its results using a 95-percent significance level. The latter is
the standard used in social science research.7

Other data obtained by the surveys but not scored by the Army, in
accordance with its evaluation plan, provides indicators of customer


5 TheArmy later learned that the Department of Energy requires such a disclaimer in all studies
performed for government entities and that the disclaimer had no relevance to the results reported by
Battelle.

6 Battelle
         also reported that the 3-percent figure was “statistically insignificant,” meaning that the
variance in responses from the two groups (personnel at Hunter and at Fort Stewart) was not great
enough to conclude that the move experiences differed significantly.

7 Significancelevel is the likelihood that a true population value may be rejected by a statistical test. In
the case of the two surveys, Battelle used a 95-percent and EDS used a 75-percent probability that the
differences in relocation satisfaction levels were due to something other than random occurrence.
Battelle found no significant difference in satisfaction levels, with a 5-percent chance of reaching an
incorrect conclusion. EDS did find a significant difference but with a 25-percent chance of reaching an
incorrect conclusion.




Page 6                                                   GAO/NSIAD-99-129 Defense Transportation
                         B-282406




                         satisfaction and highlight areas that could be addressed in future
                         evaluations. For example, the Battelle survey found that 61 percent of the
                         Hunter respondents were more satisfied with their most recent (pilot)
                         moving experience than with their prior move. Eighty-seven percent of
                         these respondents were satisfied or very satisfied with the carrier’s
                         responsiveness. Both surveys also indicated that servicemembers
                         relocating within the continental United States were generally more
                         satisfied with the pilot process than were personnel moving overseas.

                         The surveys also showed that servicemembers liked some unique features
                         the pilot offered such as one-on-one counseling and the simplified claims
                         process. However, few servicemembers took advantage of many of the
                         additional services that Cendant Mobility offered, particularly assistance in
                         house hunting. Cendant Mobility reported to us that its referrals to real
                         estate companies resulted in the sale of one residence in the Hunter area
                         and the purchase of five residences at new duty stations. Moreover,
                         officials of The Military Coalition, an association that represents 5 million
                         active duty, retired, reserve, and National Guard personnel and their
                         families, told us that more could be done to tailor relocation services to
                         meet unique military needs. Such services include assistance in finding
                         rental property, because military personnel tend to rent rather than
                         purchase, and assistance for spouses seeking employment.

                         One of the lessons learned is related to access to survey data by outside
                         reviewers. AAA was unable to compare respondent results across surveys
                         because the survey contractors, citing privacy act restrictions, did not grant
                         access to identifier documentation. The Army stated that in future pilot
                         projects it would ensure that servicemembers are not subjected to multiple
                         surveys and would establish a better survey method for determining
                         customer satisfaction.

Limited Data on Claims   Due to weaknesses in the Army’s data collection and analysis, we could not
Settlement Time          validate that the initial offer to settle claims took an average of 9 days after
                         a completed claim form was received by Cendant Mobility. Furthermore,
                         the Army’s methodology was limited to examining the time between filing a
                         completed claim form and making the initial settlement offer. This measure
                         does not capture the entire claims process and therefore does not measure
                         the extent to which it represents an improvement over the existing system.
                         The pilot process, however, does offer benefits to the servicemember in
                         terms of full replacement value protection. Household effects are insured
                         for their full replacement value (up to $75,000) rather than their
                         depreciated value (which is capped at $40,000 under the existing system).



                         Page 7                                    GAO/NSIAD-99-129 Defense Transportation
                                 B-282406




                                 The pilot process also requires less work from the servicemember because
                                 the contractor provides estimates on lost and damaged household goods
                                 and pays the servicemember directly.

                                 The claims settlement time measurement is based on the contractual
                                 requirement that Cendant Mobility offer settlement to a member within
                                 30 days after receiving a claim for damaged or lost property. Consequently,
                                 the Army developed a threshold for claims settlement of 30 days. Under the
                                 Hunter pilot, servicemembers could appeal to the contractor for
                                 reconsideration by providing additional information or file a claim with the
                                 Army if resolution with the contractor could not be achieved—a process
                                 that could take more than 2 years to complete. While we found that some
                                 claims remained unpaid and unsettled because the claims were put “on
                                 hold” by the servicemember, the Army reported that no one in the Hunter
                                 pilot filed a claim with the Army claims office as of January 1999.

                                 The Army reported that Cendant Mobility took an average of 9 days to offer
                                 settlement for lost or damaged property, but neither AAA nor we could
                                 confirm this because the Army had not collected the necessary data. AAA
                                 determined, and we concur, that Cendant Mobility may have taken as long
                                 as an average of 28 days to offer settlement. This figure was based on the
                                 time between the date the claim was signed and the date the member was
                                 offered settlement. The difference between the two estimates was
                                 primarily due to a 19-day period between the date the claim was signed and
                                 the date the contractor reported entering the claim into its database.
                                 Although Cendant Mobility officials stated that standard company practice
                                 is to enter a claim into the database the same day it is received, they said
                                 delays could be caused by (1) the form being mailed some time after it is
                                 dated, (2) mail delays, and (3) contractor delays in entering the claim into
                                 the database. Army officials stated that, due to the difficulties in measuring
                                 claims settlement time, they would change their data collection and
                                 analysis of this sub-factor in future pilot programs.

Direct Delivery Results May Be   We could not confirm that all eligible direct deliveries were made without
Overstated                       in-transit storage because the necessary data was not collected. The Army’s
                                 methodology for this sub-factor also did not allow an assessment of
                                 whether the results represented an improvement over the current process.
                                 Direct delivery is defined by the Army as delivery of a shipment to a
                                 servicemember’s residence without storage in transit. A shipment is eligible
                                 for direct delivery only if it has a destination address before the shipment is
                                 offered for delivery. Direct delivery is a contract performance




                                 Page 8                                   GAO/NSIAD-99-129 Defense Transportation
                              B-282406




                              measurement, and the contractor is paid an incentive for maintaining a
                              direct delivery rate of over 60 percent. Consequently, the Army threshold
                              for this measurement is 60 percent.

                              Neither AAA nor we could confirm that shipments delivered directly to
                              servicemembers were the only ones eligible for direct delivery because the
                              necessary information on all shipments was not collected. That is, other
                              shipments may have been eligible under the Army criteria; however, AAA
                              and we could not determine which shipments had addresses prior to being
                              offered for delivery, and were thus eligible, but ended up requiring
                              in-transit storage. The Army acknowledged the difficulty in validating
                              eligible direct deliveries and stated that, in the future, it would consider
                              using storage-in-transit rates as a test factor. In the February 5, 1999,
                              supplemental information provided to us, the Army stated that 74 percent
                              of Army-wide shipments required in-transit storage, compared with
                              66 percent requiring temporary storage during the Army pilot.

                              Servicemembers benefit from having household goods delivered directly to
                              their home because the practice limits additional handling of their property,
                              reducing opportunities for loss and damage. Also, the government avoids
                              temporary in-transit storage, which is costly and hard for the Army to
                              control. Household goods shipments require temporary storage when
                              servicemembers do not know their new address at the time of shipment,
                              and/or they move to their new duty station after completing vacations or
                              military requirements en route.

Some Overhead Cost Data Was   Due to weaknesses in the Army’s methodology for collecting overhead cost
Inaccurate and Incomplete     data and the questionable reliability of some of the overhead cost data that
                              was collected, we could not validate that the additional cost of the pilot
                              project was 18.6 percent higher than the baseline cost. As we reported in
                              June 1998, developing overhead costs has historically been difficult for the
                              government, including DOD, because such data is often unreliable and
                              unavailable. (See appendix III for a detailed breakdown of pilot and
                              baseline costs as reported by the Army and validated by AAA).

                              Neither AAA nor we could validate some overhead costs. Specifically, we
                              could not confirm the accuracy of the overhead costs attributed to MTMC
                              because the Army used data from fiscal year 1994. MTMC has changed
                              significantly since then due to downsizing, but AAA could not obtain
                              updated costs from MTMC representing the current organization to
                              determine the reasonableness of this overhead cost. AAA and we also agree




                              Page 9                                  GAO/NSIAD-99-129 Defense Transportation
                                B-282406




                                that a drop in document processing (for claims, invoices, and inbound
                                shipments) would not result in cost savings to the government unless the
                                activities that perform these functions, such as the Defense Finance and
                                Accounting Service, take corresponding actions to reduce their costs
                                (for example, by reducing personnel to reflect the workload reduction).

                                AAA and we also identified other costs not included in the Army’s analysis
                                such as those related to awarding the Hunter contract and the contractor’s
                                use of foreign-flag vessels.8 The Army could not provide an estimated
                                dollar value associated with the cost of developing the contract
                                solicitation, reviewing offers, and resolving bid protests. While we
                                recognize that some of these estimated costs may be infrequent or one-time
                                only, they should be considered. Also, the Army did not include the
                                contractor’s use of foreign-flag vessels into the pilot project’s
                                transportation costs. While individually, costs such as these are probably of
                                low-dollar value in relation to other costs in the analysis, collectively, they
                                could materially affect the difference between the baseline and pilot costs.

                                According to the Army, some of the reasons for the 18.6 percent higher cost
                                of the pilot project can be attributed to (1) difficulty in calculating accurate
                                baseline costs, which the Army believes were understated; (2) low volume
                                of moves (1,349), which did not provide enough leverage to negotiate better
                                rates and discounts; (3) higher than expected costs of overseas shipments;
                                (4) relatively high unaccompanied baggage shipment rates, which could
                                have been lower using negotiated rates; (5) packing allowance for DITY
                                moves, which was calculated using commercial rates during part of the
                                12-month test period, resulting in a higher rate; and (6) quality moving
                                services, which cost more. In regard to the high cost of overseas shipments,
                                the Army provided us with additional details of pilot project costs that
                                showed a 44-percent increase for overseas household goods shipments and
                                a 2-percent increase for domestic shipments over the baseline cost.

Small Business Usage Exceeded   Both AAA and we validated that Cendant Mobility awarded 33 percent of
Army Goals                      the shipments to small businesses—10 percent to carriers and 23 percent to
                                agents. The Army measured the impact of the pilot on these two types of
                                providers and established a threshold of 23 percent to demonstrate
                                successful participation by small businesses.9 The Army based this factor
                                on section 15(g) of the Small Business Act, which establishes a


                                8 Foreign-flag   vessels are ships registered in foreign countries.

                                9 Inthe motor freight and transportation industry, firms with annual gross revenues of $18.5 million or
                                less are classified as small businesses. See Federal Acquisition Regulation 19.102.



                                Page 10                                                    GAO/NSIAD-99-129 Defense Transportation
                          B-282406




                          governmentwide goal for participation by small business concerns at not
                          less than 23 percent of the value of all prime contracts. The Army reported
                          that 261 (33 percent) of the 790 Hunter shipments billed during the
                          12-month pilot were subcontracted by Cendant Mobility to small carriers
                          and agents.

                          As we reported in June 1998, Cendant Mobility encountered problems in
                          attracting small carriers and agents to participate in the Hunter pilot. When
                          the pilot was implemented, several local agents and the companies they
                          represented elected not to sign agreements to participate. Cendant Mobility
                          reported that historically, over 40 percent of the contracts it awards go to
                          small and small and disadvantaged businesses. However, as of
                          December 31, 1997, only 14 percent of the shipments booked by Cendant
                          Mobility during the first 6 months had been awarded to small businesses.
                          By June 30, 1998, a number of national van lines and their local agents had
                          reversed their decision and had signed contracts with Cendant Mobility.



No Comprehensive          DOD will be running multiple pilots concurrently, each with different goals
                          and objectives and expected outcomes. MTMC reengineering efforts began
Strategy for Evaluating   in 1994 followed by the Army’s in 1996 and the Navy’s in 1997. DOD is
Multiple Pilots           proposing a fourth pilot with no specific start date. While USTRANSCOM is
                          tasked with evaluating the results of these pilots and using the information
                          to recommend a redesigned Department-wide relocation program, it does
                          not have a comprehensive strategy for doing so. Specifically, it has not yet
                          determined how many approaches it will ultimately test, it has not set the
                          milestones for completing the pilots’ evaluations and implementing an
                          improved process, and it has not assured itself that a methodologically
                          sound evaluation process is in place to execute this process.


MTMC’s Reengineering      DOD first proposed reengineering its personal property program in 1994.
Efforts Began in 1994     Specifically, on June 21, 1994, USTRANSCOM directed MTMC (the Army
                          component of USTRANSCOM and the program manager for DOD’s
                          Personal Property Shipment and Storage Program) to reengineer the
                          personal property program. On June 15, 1995, the House Committee on
                          National Security,10 concurring that DOD must pursue a higher level of
                          service, directed that DOD undertake a pilot program to implement
                          commercial business practices and standards of service. MTMC planned to


                          10Now   called the House Committee on Armed Services.




                          Page 11                                             GAO/NSIAD-99-129 Defense Transportation
B-282406




award contracts for the new program pursuant to the competitive
acquisition system provisions (10 U.S.C. ch. 137 and its implementing
regulation in the Federal Acquisition Regulation). Expressing
congressional concerns about the impact the competition system and any
nonstandard commercial business practice requirements might have on
small businesses, the statement of managers in the conference report
accompanying the 1997 DOD Authorization Act directed that DOD report
on the impact of the pilot program on small businesses. After reviewing the
reports, the House Committee on National Security remained concerned
that MTMC’s pilot program did not satisfactorily address issues raised by
small moving companies and directed that the Secretary of Defense
establish a working group of military and industry representatives to
develop an alternative pilot program.

Although the working group reached consensus on many issues, including
a set of program goals, it could not reach agreement on the approach to
take for the pilot test. Consequently, the two sides presented separate
proposals. In November 1996, we reported that in our assessment, MTMC’s
proposal met the goals of reengineering the personal property program to a
greater extent than the industry’s plan.11

The MTMC pilot program, delayed by numerous bid protests, finally started
in January 1999.12 The pilot includes 50 percent of the moves originating in
North Carolina, South Carolina, and Florida. The pilot will run concurrently
with the existing MTMC-managed program at installations representing all
military services and the Coast Guard in the three states and will involve
approximately 18,500 annual moves. The key features of the pilot program
include selecting carriers based on servicemember satisfaction and past
performance rather than simply price; achieving stronger carrier
commitment with long-term contracts; and offering full replacement value
protection and direct claims settlement to users. MTMC’s pilot will run for
a 3-year test period (1 year and two 1-year option periods) which will end in
December 2002.



11Defense  Transportation: Reengineering the DOD Personal Property Program (GAO/NSIAD-97-49,
Nov. 27, 1996).

12Solicitation
             disputes in the following decisions: Aalco Forwarding, Inc., et al, B-277241.8, B-277241.9,
October 21, 1997; Aalco Forwarding, Inc. et al., B-277241.12, B-277241.13, December 29,1997;
Aalco Forwarding, Inc. et al., B-277241.15, March 11, 1998; Aalco Forwarding, Inc. et al., B-277241.16,
March 11, 1998; Aalco Forwarding, Inc., et al., B-277241, 277241.14, June 8, 1998; and
Aalco Forwarding, Inc., et al., B-277241, B-277241.20, B-277241.21, July 1, 1998.




Page 12                                                GAO/NSIAD-99-129 Defense Transportation
                               B-282406




Hunter Pilot Effort Began in   Separately from the MTMC pilot program, the Army decided to determine
1996                           whether the commercial business practice of outsourcing a non-core
                               function could alleviate known problems in its personal property program
                               while increasing quality of life. The pilot began in February 1996 as a
                               quality-of-life effort to improve the relocation process and test commercial
                               business practices in a military environment at Hunter Army Airfield,
                               Savannah, Georgia. On January 31, 1997, the Army selected PHH
                               Relocation, a relocation/move management company (now Cendant
                               Mobility, Bethesda, Maryland), as the contractor for the pilot. In July 1997,
                               Cendant Mobility began relocation operations at Hunter, offering to
                               distribute and manage shipments to transportation providers. Additionally,
                               the contractor provided point-to-point move management, personal move
                               counseling and coordination, assistance in buying/selling a residence,
                               24-hour in-transit visibility of shipments, direct claims settlement with
                               servicemembers, full replacement value, and a single point of contact for
                               servicemembers. These services and benefits are not available under the
                               existing DOD system. The current contract with Cendant Mobility will end
                               on September 30, 1999.

                               A Cendant Mobility official stated that the management fees paid by the
                               Army to the contractor pay for these services and entitlements. Cendant
                               Mobility also obtained revenues from two other sources, including
                               (1) commissions from carriers, agents, and freight forwarders and
                               (2) referral fees from real estate companies on the purchase and sale of six
                               residences. The commissions paid for services provided to carriers, agents,
                               and freight forwarders, including contract training, contract
                               administration, invoice auditing, and quality control services.


Navy Pilot Effort Began in     The Navy initiated its plans for a separate pilot program in 1997 to test the
1997                           option of allowing servicemembers to select their carrier, giving them more
                               control over the relocation process to meet their specific needs. The pilot,
                               which began in January 1998, is intended to provide an alternative to,
                               rather than a replacement of, the current program. The pilot affects a very
                               small number of military personnel. Presently, the option is being offered
                               only for shipments originating in the areas of Puget Sound, Washington;
                               San Diego, California; Norfolk, Virginia; and Groton, Connecticut.
                               Participation is limited to Navy military personnel with permanent change
                               of station orders from the participating sites. Shipments must exceed
                               3,000 pounds and cost between $2,500 and $25,000 (the pilot excludes
                               shipments from non-temporary storage or warehouses). Servicemembers




                               Page 13                                 GAO/NSIAD-99-129 Defense Transportation
                            B-282406




                            can select a mover from a list of carriers that are self-certified as small
                            businesses and approved by MTMC. As of March 26, 1999, 223 Navy
                            servicemembers had selected this option. This pilot does not have a
                            specific end date.


DOD Pilot Effort Proposed   On February 12, 1999, DOD stated that it intended to begin a fourth pilot
in 1999                     program, called the Full Service Moving Project, with certain modifications
                            based on lessons learned from the Army Hunter pilot. The new pilot will
                            include the National Capital Region (17,000 annual moves), Georgia
                            (26,000 annual moves), and North Dakota (2,000 annual moves), and will
                            involve all military services and the Coast Guard. Like the Army pilot, it is
                            intended to outsource a non-core function and alleviate known problems in
                            the DOD personal property program while improving the quality of life
                            associated with the relocation of servicemembers and their families. While
                            it is projected to begin in calendar year 1999, presently, this pilot does not
                            have an official start date.


Unresolved Issues in        USTRANSCOM is in the process of finalizing its evaluation plan for the
Evaluating Pilot Programs   ongoing and planned pilots. It is proposing to evaluate the pilots on the
                            same three factors that the Army used in its evaluation of the Hunter pilot,
                            except that the factors would be defined differently and would include a
                            more expansive point scale. Unresolved issues, as of February 1999,
                            included the development of a method that would recognize the unique
                            characteristics and/or process improvements of each pilot program and the
                            validation of the baseline indirect costs that would be used for each pilot
                            program. The evaluation plan does not currently include an evaluation of
                            the Hunter pilot, but USTRANSCOM used lessons learned in the Hunter
                            pilot to develop the plan. We provided comments to USTRANSCOM in this
                            process and, among other things, encouraged the command to seek expert
                            methodological advice before finalizing the evaluation plan to enhance the
                            quality of its assessment.



Conclusions                 We support the use of pilots as a means to test new concepts. Although this
                            process has been ongoing since the early 1990s, DOD is not yet in a position
                            to determine what changes it needs to make to the process. Also, thus far,
                            DOD has not decided how many approaches it will ultimately test and has
                            not developed evaluation plans that will yield accurate and credible data.
                            Lessons learned from the Hunter pilot do provide useful information for




                            Page 14                                   GAO/NSIAD-99-129 Defense Transportation
                      B-282406




                      evaluating the three other pilots ongoing and planned. While
                      USTRANSCOM is in the process of developing a strategy to conduct this
                      evaluation, it has not yet resolved a number of issues, including timelines
                      for completing the analysis and implementing an improved process and a
                      methodology for the evaluation.



Recommendations       We recommend that the Secretary of Defense direct the Commander in
                      Chief, USTRANSCOM, to develop a comprehensive strategy for evaluating
                      each of the pilot approaches that includes (1) tests of a finite number of
                      approaches; (2) timelines for completing the analysis and for implementing
                      a new personal property process; and (3) milestones for finalizing a
                      methodologically sound evaluation plan, including specific criteria for
                      assessing each pilot’s attributes and validating the baseline indirect costs to
                      be used for each pilot program. To facilitate this process and to enhance
                      the quality of the assessment, we also recommend that DOD seek expert
                      methodological advice before finalizing the evaluation plan.



Agency Comments and   In written comments on a draft of this report, DOD stated that it generally
                      concurred with the report and its recommendations. DOD tasked
Our Evaluation        USTRANSCOM to evaluate the results of all the pilots and provide a
                      recommendation for a DOD-wide program to implement for the long term.
                      DOD stated that USTRANSCOM is in the process of retaining a contractor
                      to independently validate success criteria for a reengineered DOD personal
                      property program, develop a comprehensive strategy for evaluating each of
                      the pilot approaches, and support the development of recommendations
                      and alternative courses of action. USTRANSCOM anticipates having the
                      contractor commence its efforts by June 1999.

                      DOD’s comments are presented in appendix IV. DOD also provided
                      technical comments and we revised our report to reflect them where
                      appropriate.



Scope and             To validate the results and savings of the Hunter pilot, we reviewed the
                      Army’s plans to determine how it would evaluate the pilot’s results. We also
Methodology           reviewed the Army’s October 1998 report and the February and March 1999
                      supplements detailing the 12-month test results. We visited both Hunter
                      Army Airfield and the transportation division at Fort Stewart, Georgia. We
                      contacted Hunter contractor officials and representatives from the two



                      Page 15                                  GAO/NSIAD-99-129 Defense Transportation
B-282406




survey firms hired by the Army and USTRANSCOM to measure customer
satisfaction. We also examined the documents and data compiled by the
AAA to validate the accuracy of the Hunter results.

We met with Army, Navy, MTMC, and USTRANSCOM officials and
reviewed documents to determine the status and goals of other DOD
personal property pilot programs. We also obtained statistics on actual use
by servicemembers of the Navy’s pilot as of March 1999. We did not request
statistics on other pilots because the MTMC pilots began in January 1999
and the DOD-proposed Full Service Moving Project is still in the planning
stage. We also met with USTRANSCOM officials and reviewed evaluation
plans to determine how DOD proposes to make changes to its current
personal property program using lessons learned from the pilots.

We met and discussed matters related to DOD pilots and efforts to improve
the existing personal property program with officials from the Office of the
Deputy Under Secretary of Defense (Logistics-Transportation Policy),
Washington, D.C.; USTRANSCOM, Scott Air Force Base, Illinois; the Office
of the Army Deputy Chief of Staff for Logistics, Washington, D.C.; MTMC,
Falls Church, Virginia; the Naval Supply Systems Command,
Mechanicsburg, Pennsylvania; Hunter Army Airfield, Savannah, Georgia;
and Fort Stewart, Georgia.

In addition, we discussed the survey results with officials from Battelle’s
Pacific Northwest National Laboratory and from Electronic Data Systems.

We conducted our review between August 1998 and April 1999 in
accordance with generally accepted government auditing standards.


We are sending copies of this report to the Honorable William S. Cohen,
Secretary of Defense; the Honorable Louis Caldera, Secretary of the Army;
General Charles T. Robertson, Jr., Commander in Chief, USTRANSCOM;
Lieutenant General John G. Coburn, Deputy Chief of Staff, Logistics,
Department of the Army; Major General Mario F. Montero, Jr., Commander,
MTMC; and the Honorable Jacob J. Lew, Director, Office of Management
and Budget. We are also sending copies to other congressional members
who expressed interest to us in DOD’s efforts to improve its personal
property program, including the Honorable Fred Thompson, United States
Senate, and the Honorable Neil Abercrombie, Duncan Hunter, Martin T.
Meehan, and James M. Talent, House of Representatives. We will also make
copies available to others upon request.



Page 16                                 GAO/NSIAD-99-129 Defense Transportation
B-282406




Please contact me on (202) 512-8412 if you or your staff have questions
concerning this report. Major contributors to this report are listed in
appendix V.




David R. Warren, Director
Defense Management Issues




Page 17                                GAO/NSIAD-99-129 Defense Transportation
B-282406




List of Congressional Committees

The Honorable John W. Warner
Chairman
The Honorable Carl Levin
Ranking Minority Member
Committee on Armed Services
United States Senate

The Honorable Ted Stevens
Chairman
The Honorable Daniel K. Inouye
Ranking Minority Member
Subcommittee on Defense
Committee on Appropriations
United States Senate

The Honorable Floyd D. Spence
Chairman
The Honorable Ike Skelton
Ranking Minority Member
Committee on Armed Services
House of Representatives

The Honorable Herbert H. Bateman
Chairman
The Honorable Solomon P. Ortiz
Ranking Minority Member
Subcommittee on Military Readiness
Committee on Armed Services
House of Representatives

The Honorable Jerry Lewis
Chairman
The Honorable John P. Murtha
Ranking Minority Member
Subcommittee on Defense
Committee on Appropriations
House of Representatives




Page 18                              GAO/NSIAD-99-129 Defense Transportation
Page 19   GAO/NSIAD-99-129 Defense Transportation
Contents



Letter                                                                   1


Appendix I                                                              22
Thresholds and Scores
Used by the Army for
Hunter Pilot
Evaluation Factors

Appendix II                                                             23
Types and Number of
Shipments Made
During the Hunter Pilot
Test

Appendix III                                                            24
Hunter Pilot and
Baseline Costs
Calculated by the
Department of the
Army (Office of the
Deputy Chief of Staff
for Logistics)

Appendix IV                                                             25
Comments From the
Department of Defense

Appendix V                                                              28
Major Contributors to
This Report


                          Page 20   GAO/NSIAD-99-129 Defense Transportation
                       Contents




Related GAO Products                                                                      32




                       Abbreviations

                       AAA             Army Audit Agency
                       DITY            do-it-yourself
                       DOD             Department of Defense
                       EDS             Electronic Data Systems
                       MTMC            Military Traffic Management Command
                       USTRANSCOM      U.S. Transportation Command




                       Page 21                        GAO/NSIAD-99-129 Defense Transportation
Appendix I

Thresholds and Scores Used by the Army for
Hunter Pilot Evaluation Factors                                                                                               ApIenx
                                                                                                                                   di




Factors/definitions                                                                           Thresholds                Scores
Quality of life/customer satisfaction rating: percent of Hunter respondents who were      25% and above              1.00 point
more satisfied than Fort Stewart respondents (control group).
                                                                                                  20-24%            0.75 points
                                                                                                  15-19%            0.50 points
                                                                                                  10-14%            0.25 points
                                                                                              Below 10%           Unsuccessful
Quality of life/claims settlement time: time period between when a claim was received           0-10 days            1.00 point
and when the contractor offered settlement.
                                                                                               11-15 days           0.75 points
                                                                                               16-20 days           0.50 points
                                                                                               20-30 days           0.25 points
                                                                                           Above 30 days          Unsuccessful
Quality of life/percentage of eligible deliveries: percent of shipments delivered to            91-100%              1.00 point
residence without storage in-transit.
                                                                                                  81-90%            0.75 points
                                                                                                  71-80%            0.50 points
                                                                                                  61-70%            0.25 points
                                                                                           60% and below          Unsuccessful
Total cost: extent total contract cost exceeded total baseline cost.                                  0%             1.00 point
                                                                                                   1-10%            0.75 points
                                                                                                  11-20%            0.50 points
                                                                                                  21-30%            0.25 points
                                                                                          More than 30 %          Unsuccessful
Impact on small business: percent of small agents and carriers used.                         20% or more             1.00 point
                                                                                                  27-28%            0.75 points
                                                                                                  25-26%            0.50 points
                                                                                                  23-24%            0.25 points
                                                                                           Less than 23%          Unsuccessful




                                                Page 22                                 GAO/NSIAD-99-129 Defense Transportation
Appendix II

Types and Number of Shipments Made During
the Hunter Pilot Test                                                                                            ApIpx
                                                                                                                     di
                                                                                                                     Ien




Shipment type                                                       Shipments initiated and recorded as completed
Household goods (intrastate – Georgia                                                                          51
Household goods (interstate – continental U.S., including Alaska)                                             418
Household goods (overseas – Hawaii)                                                                            29
Household goods (overseas – Germany)                                                                           79
Household goods (overseas – Korea)                                                                             50
Household goods (overseas – other)                                                                             47
Unaccompanied baggage (overseas – all locations)                                                              378
Do-it-yourself (DITY) moves                                                                                   292
Mobile homes                                                                                                    5
Total                                                                                                       1,349




                                              Page 23                      GAO/NSIAD 99-129 Defense Transportation
Appendix III

Hunter Pilot and Baseline Costs Calculated by
the Department of the Army (Office of the
Deputy Chief of Staff for Logistics)                                                                                               AIpIenx
                                                                                                                                         di




Cost elementsa                                          Estimated pilot costs                        Estimated baseline costs
Transportation costs
Transportation                                                       $3,005,229                                     $2,102,384
Accessorials                                                              25,767                                        30,655
Storage                                                                  210,706                                       211,655
Total transportation costs                                           $3,241,702                                     $2,344,694


Claims paid by Army
Claims paid by Army                                                            $0                                      $66,268


Overhead costs
Personnel                                                                $79,802                                      $221,721
Management price                                                         132,700                                               0
MTMC headquartersb                                                               0                                      63,510
Automation                                                                       0                                      52,612
Voucher processing                                                          4,614                                       90,658
Inbound processing                                                               0                                      46,834
Claims processing                                                                0                                      11,510
Pay for performance                                                       14,756                                               0
Building overhead                                                         28,321                                        43,286
Telephone and copier                                                        1,281                                       10,106
Consumables                                                                 3,760                                         5,189
Total overhead costs                                                   $265,234                                       $545,426
Total costs                                                          $3,506,936                                     $2,956,388


                                                     Pilot              Baseline                                       Percent
                                                    costs                 costs                Difference           difference
Percent pilot costs
exceeded baseline costs                        $3,506,936            $2,956,388                  $550,548                  18.6
                             a
                              Army Audit Agency validated all costs except those under $5,000, concluding that such amounts were
                             immaterial.
                             bMilitary   Traffic Management Command (MTMC).
                             Source: Office of the Deputy Chief of Staff for Logistics, Department of Army.




                             Page 24                                                 GAO/NSIAD-99-129 Defense Transportation
Appendix IV

Comments From the Department of Defense                                         ApIV
                                                                                   ex
                                                                                   pdi
                                                                                    n




Note: GAO comment
supplementing those in the
report text appear at the end
of this appendix.




See comment 1.




                                Page 25   GAO/NSIAD-99-129 Defense Transportation
Appendix IV
Comments From the Department of Defense




Page 26                                   GAO/NSIAD-99-129 Defense Transportation
              Appendix IV
              Comments From the Department of Defense




              The following are GAO’s comments on the Department of Defense’s letter
              dated May 11, 1999.



GAO Comment   1. As stated in the report, we could not confirm the customer satisfaction
              survey results. Specifically, we could not confirm that all survey results
              indicated higher levels of customer satisfaction because of weaknesses in
              the survey methodology. Methodological weaknesses that compromised
              survey results included using different populations and respondent rates,
              using various survey instruments with different questions, and surveying
              servicemembers multiple times. In addition, one of the two survey
              contractors found no statistically significant difference between responses
              from Hunter and Fort Stewart.

              Our assessment of customer satisfaction results, as indicated in the report,
              was based on the Army’s criteria set forth in its evaluation plan. Because
              the Army used the results of one question to score the pilot a success, we
              also used it as the basis for our assessment. We agree, however, that other
              data obtained by the surveys provided indicators of customer satisfaction
              and highlighted areas that could be addressed in future evaluations.

              We agree that the bulk of cost elements could be validated and revised our
              report to show that some overhead cost data was inaccurate and
              incomplete.

              Finally, regarding our assessment of the claims process, we did not
              acknowledge a significant improvement in the process because the Army
              measured only one aspect of it, as indicated in the report. In addition, on
              the basis of the Army’s criteria set forth in its evaluation plan for assessing
              this factor, we could not confirm the time it took to offer claims settlement
              to servicemembers. We also agree that the claims process offered benefits
              not previously available to the servicemembers.




              Page 27                                   GAO/NSIAD-99-129 Defense Transportation
Appendix V

Major Contributors to This Report                                                                     Appx
                                                                                                         end
                                                                                                          Vi




National Security and   Charles I. Patton, Jr., Associate Director
                        Nomi R. Taslitt, Assistant Director
International Affairs   Robert L. Self, Evaluator-in-Charge
Division, Washington,   Marc J. Schwartz, Senior Evaluator
                        Jacqueline E. Snead, Senior Evaluator
D.C.                    Richard R. Irving, Evaluator
                        Arthur L. James, Jr., Mathematical Statistician



Office of the General   John G. Brosnan, Assistant General Counsel

Counsel, Washington,
D.C.


Norfolk Field Office    Daniel A. Omahen, Senior Evaluator
                        John R. Beauchamp, Senior Evaluator




                        Page 28                                 GAO/NSIAD-99-129 Defense Transportation
Page 29   GAO/NSIAD-99-129 Defense Transportation
Page 30   GAO/NSIAD-99-129 Defense Transportation
Page 31   GAO/NSIAD-99-129 Defense Transportation
Related GAO Products


                   Defense Transportation: Efforts to Improve DOD’s Personal Property
                   Program (GAO/T-NSIAD-99-106, Mar. 18, 1999).

                   Defense Transportation: The Army’s Hunter Pilot Project to Outsource
                   Relocation Services (GAO/NSIAD-98-149, June 10, 1998).

                   Defense Outsourcing: Better Data Needed to Support Overhead Rates for
                   A-76 Studies (GAO/NSIAD-98-62, Feb. 27, 1998).

                   Defense Transportation: Reengineering the DOD Personal Property
                   Program (GAO/NSIAD-97-49, Nov. 27, 1996).

                   Household Goods: Administrative Changes Would Improve DOD’s Do-It-
                   Yourself Moving Program (GAO/NSIAD-94-226, Sept. 27, 1994).

                   DOD Commercial Transportation: Savings Possible Through Better Audit
                   and Negotiation of Rates (GAO/NSIAD-92-61, Dec. 27, 1991).

                   Household Goods: Competition Among Commercial Movers Serving DOD
                   Can Be Improved (GAO/NSIAD-90-50, Feb. 12, 1990).




(709365)   Leter   Page 32                               GAO/NSIAD-99-129 Defense Transportation
Ordering Information

The first copy of each GAO report and testimony is free.
Additional copies are $2 each. Orders should be sent to the
following address, accompanied by a check or money order made
out to the Superintendent of Documents, when necessary, VISA and
MasterCard credit cards are accepted, also.

Orders for 100 or more copies to be mailed to a single address are
discounted 25 percent.

Orders by mail:

U.S. General Accounting Office
P.O. Box 37050
Washington, DC 20013

or visit:

Room 1100
700 4th St. NW (corner of 4th and G Sts. NW)
U.S. General Accounting Office
Washington, DC

Orders may also be placed by calling (202) 512-6000
or by using fax number (202) 512-6061, or TDD (202) 512-2537.

Each day, GAO issues a list of newly available reports and
testimony. To receive facsimile copies of the daily list or any list
from the past 30 days, please call (202) 512-6000 using a touchtone
phone. A recorded menu will provide information on how to obtain
these lists.

For information on how to access GAO reports on the INTERNET,
send an e-mail message with “info” in the body to:

info@www.gao.gov

or visit GAO’s World Wide Web Home Page at:

http://www.gao.gov
United States                       Bulk Rate
General Accounting Office      Postage & Fees Paid
Washington, D.C. 20548-0001           GAO
                                 Permit No. GI00
Official Business
Penalty for Private Use $300

Address Correction Requested