United States General Accounting Office GAO Report to Congressional Committees June 1999 DEFENSE TRANSPORTATION The Army’s Hunter Pilot Project Is Inconclusive but Provides Lessons Learned GAO/NSIAD-99-129 United States General Accounting Office National Security and Washington, D.C. 20548 Leter International Affairs Division B-282406 Letter June 23, 1999 Congressional Committees The Department of the Army began a pilot project in July 1997 to test an alternative approach for providing relocation services for its military personnel stationed at Hunter Army Airfield, Savannah, Georgia. The Army undertook this effort to address long-standing concerns and problems associated with the current personal property program. The Department of Defense (DOD) has two other pilot programs underway to test different approaches to improving its personal property program and is proposing a fourth pilot. To determine which pilot, or portions thereof, could provide better long-term results, the Deputy Under Secretary of Defense (Logistics) tasked the U.S. Transportation Command (USTRANSCOM) to oversee all personal property pilot tests and recommend the follow-on course of action. The statement of managers in the conference report on the 1997 DOD Appropriations Act directed us to validate the results and savings achieved before DOD expands any of its personal property pilot projects.1 In this regard, we testified in March 1999 on the status of the Army’s pilot and DOD’s efforts to improve its personal property program.2 This report supplements our testimony and principally addresses the results of our review of the 12-month Hunter pilot test and lessons for evaluating other pilots. Our objectives were to (1) assess the Army’s evaluation methodology of the Hunter pilot, including the validity of data and reported results and (2) determine the status of all ongoing and planned pilot projects and the adequacy of DOD’s plans to evaluate the pilot projects. Results in Brief Although the Army reported that the Hunter pilot was a success, we found that most of the results of its evaluation were inconclusive. While the Hunter pilot provides services and benefits that were not previously available during the moving process, we were unable to validate all 1 House Report 104-863 (Sept. 28, 1996) p. 865. 2 Defense Transportation: Efforts to Improve DOD’s Personal Property Program (GAO/T-NSIAD-99-106, Mar. 18, 1999). Leter Page 1 GAO/NSIAD-99-129 Defense Transportation B-282406 reported results of the Army’s evaluation of this pilot because of weaknesses in the evaluation methodology and data. Specifically, because of the methods used in conducting the customer surveys, we could not confirm that customer satisfaction improved. Also, due to weaknesses in the Army’s methodology and data reliability, we could not validate the extent to which pilot costs exceeded baseline costs. However, we were able to confirm that 33 percent of the pilot shipments were awarded to small business carriers and agents. Further, lessons learned by the Army in developing an evaluation plan, conducting the pilot test, and evaluating results can provide useful information to DOD as it conducts and assesses other pilot efforts. The Navy and the Military Traffic Management Command (MTMC) each have a personal property pilot project underway, and DOD is proposing a fourth pilot to test different approaches to improve its personal property program. As a result, DOD will be running multiple pilots concurrently, with different goals, objectives, and expected outcomes. USTRANSCOM3 is tasked with evaluating the results of the pilots and using that information to recommend a redesigned Department-wide relocation program. However, DOD has not yet determined how many approaches will ultimately be tested and the milestones for completing the pilots’ evaluation and implementing an improved process, nor has it assured itself that a methodologically sound evaluation process is in place to execute this process. Improving DOD’s personal property program has been a slow, complex process. DOD and the services have spent a large amount of time and effort to dramatically change the quality of services military personnel receive. We support pilots as a tool to test different approaches. However, before the Department can make any credible decisions on changing the current program, we recommend that the Secretary of Defense direct USTRANSCOM to develop a comprehensive strategy for evaluating each of the pilots’ attributes in a comparable manner. Further, on the basis of the Army’s lessons learned in conducting the Hunter pilot and of our evaluation of the pilot, we recommend that the Department seek expert methodological advice to enhance the quality of its assessment. 3 The mission of USTRANSCOM, which is DOD’s single manager of all Defense transportation services, is to provide global air, land, and sea transportation to meet national security needs. USTRANSCOM executes its mission through three component commands: MTMC for land transportation and port operations, the Military Sealift Command for sea transportation, and the Air Mobility Command for air transportation. Leter Page 2 GAO/NSIAD-99-129 Defense Transportation B-282406 Background DOD has long been concerned about the quality of service it provides military personnel and their families when they relocate. DOD spends approximately $3 billion annually to transport, store, and manage the household goods and unaccompanied baggage of its servicemembers and families. Past problems included poor service from movers, excessive incidence of loss or damage to servicemembers’ property, and high claims costs to the government. All of these problems contributed to poor quality of service for people using the system. Most Results of the Because of weaknesses in the Army’s evaluation methodology and data, we were unable to validate the reported results of the quality of life and cost Army Hunter Pilot Are factors of the Hunter pilot program. Moreover, the Army’s evaluation plan Inconclusive methodology was not an effective tool for collecting and analyzing the pilot results. However, lessons learned by the Army in conducting the 12-month pilot test do provide useful information to DOD as it conducts and assesses its pilot efforts. Also, the Hunter pilot is providing services and benefits that were not previously available, including point-to-point move management, personal move counseling and coordination, direct claims settlement, assistance in buying/selling a residence, and visibility of the shipment throughout the move. The Army’s Assessment and The Army reported that through its first 12 months of operations, the Evaluation Approach Hunter pilot successfully demonstrated that commercial practices could be applied to the military relocation process. Specifically, the Army was able to contract with Cendant Mobility, a move management company, to provide services similar to those available in the private sector. In January 1997, the Army began developing a test evaluation plan that defined roles and responsibilities, test factors, and processes for capturing data related to each test factor and stated how test data would be compared to that from the existing program (i.e., the baseline). The plan stated that the Army Audit Agency (AAA) would be responsible for validating the baseline and test data and for providing the Army a report on the test results. We reported in June 1998 that the Army needed to further define the various factors and measurements to be included in its evaluation plan.4 The Army clarified how it would measure pilot success and revised its methodology 4 Defense Transportation: The Army’s Hunter Pilot Project to Outsource Relocation Services (GAO/NSIAD-98-149, June 10, 1998). Page 3 GAO/NSIAD-99-129 Defense Transportation B-282406 in the evaluation plan, dated August 12, 1998. This represented the third such revision, which was not completed until after the 12-month test period. The modified plan indicated that the pilot results would be reviewed on the basis of three factors—quality of life, cost, and impact on small business— and would use a 5-point scoring system. The Army assigned the greatest weight to the quality of life factor, which consisted of one point each for three sub-factors—customer satisfaction, average claims settlement time, and percentage of direct deliveries. Along with the 3 points for quality of life, the Army assigned 1 point each for two other factors, total cost and impact on small business, for a total of 5 points. Appendix I provides information on the scale and scores used by the Army to evaluate the results of each factor and sub-factor. In its October 1998 evaluation report and February and March 1999 supplements, the Army reported to us that Cendant Mobility managed 1,349 shipments involving household goods, unaccompanied baggage, do-it-yourself (DITY) moves, and mobile homes during the 12-month test. (See appendix II for the number of shipments in each category, including domestic and international volumes.) The Army also reported that the pilot project earned a passing score on each of the test factors that the Army measured and deemed the Hunter pilot a success because the factor points exceeded the minimum threshold of 3 based on a 5-point scale. According to the Army’s analysis, the Hunter pilot test earned 3.75 points as follows: • customer satisfaction improved by over 11 percent (0.25 point); • the initial offer to settle claims averaged 9 days (1 point); • 100 percent of eligible shipments were delivered directly to the servicemember without storage in transit (1 point); • pilot project costs exceeded baseline costs by 18.6 percent (0.50 point); and • 33 percent of the shipments were awarded to small businesses (1 point). The Army stated that it also considered other factors generally related to transportation process improvements and did not rely on test scores alone to determine the pilot a success. These factors included simplifying the price structure and claims process, providing pre-audit services, and minimizing government-unique requirements. Each servicemember worked with one personal move coordinator who integrated the member’s move, providing point-to-point move management, coordination, and prompt resolution of all problems. This was in contrast to the current process, in Page 4 GAO/NSIAD-99-129 Defense Transportation B-282406 which a servicemember may need to contact four or five different offices over the course of a single move. Neither AAA nor we reviewed these cited benefits because they were not included as measurable factors in the evaluation plan. Evaluation Weaknesses Led We identified a number of shortfalls in the Army’s evaluation methodology to Inconclusive Results and data. Specifically, the methodology for surveying customer satisfaction and calculating claims settlement time, percentage of direct deliveries made, and overhead costs did not allow the Army to obtain and analyze the data needed to accurately assess pilot results. Furthermore, in several instances the data collected to support the measurements did not demonstrate achievement of the Army’s reported results. Most importantly, we could not confirm that customer satisfaction improved by 11.5 percent. Further, we could not confirm that all eligible shipments were delivered directly without incurring in-transit storage costs, offer of claims settlement time averaged 9 days, and the pilot program cost 18.6 percent more than the baseline cost. On the other hand, we were able to confirm that 33 percent of the shipments were awarded by Cendant Mobility to small business carriers and agents—10 percent over the Army’s minimum threshold. Inconclusive Results for The Army’s methodology for evaluating customer satisfaction results was Customer Satisfaction flawed as it allowed servicemembers to be surveyed multiple times—by Cendant Mobility, two survey contractors, and carriers. In addition to the use of different populations and respondent rates, the various survey instruments also contained different questions used to measure the level of customer satisfaction. Therefore, we could not validate that customer satisfaction improved by 11.5 percent, as reported by the Army, on the basis of the conclusions reached by one of the four survey instruments. Notwithstanding these concerns, other data gathered in these surveys but not scored by the Army in accordance with its evaluation plan, provided indicators of customer satisfaction as well as lessons learned that may be useful to DOD for conducting other pilots. To measure customer satisfaction, the Army directed its contractor (Battelle’s Pacific Northwest National Laboratory) to survey members at Hunter and those making similar moves at Fort Stewart, Georgia (the baseline). Responses from servicemembers at both locations would be compared. Subsequently, the Army decided to use USTRANSCOM’s survey, conducted by Electronic Data Systems (EDS), to evaluate and score the pilot project. Page 5 GAO/NSIAD-99-129 Defense Transportation B-282406 Nonetheless, servicemembers were surveyed multiple times, possibly by as many as four different entities. EDS and Battelle (the survey contractors), Cendant Mobility (which was contractually required to survey a sample of customers), and the carrier (for its own quality control purposes) each may have contacted the same servicemembers to query them on their move experiences. As a result, the customer satisfaction results were compromised and therefore inconclusive. EDS reported that some servicemembers refused to respond to the survey and that an undetermined number who did respond gave perfunctory answers. In a caveat in its report, Battelle stated that it was aware that some individuals were interviewed multiple times. The Army informed us that it did not use Battelle’s results because the Army found the draft report difficult to interpret and inconclusive. The Army also had concerns about a disclaimer on the survey results.5 Moreover, the contractors’ survey results varied significantly. While EDS reported that 11.5 percent more of the Hunter respondents were satisfied with their current relocation experience than the Fort Stewart respondents, Battelle reported only a 3-percent difference between the two groups of respondents.6 We cannot explain all the reasons for these differences. We do know that the questions asked by each contractor to determine customer satisfaction were not identical and that the population and number of respondents in the two surveys varied significantly. EDS also had a much lower response rate (53 percent) than Battelle (89 percent). Further, EDS used a 75-percent significance level, while Battelle based its results using a 95-percent significance level. The latter is the standard used in social science research.7 Other data obtained by the surveys but not scored by the Army, in accordance with its evaluation plan, provides indicators of customer 5 TheArmy later learned that the Department of Energy requires such a disclaimer in all studies performed for government entities and that the disclaimer had no relevance to the results reported by Battelle. 6 Battelle also reported that the 3-percent figure was “statistically insignificant,” meaning that the variance in responses from the two groups (personnel at Hunter and at Fort Stewart) was not great enough to conclude that the move experiences differed significantly. 7 Significancelevel is the likelihood that a true population value may be rejected by a statistical test. In the case of the two surveys, Battelle used a 95-percent and EDS used a 75-percent probability that the differences in relocation satisfaction levels were due to something other than random occurrence. Battelle found no significant difference in satisfaction levels, with a 5-percent chance of reaching an incorrect conclusion. EDS did find a significant difference but with a 25-percent chance of reaching an incorrect conclusion. Page 6 GAO/NSIAD-99-129 Defense Transportation B-282406 satisfaction and highlight areas that could be addressed in future evaluations. For example, the Battelle survey found that 61 percent of the Hunter respondents were more satisfied with their most recent (pilot) moving experience than with their prior move. Eighty-seven percent of these respondents were satisfied or very satisfied with the carrier’s responsiveness. Both surveys also indicated that servicemembers relocating within the continental United States were generally more satisfied with the pilot process than were personnel moving overseas. The surveys also showed that servicemembers liked some unique features the pilot offered such as one-on-one counseling and the simplified claims process. However, few servicemembers took advantage of many of the additional services that Cendant Mobility offered, particularly assistance in house hunting. Cendant Mobility reported to us that its referrals to real estate companies resulted in the sale of one residence in the Hunter area and the purchase of five residences at new duty stations. Moreover, officials of The Military Coalition, an association that represents 5 million active duty, retired, reserve, and National Guard personnel and their families, told us that more could be done to tailor relocation services to meet unique military needs. Such services include assistance in finding rental property, because military personnel tend to rent rather than purchase, and assistance for spouses seeking employment. One of the lessons learned is related to access to survey data by outside reviewers. AAA was unable to compare respondent results across surveys because the survey contractors, citing privacy act restrictions, did not grant access to identifier documentation. The Army stated that in future pilot projects it would ensure that servicemembers are not subjected to multiple surveys and would establish a better survey method for determining customer satisfaction. Limited Data on Claims Due to weaknesses in the Army’s data collection and analysis, we could not Settlement Time validate that the initial offer to settle claims took an average of 9 days after a completed claim form was received by Cendant Mobility. Furthermore, the Army’s methodology was limited to examining the time between filing a completed claim form and making the initial settlement offer. This measure does not capture the entire claims process and therefore does not measure the extent to which it represents an improvement over the existing system. The pilot process, however, does offer benefits to the servicemember in terms of full replacement value protection. Household effects are insured for their full replacement value (up to $75,000) rather than their depreciated value (which is capped at $40,000 under the existing system). Page 7 GAO/NSIAD-99-129 Defense Transportation B-282406 The pilot process also requires less work from the servicemember because the contractor provides estimates on lost and damaged household goods and pays the servicemember directly. The claims settlement time measurement is based on the contractual requirement that Cendant Mobility offer settlement to a member within 30 days after receiving a claim for damaged or lost property. Consequently, the Army developed a threshold for claims settlement of 30 days. Under the Hunter pilot, servicemembers could appeal to the contractor for reconsideration by providing additional information or file a claim with the Army if resolution with the contractor could not be achieved—a process that could take more than 2 years to complete. While we found that some claims remained unpaid and unsettled because the claims were put “on hold” by the servicemember, the Army reported that no one in the Hunter pilot filed a claim with the Army claims office as of January 1999. The Army reported that Cendant Mobility took an average of 9 days to offer settlement for lost or damaged property, but neither AAA nor we could confirm this because the Army had not collected the necessary data. AAA determined, and we concur, that Cendant Mobility may have taken as long as an average of 28 days to offer settlement. This figure was based on the time between the date the claim was signed and the date the member was offered settlement. The difference between the two estimates was primarily due to a 19-day period between the date the claim was signed and the date the contractor reported entering the claim into its database. Although Cendant Mobility officials stated that standard company practice is to enter a claim into the database the same day it is received, they said delays could be caused by (1) the form being mailed some time after it is dated, (2) mail delays, and (3) contractor delays in entering the claim into the database. Army officials stated that, due to the difficulties in measuring claims settlement time, they would change their data collection and analysis of this sub-factor in future pilot programs. Direct Delivery Results May Be We could not confirm that all eligible direct deliveries were made without Overstated in-transit storage because the necessary data was not collected. The Army’s methodology for this sub-factor also did not allow an assessment of whether the results represented an improvement over the current process. Direct delivery is defined by the Army as delivery of a shipment to a servicemember’s residence without storage in transit. A shipment is eligible for direct delivery only if it has a destination address before the shipment is offered for delivery. Direct delivery is a contract performance Page 8 GAO/NSIAD-99-129 Defense Transportation B-282406 measurement, and the contractor is paid an incentive for maintaining a direct delivery rate of over 60 percent. Consequently, the Army threshold for this measurement is 60 percent. Neither AAA nor we could confirm that shipments delivered directly to servicemembers were the only ones eligible for direct delivery because the necessary information on all shipments was not collected. That is, other shipments may have been eligible under the Army criteria; however, AAA and we could not determine which shipments had addresses prior to being offered for delivery, and were thus eligible, but ended up requiring in-transit storage. The Army acknowledged the difficulty in validating eligible direct deliveries and stated that, in the future, it would consider using storage-in-transit rates as a test factor. In the February 5, 1999, supplemental information provided to us, the Army stated that 74 percent of Army-wide shipments required in-transit storage, compared with 66 percent requiring temporary storage during the Army pilot. Servicemembers benefit from having household goods delivered directly to their home because the practice limits additional handling of their property, reducing opportunities for loss and damage. Also, the government avoids temporary in-transit storage, which is costly and hard for the Army to control. Household goods shipments require temporary storage when servicemembers do not know their new address at the time of shipment, and/or they move to their new duty station after completing vacations or military requirements en route. Some Overhead Cost Data Was Due to weaknesses in the Army’s methodology for collecting overhead cost Inaccurate and Incomplete data and the questionable reliability of some of the overhead cost data that was collected, we could not validate that the additional cost of the pilot project was 18.6 percent higher than the baseline cost. As we reported in June 1998, developing overhead costs has historically been difficult for the government, including DOD, because such data is often unreliable and unavailable. (See appendix III for a detailed breakdown of pilot and baseline costs as reported by the Army and validated by AAA). Neither AAA nor we could validate some overhead costs. Specifically, we could not confirm the accuracy of the overhead costs attributed to MTMC because the Army used data from fiscal year 1994. MTMC has changed significantly since then due to downsizing, but AAA could not obtain updated costs from MTMC representing the current organization to determine the reasonableness of this overhead cost. AAA and we also agree Page 9 GAO/NSIAD-99-129 Defense Transportation B-282406 that a drop in document processing (for claims, invoices, and inbound shipments) would not result in cost savings to the government unless the activities that perform these functions, such as the Defense Finance and Accounting Service, take corresponding actions to reduce their costs (for example, by reducing personnel to reflect the workload reduction). AAA and we also identified other costs not included in the Army’s analysis such as those related to awarding the Hunter contract and the contractor’s use of foreign-flag vessels.8 The Army could not provide an estimated dollar value associated with the cost of developing the contract solicitation, reviewing offers, and resolving bid protests. While we recognize that some of these estimated costs may be infrequent or one-time only, they should be considered. Also, the Army did not include the contractor’s use of foreign-flag vessels into the pilot project’s transportation costs. While individually, costs such as these are probably of low-dollar value in relation to other costs in the analysis, collectively, they could materially affect the difference between the baseline and pilot costs. According to the Army, some of the reasons for the 18.6 percent higher cost of the pilot project can be attributed to (1) difficulty in calculating accurate baseline costs, which the Army believes were understated; (2) low volume of moves (1,349), which did not provide enough leverage to negotiate better rates and discounts; (3) higher than expected costs of overseas shipments; (4) relatively high unaccompanied baggage shipment rates, which could have been lower using negotiated rates; (5) packing allowance for DITY moves, which was calculated using commercial rates during part of the 12-month test period, resulting in a higher rate; and (6) quality moving services, which cost more. In regard to the high cost of overseas shipments, the Army provided us with additional details of pilot project costs that showed a 44-percent increase for overseas household goods shipments and a 2-percent increase for domestic shipments over the baseline cost. Small Business Usage Exceeded Both AAA and we validated that Cendant Mobility awarded 33 percent of Army Goals the shipments to small businesses—10 percent to carriers and 23 percent to agents. The Army measured the impact of the pilot on these two types of providers and established a threshold of 23 percent to demonstrate successful participation by small businesses.9 The Army based this factor on section 15(g) of the Small Business Act, which establishes a 8 Foreign-flag vessels are ships registered in foreign countries. 9 Inthe motor freight and transportation industry, firms with annual gross revenues of $18.5 million or less are classified as small businesses. See Federal Acquisition Regulation 19.102. Page 10 GAO/NSIAD-99-129 Defense Transportation B-282406 governmentwide goal for participation by small business concerns at not less than 23 percent of the value of all prime contracts. The Army reported that 261 (33 percent) of the 790 Hunter shipments billed during the 12-month pilot were subcontracted by Cendant Mobility to small carriers and agents. As we reported in June 1998, Cendant Mobility encountered problems in attracting small carriers and agents to participate in the Hunter pilot. When the pilot was implemented, several local agents and the companies they represented elected not to sign agreements to participate. Cendant Mobility reported that historically, over 40 percent of the contracts it awards go to small and small and disadvantaged businesses. However, as of December 31, 1997, only 14 percent of the shipments booked by Cendant Mobility during the first 6 months had been awarded to small businesses. By June 30, 1998, a number of national van lines and their local agents had reversed their decision and had signed contracts with Cendant Mobility. No Comprehensive DOD will be running multiple pilots concurrently, each with different goals and objectives and expected outcomes. MTMC reengineering efforts began Strategy for Evaluating in 1994 followed by the Army’s in 1996 and the Navy’s in 1997. DOD is Multiple Pilots proposing a fourth pilot with no specific start date. While USTRANSCOM is tasked with evaluating the results of these pilots and using the information to recommend a redesigned Department-wide relocation program, it does not have a comprehensive strategy for doing so. Specifically, it has not yet determined how many approaches it will ultimately test, it has not set the milestones for completing the pilots’ evaluations and implementing an improved process, and it has not assured itself that a methodologically sound evaluation process is in place to execute this process. MTMC’s Reengineering DOD first proposed reengineering its personal property program in 1994. Efforts Began in 1994 Specifically, on June 21, 1994, USTRANSCOM directed MTMC (the Army component of USTRANSCOM and the program manager for DOD’s Personal Property Shipment and Storage Program) to reengineer the personal property program. On June 15, 1995, the House Committee on National Security,10 concurring that DOD must pursue a higher level of service, directed that DOD undertake a pilot program to implement commercial business practices and standards of service. MTMC planned to 10Now called the House Committee on Armed Services. Page 11 GAO/NSIAD-99-129 Defense Transportation B-282406 award contracts for the new program pursuant to the competitive acquisition system provisions (10 U.S.C. ch. 137 and its implementing regulation in the Federal Acquisition Regulation). Expressing congressional concerns about the impact the competition system and any nonstandard commercial business practice requirements might have on small businesses, the statement of managers in the conference report accompanying the 1997 DOD Authorization Act directed that DOD report on the impact of the pilot program on small businesses. After reviewing the reports, the House Committee on National Security remained concerned that MTMC’s pilot program did not satisfactorily address issues raised by small moving companies and directed that the Secretary of Defense establish a working group of military and industry representatives to develop an alternative pilot program. Although the working group reached consensus on many issues, including a set of program goals, it could not reach agreement on the approach to take for the pilot test. Consequently, the two sides presented separate proposals. In November 1996, we reported that in our assessment, MTMC’s proposal met the goals of reengineering the personal property program to a greater extent than the industry’s plan.11 The MTMC pilot program, delayed by numerous bid protests, finally started in January 1999.12 The pilot includes 50 percent of the moves originating in North Carolina, South Carolina, and Florida. The pilot will run concurrently with the existing MTMC-managed program at installations representing all military services and the Coast Guard in the three states and will involve approximately 18,500 annual moves. The key features of the pilot program include selecting carriers based on servicemember satisfaction and past performance rather than simply price; achieving stronger carrier commitment with long-term contracts; and offering full replacement value protection and direct claims settlement to users. MTMC’s pilot will run for a 3-year test period (1 year and two 1-year option periods) which will end in December 2002. 11Defense Transportation: Reengineering the DOD Personal Property Program (GAO/NSIAD-97-49, Nov. 27, 1996). 12Solicitation disputes in the following decisions: Aalco Forwarding, Inc., et al, B-277241.8, B-277241.9, October 21, 1997; Aalco Forwarding, Inc. et al., B-277241.12, B-277241.13, December 29,1997; Aalco Forwarding, Inc. et al., B-277241.15, March 11, 1998; Aalco Forwarding, Inc. et al., B-277241.16, March 11, 1998; Aalco Forwarding, Inc., et al., B-277241, 277241.14, June 8, 1998; and Aalco Forwarding, Inc., et al., B-277241, B-277241.20, B-277241.21, July 1, 1998. Page 12 GAO/NSIAD-99-129 Defense Transportation B-282406 Hunter Pilot Effort Began in Separately from the MTMC pilot program, the Army decided to determine 1996 whether the commercial business practice of outsourcing a non-core function could alleviate known problems in its personal property program while increasing quality of life. The pilot began in February 1996 as a quality-of-life effort to improve the relocation process and test commercial business practices in a military environment at Hunter Army Airfield, Savannah, Georgia. On January 31, 1997, the Army selected PHH Relocation, a relocation/move management company (now Cendant Mobility, Bethesda, Maryland), as the contractor for the pilot. In July 1997, Cendant Mobility began relocation operations at Hunter, offering to distribute and manage shipments to transportation providers. Additionally, the contractor provided point-to-point move management, personal move counseling and coordination, assistance in buying/selling a residence, 24-hour in-transit visibility of shipments, direct claims settlement with servicemembers, full replacement value, and a single point of contact for servicemembers. These services and benefits are not available under the existing DOD system. The current contract with Cendant Mobility will end on September 30, 1999. A Cendant Mobility official stated that the management fees paid by the Army to the contractor pay for these services and entitlements. Cendant Mobility also obtained revenues from two other sources, including (1) commissions from carriers, agents, and freight forwarders and (2) referral fees from real estate companies on the purchase and sale of six residences. The commissions paid for services provided to carriers, agents, and freight forwarders, including contract training, contract administration, invoice auditing, and quality control services. Navy Pilot Effort Began in The Navy initiated its plans for a separate pilot program in 1997 to test the 1997 option of allowing servicemembers to select their carrier, giving them more control over the relocation process to meet their specific needs. The pilot, which began in January 1998, is intended to provide an alternative to, rather than a replacement of, the current program. The pilot affects a very small number of military personnel. Presently, the option is being offered only for shipments originating in the areas of Puget Sound, Washington; San Diego, California; Norfolk, Virginia; and Groton, Connecticut. Participation is limited to Navy military personnel with permanent change of station orders from the participating sites. Shipments must exceed 3,000 pounds and cost between $2,500 and $25,000 (the pilot excludes shipments from non-temporary storage or warehouses). Servicemembers Page 13 GAO/NSIAD-99-129 Defense Transportation B-282406 can select a mover from a list of carriers that are self-certified as small businesses and approved by MTMC. As of March 26, 1999, 223 Navy servicemembers had selected this option. This pilot does not have a specific end date. DOD Pilot Effort Proposed On February 12, 1999, DOD stated that it intended to begin a fourth pilot in 1999 program, called the Full Service Moving Project, with certain modifications based on lessons learned from the Army Hunter pilot. The new pilot will include the National Capital Region (17,000 annual moves), Georgia (26,000 annual moves), and North Dakota (2,000 annual moves), and will involve all military services and the Coast Guard. Like the Army pilot, it is intended to outsource a non-core function and alleviate known problems in the DOD personal property program while improving the quality of life associated with the relocation of servicemembers and their families. While it is projected to begin in calendar year 1999, presently, this pilot does not have an official start date. Unresolved Issues in USTRANSCOM is in the process of finalizing its evaluation plan for the Evaluating Pilot Programs ongoing and planned pilots. It is proposing to evaluate the pilots on the same three factors that the Army used in its evaluation of the Hunter pilot, except that the factors would be defined differently and would include a more expansive point scale. Unresolved issues, as of February 1999, included the development of a method that would recognize the unique characteristics and/or process improvements of each pilot program and the validation of the baseline indirect costs that would be used for each pilot program. The evaluation plan does not currently include an evaluation of the Hunter pilot, but USTRANSCOM used lessons learned in the Hunter pilot to develop the plan. We provided comments to USTRANSCOM in this process and, among other things, encouraged the command to seek expert methodological advice before finalizing the evaluation plan to enhance the quality of its assessment. Conclusions We support the use of pilots as a means to test new concepts. Although this process has been ongoing since the early 1990s, DOD is not yet in a position to determine what changes it needs to make to the process. Also, thus far, DOD has not decided how many approaches it will ultimately test and has not developed evaluation plans that will yield accurate and credible data. Lessons learned from the Hunter pilot do provide useful information for Page 14 GAO/NSIAD-99-129 Defense Transportation B-282406 evaluating the three other pilots ongoing and planned. While USTRANSCOM is in the process of developing a strategy to conduct this evaluation, it has not yet resolved a number of issues, including timelines for completing the analysis and implementing an improved process and a methodology for the evaluation. Recommendations We recommend that the Secretary of Defense direct the Commander in Chief, USTRANSCOM, to develop a comprehensive strategy for evaluating each of the pilot approaches that includes (1) tests of a finite number of approaches; (2) timelines for completing the analysis and for implementing a new personal property process; and (3) milestones for finalizing a methodologically sound evaluation plan, including specific criteria for assessing each pilot’s attributes and validating the baseline indirect costs to be used for each pilot program. To facilitate this process and to enhance the quality of the assessment, we also recommend that DOD seek expert methodological advice before finalizing the evaluation plan. Agency Comments and In written comments on a draft of this report, DOD stated that it generally concurred with the report and its recommendations. DOD tasked Our Evaluation USTRANSCOM to evaluate the results of all the pilots and provide a recommendation for a DOD-wide program to implement for the long term. DOD stated that USTRANSCOM is in the process of retaining a contractor to independently validate success criteria for a reengineered DOD personal property program, develop a comprehensive strategy for evaluating each of the pilot approaches, and support the development of recommendations and alternative courses of action. USTRANSCOM anticipates having the contractor commence its efforts by June 1999. DOD’s comments are presented in appendix IV. DOD also provided technical comments and we revised our report to reflect them where appropriate. Scope and To validate the results and savings of the Hunter pilot, we reviewed the Army’s plans to determine how it would evaluate the pilot’s results. We also Methodology reviewed the Army’s October 1998 report and the February and March 1999 supplements detailing the 12-month test results. We visited both Hunter Army Airfield and the transportation division at Fort Stewart, Georgia. We contacted Hunter contractor officials and representatives from the two Page 15 GAO/NSIAD-99-129 Defense Transportation B-282406 survey firms hired by the Army and USTRANSCOM to measure customer satisfaction. We also examined the documents and data compiled by the AAA to validate the accuracy of the Hunter results. We met with Army, Navy, MTMC, and USTRANSCOM officials and reviewed documents to determine the status and goals of other DOD personal property pilot programs. We also obtained statistics on actual use by servicemembers of the Navy’s pilot as of March 1999. We did not request statistics on other pilots because the MTMC pilots began in January 1999 and the DOD-proposed Full Service Moving Project is still in the planning stage. We also met with USTRANSCOM officials and reviewed evaluation plans to determine how DOD proposes to make changes to its current personal property program using lessons learned from the pilots. We met and discussed matters related to DOD pilots and efforts to improve the existing personal property program with officials from the Office of the Deputy Under Secretary of Defense (Logistics-Transportation Policy), Washington, D.C.; USTRANSCOM, Scott Air Force Base, Illinois; the Office of the Army Deputy Chief of Staff for Logistics, Washington, D.C.; MTMC, Falls Church, Virginia; the Naval Supply Systems Command, Mechanicsburg, Pennsylvania; Hunter Army Airfield, Savannah, Georgia; and Fort Stewart, Georgia. In addition, we discussed the survey results with officials from Battelle’s Pacific Northwest National Laboratory and from Electronic Data Systems. We conducted our review between August 1998 and April 1999 in accordance with generally accepted government auditing standards. We are sending copies of this report to the Honorable William S. Cohen, Secretary of Defense; the Honorable Louis Caldera, Secretary of the Army; General Charles T. Robertson, Jr., Commander in Chief, USTRANSCOM; Lieutenant General John G. Coburn, Deputy Chief of Staff, Logistics, Department of the Army; Major General Mario F. Montero, Jr., Commander, MTMC; and the Honorable Jacob J. Lew, Director, Office of Management and Budget. We are also sending copies to other congressional members who expressed interest to us in DOD’s efforts to improve its personal property program, including the Honorable Fred Thompson, United States Senate, and the Honorable Neil Abercrombie, Duncan Hunter, Martin T. Meehan, and James M. Talent, House of Representatives. We will also make copies available to others upon request. Page 16 GAO/NSIAD-99-129 Defense Transportation B-282406 Please contact me on (202) 512-8412 if you or your staff have questions concerning this report. Major contributors to this report are listed in appendix V. David R. Warren, Director Defense Management Issues Page 17 GAO/NSIAD-99-129 Defense Transportation B-282406 List of Congressional Committees The Honorable John W. Warner Chairman The Honorable Carl Levin Ranking Minority Member Committee on Armed Services United States Senate The Honorable Ted Stevens Chairman The Honorable Daniel K. Inouye Ranking Minority Member Subcommittee on Defense Committee on Appropriations United States Senate The Honorable Floyd D. Spence Chairman The Honorable Ike Skelton Ranking Minority Member Committee on Armed Services House of Representatives The Honorable Herbert H. Bateman Chairman The Honorable Solomon P. Ortiz Ranking Minority Member Subcommittee on Military Readiness Committee on Armed Services House of Representatives The Honorable Jerry Lewis Chairman The Honorable John P. Murtha Ranking Minority Member Subcommittee on Defense Committee on Appropriations House of Representatives Page 18 GAO/NSIAD-99-129 Defense Transportation Page 19 GAO/NSIAD-99-129 Defense Transportation Contents Letter 1 Appendix I 22 Thresholds and Scores Used by the Army for Hunter Pilot Evaluation Factors Appendix II 23 Types and Number of Shipments Made During the Hunter Pilot Test Appendix III 24 Hunter Pilot and Baseline Costs Calculated by the Department of the Army (Office of the Deputy Chief of Staff for Logistics) Appendix IV 25 Comments From the Department of Defense Appendix V 28 Major Contributors to This Report Page 20 GAO/NSIAD-99-129 Defense Transportation Contents Related GAO Products 32 Abbreviations AAA Army Audit Agency DITY do-it-yourself DOD Department of Defense EDS Electronic Data Systems MTMC Military Traffic Management Command USTRANSCOM U.S. Transportation Command Page 21 GAO/NSIAD-99-129 Defense Transportation Appendix I Thresholds and Scores Used by the Army for Hunter Pilot Evaluation Factors ApIenx di Factors/definitions Thresholds Scores Quality of life/customer satisfaction rating: percent of Hunter respondents who were 25% and above 1.00 point more satisfied than Fort Stewart respondents (control group). 20-24% 0.75 points 15-19% 0.50 points 10-14% 0.25 points Below 10% Unsuccessful Quality of life/claims settlement time: time period between when a claim was received 0-10 days 1.00 point and when the contractor offered settlement. 11-15 days 0.75 points 16-20 days 0.50 points 20-30 days 0.25 points Above 30 days Unsuccessful Quality of life/percentage of eligible deliveries: percent of shipments delivered to 91-100% 1.00 point residence without storage in-transit. 81-90% 0.75 points 71-80% 0.50 points 61-70% 0.25 points 60% and below Unsuccessful Total cost: extent total contract cost exceeded total baseline cost. 0% 1.00 point 1-10% 0.75 points 11-20% 0.50 points 21-30% 0.25 points More than 30 % Unsuccessful Impact on small business: percent of small agents and carriers used. 20% or more 1.00 point 27-28% 0.75 points 25-26% 0.50 points 23-24% 0.25 points Less than 23% Unsuccessful Page 22 GAO/NSIAD-99-129 Defense Transportation Appendix II Types and Number of Shipments Made During the Hunter Pilot Test ApIpx di Ien Shipment type Shipments initiated and recorded as completed Household goods (intrastate – Georgia 51 Household goods (interstate – continental U.S., including Alaska) 418 Household goods (overseas – Hawaii) 29 Household goods (overseas – Germany) 79 Household goods (overseas – Korea) 50 Household goods (overseas – other) 47 Unaccompanied baggage (overseas – all locations) 378 Do-it-yourself (DITY) moves 292 Mobile homes 5 Total 1,349 Page 23 GAO/NSIAD 99-129 Defense Transportation Appendix III Hunter Pilot and Baseline Costs Calculated by the Department of the Army (Office of the Deputy Chief of Staff for Logistics) AIpIenx di Cost elementsa Estimated pilot costs Estimated baseline costs Transportation costs Transportation $3,005,229 $2,102,384 Accessorials 25,767 30,655 Storage 210,706 211,655 Total transportation costs $3,241,702 $2,344,694 Claims paid by Army Claims paid by Army $0 $66,268 Overhead costs Personnel $79,802 $221,721 Management price 132,700 0 MTMC headquartersb 0 63,510 Automation 0 52,612 Voucher processing 4,614 90,658 Inbound processing 0 46,834 Claims processing 0 11,510 Pay for performance 14,756 0 Building overhead 28,321 43,286 Telephone and copier 1,281 10,106 Consumables 3,760 5,189 Total overhead costs $265,234 $545,426 Total costs $3,506,936 $2,956,388 Pilot Baseline Percent costs costs Difference difference Percent pilot costs exceeded baseline costs $3,506,936 $2,956,388 $550,548 18.6 a Army Audit Agency validated all costs except those under $5,000, concluding that such amounts were immaterial. bMilitary Traffic Management Command (MTMC). Source: Office of the Deputy Chief of Staff for Logistics, Department of Army. Page 24 GAO/NSIAD-99-129 Defense Transportation Appendix IV Comments From the Department of Defense ApIV ex pdi n Note: GAO comment supplementing those in the report text appear at the end of this appendix. See comment 1. Page 25 GAO/NSIAD-99-129 Defense Transportation Appendix IV Comments From the Department of Defense Page 26 GAO/NSIAD-99-129 Defense Transportation Appendix IV Comments From the Department of Defense The following are GAO’s comments on the Department of Defense’s letter dated May 11, 1999. GAO Comment 1. As stated in the report, we could not confirm the customer satisfaction survey results. Specifically, we could not confirm that all survey results indicated higher levels of customer satisfaction because of weaknesses in the survey methodology. Methodological weaknesses that compromised survey results included using different populations and respondent rates, using various survey instruments with different questions, and surveying servicemembers multiple times. In addition, one of the two survey contractors found no statistically significant difference between responses from Hunter and Fort Stewart. Our assessment of customer satisfaction results, as indicated in the report, was based on the Army’s criteria set forth in its evaluation plan. Because the Army used the results of one question to score the pilot a success, we also used it as the basis for our assessment. We agree, however, that other data obtained by the surveys provided indicators of customer satisfaction and highlighted areas that could be addressed in future evaluations. We agree that the bulk of cost elements could be validated and revised our report to show that some overhead cost data was inaccurate and incomplete. Finally, regarding our assessment of the claims process, we did not acknowledge a significant improvement in the process because the Army measured only one aspect of it, as indicated in the report. In addition, on the basis of the Army’s criteria set forth in its evaluation plan for assessing this factor, we could not confirm the time it took to offer claims settlement to servicemembers. We also agree that the claims process offered benefits not previously available to the servicemembers. Page 27 GAO/NSIAD-99-129 Defense Transportation Appendix V Major Contributors to This Report Appx end Vi National Security and Charles I. Patton, Jr., Associate Director Nomi R. Taslitt, Assistant Director International Affairs Robert L. Self, Evaluator-in-Charge Division, Washington, Marc J. Schwartz, Senior Evaluator Jacqueline E. Snead, Senior Evaluator D.C. Richard R. Irving, Evaluator Arthur L. James, Jr., Mathematical Statistician Office of the General John G. Brosnan, Assistant General Counsel Counsel, Washington, D.C. Norfolk Field Office Daniel A. Omahen, Senior Evaluator John R. Beauchamp, Senior Evaluator Page 28 GAO/NSIAD-99-129 Defense Transportation Page 29 GAO/NSIAD-99-129 Defense Transportation Page 30 GAO/NSIAD-99-129 Defense Transportation Page 31 GAO/NSIAD-99-129 Defense Transportation Related GAO Products Defense Transportation: Efforts to Improve DOD’s Personal Property Program (GAO/T-NSIAD-99-106, Mar. 18, 1999). Defense Transportation: The Army’s Hunter Pilot Project to Outsource Relocation Services (GAO/NSIAD-98-149, June 10, 1998). Defense Outsourcing: Better Data Needed to Support Overhead Rates for A-76 Studies (GAO/NSIAD-98-62, Feb. 27, 1998). Defense Transportation: Reengineering the DOD Personal Property Program (GAO/NSIAD-97-49, Nov. 27, 1996). Household Goods: Administrative Changes Would Improve DOD’s Do-It- Yourself Moving Program (GAO/NSIAD-94-226, Sept. 27, 1994). DOD Commercial Transportation: Savings Possible Through Better Audit and Negotiation of Rates (GAO/NSIAD-92-61, Dec. 27, 1991). Household Goods: Competition Among Commercial Movers Serving DOD Can Be Improved (GAO/NSIAD-90-50, Feb. 12, 1990). (709365) Leter Page 32 GAO/NSIAD-99-129 Defense Transportation Ordering Information The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary, VISA and MasterCard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent. Orders by mail: U.S. General Accounting Office P.O. Box 37050 Washington, DC 20013 or visit: Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000 or by using fax number (202) 512-6061, or TDD (202) 512-2537. Each day, GAO issues a list of newly available reports and testimony. To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone. A recorded menu will provide information on how to obtain these lists. For information on how to access GAO reports on the INTERNET, send an e-mail message with “info” in the body to: firstname.lastname@example.org or visit GAO’s World Wide Web Home Page at: http://www.gao.gov United States Bulk Rate General Accounting Office Postage & Fees Paid Washington, D.C. 20548-0001 GAO Permit No. GI00 Official Business Penalty for Private Use $300 Address Correction Requested
Defense Transportation: The Army's Hunter Pilot Project Is Inconclusive but Provides Lessons Learned
Published by the Government Accountability Office on 1999-06-23.
Below is a raw (and likely hideous) rendition of the original report. (PDF)