United States Government Accountability Office Report to the Ranking Member, Permanent Subcommittee on Investigations, Committee on Homeland Security and Governmental Affairs, U.S. Senate U.S. POSTAL March 2019 SERVICE Addressing Policy Gaps Could Improve Pilot Design and Evaluation for Postal Innovations GAO-19-293 March 2019 U.S. POSTAL SERVICE Addressing Policy Gaps Could Improve Pilot Design and Evaluation for Postal Innovations Highlights of GAO-19-293, a report to the Ranking Member, Permanent Subcommittee on Investigations, Committee on Homeland Security and Governmental Affairs, U.S. Senate Why GAO Did This Study What GAO Found USPS faces a challenging business From fiscal years 2013 through 2017, the U.S. Postal Service (USPS) piloted 24 environment that has led to reduced key innovations intended primarily to generate revenue or improve customers’ demand for its traditional services and experience. The following four selected innovations illustrate these efforts: significant financial losses. USPS aims to address this challenge by offering • Same-Day Delivery: USPS delivered goods consumers bought online or in innovative products and services. The stores. The pilot sought to test the product’s feasibility and revenue potential. success of these efforts will depend, in • Grocery Delivery: USPS delivered groceries to consumers in metropolitan part, on how effectively USPS tests areas. The pilot sought to test the product’s feasibility and revenue potential. each innovation’s performance on a • Informed Delivery: USPS emailed customers an advance image of the mail small scale to determine whether, how, they would receive. The pilot sought to test the service’s potential benefits, and when to launch an innovation more such as generating new revenue from advertisers that may use the service. broadly—a practice known as “piloting.” • Keyless Parcel Lockers: USPS is testing lockers where customers can GAO was asked to review USPS’s independently pick up packages at post offices. The pilot seeks to test the efforts to develop postal innovations. service’s operation and potential benefits for USPS and customers. This report (1) describes key (From Left to Right) Metro Post (same-day delivery), Grocery Delivery Tote Bag, Informed innovations that USPS recently piloted Delivery Email Notification, and Keyless Parcel Locker Unit and (2) examines the extent to which USPS's policies reflect leading practices for pilot design and evaluation. GAO analyzed information on USPS pilots from fiscal years 2013 through 2017; compared USPS policies for piloting innovations to leading practices for pilot design and evaluation in prior GAO work and relevant standards for internal control; and selected four key innovations based on various characteristics (e.g., innovation type) to USPS’s policies for piloting innovations do not fully reflect the five leading serve as illustrative examples of USPS’s practices for pilot design and evaluation identified in GAO’s prior work. The piloting efforts. policies fully reflect two of the leading practices because they require articulating What GAO Recommends a methodology for evaluating pilot performance and documenting lessons learned. The policies do not fully reflect the other three practices because they GAO recommends that USPS (1) do not require: (1) linking pilot objectives to identified performance measures; (2) develop policies that fully reflect leading documenting conclusions based on pilot results; or (3) communicating with key practices for pilot design and evaluation external stakeholders, as appropriate. These policy gaps limit the extent to which and (2) develop tools or training to USPS can ensure that it is making good resource allocation decisions based on ensure consistent documentation of pilot experiences. For example, GAO found that USPS did not document its lessons learned from pilots. USPS conclusions based on the results of its pilots of same-day delivery, grocery neither agreed nor disagreed with the delivery, and Informed Delivery. Documenting conclusions can be especially recommendations but described actions it plans to take related to each. important when USPS continues to offer the product or service after the pilot has concluded, even though the pilot did not achieve all of its objectives, as was the case with these three innovations. Further, while USPS’s policies require documenting lessons learned from its pilots, USPS did not do so for some pilots GAO reviewed. Senior USPS officials said that USPS did not consistently follow this policy because it had not developed tools or training that could help ensure View GAO-19-293. For more information, such consistency. As a result, USPS risks losing information that could be contact Lori Rectanus at (202) 512-2834 or relevant to future innovation efforts. RectanusL@gao.gov. ______________________________________ United States Government Accountability Office Contents Letter 1 Background 4 USPS Piloted Key Innovations Are Intended Primarily to Generate Revenue or Improve Customers’ Experience 8 USPS’s Policies for Piloting Key Innovations Reflect Some but Not All Leading Practices for Pilot Design and Evaluation 15 Conclusions 20 Recommendations for Executive Action 21 Agency Comments and Our Evaluation 21 Appendix I List of U.S. Postal Service Key Piloted Innovations, Fiscal Years 2013-2017 24 Appendix II Comments from the U.S. Postal Service 26 Appendix III GAO Contact and Staff Acknowledgments 28 Tables Table 1: Extent to Which the U.S. Postal Service’s (USPS) Policies for Pilot Design and Evaluation Reflect Leading Practices 15 Table 2: Extent to Which the U.S. Postal Service (USPS) Documented Lessons Learned for Key Piloted Innovations Selected by GAO 17 Table 3: U.S. Postal Service (USPS) Key Piloted Innovations, Fiscal Years 2013 through 2017 24 Figures Figure 1: U.S. Postal Service (USPS) Total Operating Revenue and Total Operating Expenses, Fiscal Years 2007 through 2017 5 Figure 2: Key Piloted U.S. Postal Service (USPS) Innovations by Primary Goal, Fiscal Years 2013 through 2017 9 Figure 3: Snapshot of U.S. Postal Service Website for Same-Day Delivery Product Innovation 10 Page i GAO-19-293 Postal Innovations Figure 4: Illustration of a Tote Used for U.S. Postal Service Grocery Delivery Product Innovation 11 Figure 5: Snapshot of U.S. Postal Service Informed Delivery Notification 13 Figure 6: Photo of U.S. Postal Service Keyless Parcel Locker Unit 14 Page ii GAO-19-293 Postal Innovations Abbreviations MTAC Mailers’ Technical Advisory Committee PRC Postal Regulatory Commission USPS U.S. Postal Service This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. Page iii GAO-19-293 Postal Innovations 441 G St. N.W. Washington, DC 20548 March 14, 2019 The Honorable Thomas R. Carper Ranking Member Permanent Subcommittee on Investigations Committee on Homeland Security and Governmental Affairs United States Senate Dear Mr. Carper: The U.S. Postal Service (USPS) faces a challenging business environment that has reduced demand for traditional postal services and led to significant financial losses. Revenue from USPS’s most profitable postal-service category—First-Class Mail—has declined over the last decade, from $38.2 billion in fiscal year 2008 to $25 billion in fiscal year 2018. According to USPS, offering innovative products and services is critical to adapting to evolving business and customer needs and improving its financial condition. The success of any particular innovation will depend, in part, on how effectively USPS tests the innovative product’s or service’s performance on a small scale to determine whether, how, and when to launch the innovation more broadly—a practice known as “piloting.” Effectively piloting innovations can help ensure that USPS invests its limited resources on innovations that are most likely to improve its long-term viability. You asked us to review USPS’s efforts to develop innovative products and services. This report (1) describes key innovations USPS has recently piloted and (2) examines the extent to which USPS’s policies for piloting key innovations reflect leading practices for pilot design and evaluation. 1 To identify key innovations that USPS has recently piloted, we first requested information from USPS on all such innovations piloted between 1 For the purpose of this report, we focused on “key innovations,” which USPS defines as new products and services that may have a significant effect on cost, the end consumer, the shipper, or the mailer. Page 1 GAO-19-293 Postal Innovations fiscal years 2013 and 2017. 2 In response, USPS identified 24 key piloted innovations. We compared the list provided by USPS against information in USPS, USPS Office of Inspector General, and Postal Regulatory Commission (PRC)—an independent agency with regulatory oversight over USPS—documents to help corroborate the completeness and accuracy of USPS’s response. We conducted an in-depth review of 4 of the 24 innovations to provide illustrative examples of USPS’s piloting efforts. We selected the 4 innovations—Metro Post (i.e., same-day delivery), Customized Delivery (i.e., grocery delivery), Informed Delivery, and keyless parcel lockers—to represent a range of the innovations’ primary characteristics, such as innovation type (e.g., new product or new service); primary innovation goal (e.g., generating revenue or improving customer service); and investment level. 3 To evaluate the extent to which USPS’s policies for piloting key innovations reflect leading practices for pilot design and evaluation, we compared policies identified by USPS officials as applicable to its efforts to pilot innovative products and services 4 against five leading practices we identified in our prior work, as well as standards for internal control 2 We selected fiscal year 2013 because it corresponds with the fiscal year that USPS issued its Five-Year Business Plan to return USPS to financial and operational viability, including actions to confront revenue declines through innovation. We selected fiscal year 2017 because it was the most recent fiscal year for which USPS had available information on piloted innovations at the time of our review. 3 A postal “product” is statutorily defined as a postal service with a distinct cost or market characteristic for which a rate or rates are, or may reasonably be, applied, and a “postal service” refers to the delivery of letters, printed matter, or mailable packages, including acceptance, collection, sorting, and transportation, or other ancillary function. See 39 U.S.C. §§ 102(5), (6). In addition, the Postal Service was statutorily authorized, subject to Postal Regulatory Commission approval, to provide “nonpostal services” offered as of January 1, 2006. In general, the term “nonpostal service” is defined by statute to mean any service that is not a “postal service.” See 39 U.S.C. § 404(e)(1). We did not independently evaluate whether USPS’s categorization of each innovation aligned with these definitions. For the purpose of this report, we collectively refer to USPS’s three types of key innovations—new service, new product, or enhancement to an existing product or postal/nonpostal service—as “innovative products and services.” We are not reporting specific investment, cost, revenue, or volume data for piloted innovations because USPS considers this information to be proprietary. 4 USPS, Handbook F-66, General Investment Policies and Procedures (Sept. 2018) and Handbook F-66D, Investment Policies and Procedures — Business Initiatives, Alliances, Real Estate Development, and Major Operating Expense Investments (February 2006). Page 2 GAO-19-293 Postal Innovations related to documenting key information: 5 The five practices we identified include: • establish appropriate and measurable objectives linked with identified performance measures; • articulate a methodology for evaluating pilot performance; • evaluate pilot performance and identify and document lessons learned; • draw and document conclusions about scalability (i.e., determining whether, how, and when to launch the innovation more broadly) based on pilot results; and • ensure appropriate two-way communication at all stages of the pilot with key internal and external stakeholders in order to understand and address their views. We discussed these leading practices with senior USPS officials, and they agreed that the practices were reasonable and relevant to USPS’s efforts to pilot innovative products and services. We examined how USPS followed its policies and applied these leading practices for the four selected key innovations by reviewing the pilot 5 Our prior work identifies leading practices for designing and evaluating pilot programs. In our prior work, we have applied these practices to multiple federal agencies and programs. For the purpose of this report, we made some modifications to these leading practices. For example, we modified the leading practice related to evaluating pilot performance to make explicit that the leading practice includes documenting lessons learned. For the prior work that we reviewed to identify these five leading practices, see GAO, Tax Administration: IRS Needs to Strengthen Its Approach for Evaluating the SRFMI Data-Sharing Pilot Program, GAO-09-45 (Washington, D.C.: Nov. 7, 2008); DATA Act: Section 5 Pilot Design Issues Need to Be Addressed to Meet Goal of Reducing Recipient Reporting Burden, GAO-16-438 (Washington, D.C.: Apr. 19, 2016); Performance Partnerships: Agencies Need to Better Identify Resource Contributions to Sustain Disconnected Youth Pilot Programs and Data to Assess Pilot Results, GAO-17-208 (Washington, D.C.: Apr. 18, 2017); and Identity Theft: Improved Collaboration Could Increase Success of IRS Initiatives to Prevent Refund Fraud, GAO-18-20 (Washington, D.C., Nov. 28, 2017). For standards for internal control that we reviewed, see GAO, Standards for Internal Control in the Federal Government, GAO-14-704G (Washington, D.C.: Sept. 10, 2014), Principle 3, Paragraph 9, and Committee of Sponsoring Organizations of the Treadway Commission, Internal Control- Integrated Framework (2013), Chapter 4, Additional Considerations, Documentation. Page 3 GAO-19-293 Postal Innovations proposals and revenue and other performance data. 6 Our findings related to these four selected key innovations are not generalizable to all innovations piloted by USPS but provide illustrative examples of how USPS has followed its policies and applied the leading practices that we identified. For both objectives, we also interviewed USPS officials and representatives from four postal associations knowledgeable about USPS’s efforts to develop innovative products and services, as well as two mailers directly affected by USPS’s Informed Delivery innovation. 7 We conducted this performance audit from December 2017 to March 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Background USPS’s Financial Position USPS’s mission is to provide universal postal service while operating as a and Strategic Goals self-financing entity, but USPS’s current financial position is not sustainable. To achieve its mission, USPS must cover its expenses through revenues generated from the sale of its products and services. However, USPS’s total operating expenses have exceeded total operating revenue each year since fiscal year 2007, including a $2.6 billion loss from operations in fiscal year 2017 alone (see fig. 1). Moreover, we have reported that USPS’s overall financial condition is 6 As we discuss later in this report, USPS had not concluded the pilot for one of the four key innovations we selected for review. Consequently, for this innovation, we did not compare USPS’s application of two leading practices: evaluate pilot performance and identify and document lessons learned and draw and document conclusions about scalability based on pilot results. 7 We did not interview mailers for other innovations that we selected either because representatives of mailers that worked directly with USPS during the pilots of these innovations were no longer available or because individual mailers were not directly affected by the innovation. Page 4 GAO-19-293 Postal Innovations deteriorating. 8 For example, in August 2018 we reported that USPS had about $149 billion in unfunded liabilities and debt at the end of fiscal year 2017. 9 As a result, USPS’s financial condition remains on our list of high- risk areas needing attention by Congress and the executive branch. 10 Figure 1: U.S. Postal Service (USPS) Total Operating Revenue and Total Operating Expenses, Fiscal Years 2007 through 2017 8 For example, see GAO, U.S. Postal Service: Key Considerations for Restoring Fiscal Sustainability, GAO-17-404T (Washington, D.C.: Feb. 7, 2017) and U.S. Postal Service: Action Needed to Address Unfunded Benefit Liabilities, GAO-14-398T (Washington, D.C.: Mar. 13, 2014) 9 GAO, Postal Retiree Health Benefits: Unsustainable Finances Need to Be Addressed, GAO-18-602 (Washington, D.C.: Aug. 31, 2018). 10 GAO, High-Risk Series: Progress on Many High-Risk Areas, While Substantial Efforts Needed on Others, GAO-17-317 (Washington, D.C.: Feb. 15, 2017), p.130. We added USPS’s financial condition to our list of high-risk areas in July 2009. Page 5 GAO-19-293 Postal Innovations According to USPS financial documents, its ability to sell innovative products and services will be a key factor in improving its financial condition. Thus, USPS established a strategic goal to “innovate faster to deliver value” to its customers, by making investments in innovations that respond to rapidly evolving customer needs. 11 A key element of this effort is to accelerate testing of innovative products and services to better serve these needs, according to USPS. While USPS is allowed to develop certain new postal products and services, there are statutory restrictions that currently limit the range of innovations USPS can offer. For example, under current statute, USPS is not permitted to ship alcoholic beverages. 12 Similarly, although USPS is explicitly authorized to provide services to federal executive agencies (e.g., passport services), such authorization does not include services to state, local, and tribal governments. Legislation has been introduced in previous sessions of Congress that would permit USPS to deliver alcoholic beverages and allow USPS to provide property and services to state, local, and tribal governments under certain conditions. 13 According to USPS officials, USPS supports these legislative proposals, which could enhance its ability to offer innovative products and services. USPS Postal Innovation According to USPS officials, two USPS handbooks include policies Pilot Policies and applicable to piloting key innovations. Specifically, the first handbook includes requirements, procedures, and responsibilities for all types of Responsibilities investment programs and projects undertaken by USPS, regardless of size, cost, or complexity. 14 This handbook, for example, requires the identification and documentation of lessons learned for all investments and projects. The second handbook includes requirements and 11 USPS, Future Ready: U.S. Postal Service Five-Year Strategic Plan Fiscal Years 2017 to 2021 (Sept. 30, 2016). 12 18 U.S.C. § 1716(f). 13 For provision of property and services to state, local, and tribal governments, see e.g., S. 1486, § 302, 113th Cong (2013); S. 2629, § 203, 115th Cong (2018); H.R. 756, § 204, th th 115 Cong (2017); and H.R. 6076, § 204, 115 Cong (2018). In addition, for shipment of alcoholic beverages, see e.g., S. 1486, § 303, 113th Cong (2013) and S. 2629, § 204, 115th Cong (2018). 14 USPS, Handbook F-66, General Investment Policies and Procedures (Washington, D.C.: September 2018). Page 6 GAO-19-293 Postal Innovations procedures specifically for major operating expense investments. 15 This handbook, among other things, establishes requirements and procedures meant to ensure that new and enhanced products and services consistently meet customer needs, generate new revenue, and strengthen USPS as a business. This handbook further states that USPS has a responsibility to subject new initiatives to rigorous financial analysis, testing, and measurement, to determine whether these initiatives will make a positive financial contribution to the organization and ensure that USPS’s leadership has appropriate information for effective decision- making. USPS’s Office of Product Innovation generally has lead responsibility for piloting innovations. 16 According to USPS’s policies, a project manager is responsible for establishing and coordinating a cross-functional team to design and evaluate the pilot. This team typically includes officials from a variety of USPS departments, such as finance, general counsel, information technology, marketing, and operations. The project manager, with support from the cross-functional team, is responsible for preparing a proposal for the pilot that includes key information, such as the pilot’s objectives and performance measures, and overseeing pilot implementation and communication with key stakeholders. In some cases, PRC has a role in overseeing postal innovation pilots. 17 For example, USPS must notify PRC before it pilots any postal product innovation for which it will impose a price (i.e., a pilot that generates revenue for USPS) and must subsequently report quarterly revenue, 15 USPS, Handbook F-66D, Investment Policies and Procedures — Business Initiatives, Alliances, Real Estate Development, and Major Operating Expense Investments (Washington, D.C.: February 2006). A major operating expense investment is an investment associated with a new initiative, project, or program and may include ongoing operating expenses associated with the initiative, as well as capital expenditures that may fall under $5 million. 16 According to USPS officials, other USPS offices have lead responsibility for piloting some innovations. For example, the Office of the Chief Information Officer may have lead responsibility for piloting information technology-based innovations. 17 The PRC is composed of five commissioners and exercises regulatory oversight over USPS. It was created by the Postal Reorganization Act of 1970, with expanded responsibilities under the Postal Accountability and Enhancement Act of 2006. We did not assess actions taken by PRC to consider and approve innovation pilots, including grocery delivery and same-day delivery. Page 7 GAO-19-293 Postal Innovations volume, and cost data. 18 PRC also ensures that certain safeguards are maintained during the pilot, such as limitations on the pilot’s duration and revenue. However, according to PRC officials, the commission has limited involvement in other areas of USPS’s efforts to develop innovations. USPS piloted 24 key innovations from fiscal years 2013 through 2017. 19 USPS Piloted Key For example, USPS piloted an Identity Verification Service that allows Innovations Are users to verify their identity either remotely (i.e., online) or in person at a postal facility. Similarly, USPS piloted an innovation to allow mailers to Intended Primarily to print shipping labels, track packages, and schedule package pick-ups by Generate Revenue or accessing USPS data. The primary goal of the majority of these key innovations (16 of 24) was to generate revenue, while the primary goal for Improve Customers’ the remaining innovations was generally to improve customers’ Experience experience using USPS products or services (see fig. 2). Appendix I includes a complete list of key innovations USPS piloted from fiscal years 2013 through 2017. 18 See 39 U.S.C. § 3641, 39 C.F.R. Part 3035. In September 2014, PRC issued a final rule for conducting “market tests” of “experimental products” that it regulates See Market Tests of Experimental Products, 79 Fed. Reg. 54552 (Sept. 11, 2014). We refer to these tests as “pilots” and products as “postal product innovations” for purposes of our report. In January 2019, PRC issued an order amending these rules to revise the method for calculating applicable revenue limitations, among other things. See PRC Order Amending Rules Relating to Market Tests, Order No. 4973 (Jan. 8, 2019). 19 USPS officials noted that USPS implemented a variety of other key innovations during this time frame, but did not pilot them because the investments carried relatively low risk or implementation cost. For example, USPS introduced High Density Plus, a product enhancement for mailers delivering more than 300 pieces on a single mail route. Similarly, USPS offered annual-mailing promotions (i.e., postal rate discounts offered over a limited period to qualifying mailings) intended to create awareness of innovative uses of mail. Page 8 GAO-19-293 Postal Innovations Figure 2: Key Piloted U.S. Postal Service (USPS) Innovations by Primary Goal, Fiscal Years 2013 through 2017 a USPS stated that the primary goal of one innovation was to improve internal infrastructure. The following discussion provides additional information about the 4 key innovations we selected as illustrative examples of USPS’s efforts to pilot innovative products and services. • Same-Day Delivery: From December 2012 to December 2015, USPS piloted same-day delivery for consumer e-commerce purchases (see fig. 3). According to the pilot proposal, this innovation was intended to generate revenue for USPS by allowing it to leverage its existing delivery infrastructure to capture part of the growing e-commerce market. To determine the potential scalability of same-day delivery, USPS first tested its operational feasibility and potential demand in several major metropolitan areas, including San Francisco, New York, and Phoenix. During the pilot, USPS delivered photos, chocolates, water, electronics, and other goods from 38 participating mailers to consumers in these areas. At the pilot’s conclusion, USPS decided to continue offering same-day delivery to interested participating mailers under Priority Mail contracts. 20 20 According to USPS, as of October 2018, it has signed nine Priority Mail contracts with eight mailers that participated in the pilot. Priority Mail provides flat-rate 1- to 3-day expected delivery for domestic parcels. Page 9 GAO-19-293 Postal Innovations Figure 3: Snapshot of U.S. Postal Service Website for Same-Day Delivery Product Innovation • Grocery Delivery: From November 2014 to October 2017, USPS piloted a grocery delivery product in nine selected metropolitan areas. 21 According to USPS, the innovation was intended to generate additional revenue by taking advantage of the growing market for grocery delivery. To test the innovation’s operational feasibility, USPS required the pilot’s sole participating mailer to bring totes containing 21 Metropolitan areas included San Francisco, Los Angeles, San Diego, New York, Sacramento, Stamford, Philadelphia, Boston, and the Washington/Baltimore capital region. Page 10 GAO-19-293 Postal Innovations groceries and other prepackaged goods ordered by customers directly to post offices (see fig. 4). USPS was then responsible for sorting the totes and delivering them to customers. According to its proposal for this pilot, USPS expected grocery delivery to provide a substantial revenue generation opportunity. At the pilot’s conclusion, like same- day delivery, USPS decided to continue offering grocery delivery with the participating mailer under a Parcel Select contract. 22 Figure 4: Illustration of a Tote Used for U.S. Postal Service Grocery Delivery Product Innovation 22 Specifically, USPS signed a Parcel Select contract with the participating mailer in October 2017 that was reviewed and approved by the PRC. According to USPS, Parcel Select is a ground delivery product for bulk package shipments. Page 11 GAO-19-293 Postal Innovations • Informed Delivery: From spring 2014 through July 2016, USPS piloted a notification service called Informed Delivery in Northern Virginia and New York. 23 According to USPS, this innovation is intended to bridge the gap between the physical and digital worlds by, for example, emailing customers with a scanned image of the exterior address side of letter-sized mail they should receive later that day (see fig. 5). Informed Delivery can also allow mailers to conduct marketing campaigns by integrating other elements—such as hyperlinks to mailers’ websites—into the email and other notifications that customers receive. In its proposal to pilot Informed Delivery, USPS stated the pilot was intended to help USPS understand the service’s business opportunity and increase the certainty of its potential benefits, which included retaining mail volume and generating new revenue from large advertisers. In addition, the pilot aimed to generate “statistically valid data” on how subscribers respond to marketing campaigns that mailers conduct. According to USPS, more than 70,000 customers were actively using the service at the pilot’s conclusion. In July 2016, USPS decided to end the pilot and launch the service nationally. According to USPS, about 13 million customers were subscribed to the service as of October 2018. USPS aims to have 40 million customers subscribed to the service by 2020. 23 According to USPS officials, the Informed Delivery pilot did not start during any one month, but rather was rolled out over a few months during the spring of 2014. Page 12 GAO-19-293 Postal Innovations Figure 5: Snapshot of U.S. Postal Service Informed Delivery Notification • Keyless Parcel Lockers: Since October 2013, USPS has piloted keyless parcel lockers that allow customers to independently pick up packages in 98 selected post offices. According to USPS, among other things, this innovation is intended to reduce the number of missed package deliveries to customers’ post office boxes and thereby reduce USPS’s delivery costs (see fig. 6). 24 The purpose of the pilot is to assess the performance and use of the lockers and to assess their performance. In October 2013, USPS began pre-testing the technical performance of 10 prototype keyless parcel locker units 24 According to USPS, missed deliveries to post office boxes occur when a package is unable to fit into a post office box and a parcel locker is unavailable, or when package does not fit into an available locker. Page 13 GAO-19-293 Postal Innovations at post offices in New York City and Northern Virginia. 25 Following this pre-test, in February 2015, USPS approved the installation of 50 made-to-order locker units in selected post offices across the country. Finally, in May 2016 USPS expanded the pilot to include an additional 50 units, including 2 units that a senior USPS official told us were not yet installed. As of November 2018, the pilot is still ongoing. Figure 6: Photo of U.S. Postal Service Keyless Parcel Locker Unit 25 According to USPS’s proposal for the pilot, a keyless locker unit typically contains 12 individual lockers and is capable of electronically recording inventory and locker use. Page 14 GAO-19-293 Postal Innovations USPS’s policies applicable to piloting key innovations fully reflect two of USPS’s Policies for the five leading practices for pilot design and evaluation that we identified Piloting Key in prior GAO work and relevant standards for internal control (see table 1). 26 These policies do not, however, fully reflect the other three leading Innovations Reflect practices due to policy gaps. Further, we found that USPS had not Some but Not All consistently followed its policies to document lessons learned at the conclusion of each pilot, as discussed below. Leading Practices for Pilot Design and Evaluation Table 1: Extent to Which the U.S. Postal Service’s (USPS) Policies for Pilot Design and Evaluation Reflect Leading Practices Leading practices for pilot design and evaluation Assessment of USPS’s policies Establish appropriate and measurable objectives linked with identified performance measures Do not fully reflect Articulate a methodology for evaluating pilot performance Fully reflect Evaluate pilot performance and identify and document lessons learned Fully reflect Draw and document conclusions about scalability (i.e., determining whether, how, and when to Do not fully reflect launch the innovation more broadly) based on pilot results Ensure appropriate two-way communication at all stages of the pilot with key internal and Do not fully reflect external stakeholders in order to understand and address their views Source: GAO. | GAO-19-293 Senior USPS officials acknowledged that gaps exist in its policies for pilot design and evaluation because they were not developed by USPS to fully reflect all leading practices. These policy gaps limit the extent to which USPS can ensure that it is making good resource allocation decisions based on pilot experiences. Below we further discuss the extent to which USPS’s policies reflect the five leading pilot practices we identified as well as how USPS applied these leading practices among the four piloted innovations that we reviewed. Establish appropriate and measurable objectives linked with identified performance measures: We found USPS’s policies do not fully reflect this leading practice. While USPS policies require that project managers 26 As discussed earlier, we identified leading practices from prior GAO reports on pilot design and evaluation, including GAO-09-45 and GAO-16-438, as well as from our review of standards for internal controls related to documentation in GAO-14-704G and the Committee of Sponsoring Organizations of the Treadway Commission’s Internal Control- Integrated Framework (2013). USPS officials identified Handbook F-66 and USPS Handbook F-66D as policies applicable to its piloting of key innovations. Page 15 GAO-19-293 Postal Innovations establish pilot objectives and performance measures, they do not require that each objective be linked with identified performance measures. As a result, some pilots may have objectives without an associated performance measure. For example, although USPS established both objectives and performance measures for each of the four innovations we selected for review, it did not consistently link each established objective to performance measures. USPS’s proposal to pilot same-day delivery, for example, had objectives of generating new revenue and improving customers’ experience. However, while the proposal included performance measures associated with generating new revenue—i.e., package volume, gross revenue, and net revenue—it did not identify and link any performance measures with its objective of improving customer experience. Similarly, USPS’s proposal to pilot keyless parcel lockers included improving customers’ experience as one of its objectives. However, while the proposal included a variety of performance measures—reduction in the number of missed deliveries to post office boxes, locker rate utilization, and on-time locker installation—it did not identify and link any performance measures with its improving customers’ experience objective. Absent such measures, USPS may not know whether customers have experienced an improvement using keyless parcel lockers compared to using manual, keyed parcel lockers. Linking all objectives to performance measures could help ensure that USPS has the performance information to assess the extent to which a pilot has achieved all of its objectives. USPS officials told us that it can be difficult to measure performance for some objectives related to customer experience. While measuring customers’ experience can be challenging, it is important to understand the extent to which a pilot has achieved all of its objectives. Further, USPS has demonstrated that it can measure improvement in customers’ experience. For example, during its Informed Delivery pilot, USPS conducted a consumer survey with approximately 5,500 Informed Delivery subscribers to collect data on consumer adoption and satisfaction. In the survey, USPS found that over 80 percent were satisfied or very satisfied with the service. According to USPS officials, this data helped USPS to measure the pilot’s success in meeting its objective of improving customers’ experience. Articulate a methodology for evaluating pilot performance: We found that USPS’s policies fully reflect this leading practice because the policies require officials to develop and communicate a methodology for Page 16 GAO-19-293 Postal Innovations evaluating pilot performance. Articulating such a methodology helps managers to identify the types and sources of performance information necessary to evaluate the pilot. 27 USPS’s policies require project managers to work with the pilot’s cross-functional team to develop and reach consensus on the methodology. These policies also require the project manager and cross-functional team to identify data needs, data sources, and how the data will be evaluated. For the four innovations we reviewed, we found that USPS articulated a methodology for evaluating the pilot’s performance. For example, for the Informed Delivery pilot, USPS identified its customer registration system as the method for tracking progress toward performance measures related to the number of Informed Delivery subscribers. Similarly, in its proposal to pilot keyless parcel lockers, USPS identified its central parcel locker monitoring system as a method of tracking progress toward performance measures related to utilization of keyless parcel lockers. Evaluate pilot performance and identify and document lessons learned: We found that USPS’s policies fully reflect this leading practice, but USPS did not consistently follow its policy that requires documenting lessons learned. Specifically, the policies require project managers to evaluate performance and document lessons learned at the conclusion of each pilot. Doing so can enable USPS to identify information needed to make conclusions about the pilot’s scalability and ensures that such information will be accessible to inform future related efforts. However, among the key innovations we selected for review USPS had not consistently documented lessons learned (see table 2). Table 2: Extent to Which the U.S. Postal Service (USPS) Documented Lessons Learned for Key Piloted Innovations Selected by GAO Documented lessons learned at Key piloted innovations pilot’s conclusion Metro Post (i.e., same-day delivery) Not documented Customized Delivery (i.e., grocery delivery) Not documented Informed Delivery Documented a Keyless parcel lockers Source: GAO. | GAO-19-293 a As of November 2018, USPS had not concluded its pilot for keyless parcel lockers. 27 GAO-16-438. Page 17 GAO-19-293 Postal Innovations USPS officials told us that they discussed lessons learned during ongoing monitoring of pilot performance for these innovations, but had only documented lessons learned for its Informed Delivery pilot. Specifically, USPS identified lessons learned in its July 2016 proposal to launch its Informed Delivery service nationally. In this proposal, we found that USPS identified some lessons learned about the pilot related to user satisfaction and adoption rates. USPS officials told us that this information helped to inform USPS’s decision to launch the service nationally. However, USPS officials acknowledged that USPS did not document lessons learned for the other two concluded pilots that we selected for review (same-day delivery and grocery delivery). Senior USPS officials told us that USPS had not consistently documented lessons learned at the conclusion of pilots across the 24 key innovations because it had not developed tools, such as a template, or training that could help ensure such consistency. Without consistently documenting lessons learned for all of its pilots, USPS risks losing information garnered during pilot implementation that could be relevant to future innovation efforts. Doing so can be particularly important because, according to a senior USPS official, officials responsible for pilot projects sometimes retire or leave USPS for employment elsewhere, creating a gap in knowledge of pilot experiences. Standards for internal control underscore the importance of maintaining documentation in order to retain organizational knowledge and mitigate the risk of having knowledge limited to a few personnel. 28 Draw and document conclusions about scalability based on pilot results: USPS’s policies do not fully reflect this leading practice. These policies require that project managers draw conclusions based on the results and lessons learned from the pilot. According to USPS officials, conclusions may include determining scalability—i.e., whether, how, and when to integrate pilot activities into overall efforts. However, USPS’s policies do not specifically require that officials document these conclusions. Documenting conclusions about scalability based on pilot results helps to ensure retention of organizational knowledge related to the pilot that may inform future decisions. 28 GAO-14-704G and Committee of Sponsoring Organizations of the Treadway Commission, Internal Control-Integrated Framework (2013). Page 18 GAO-19-293 Postal Innovations Among the three innovations that we selected for review for which the pilots had concluded (i.e., same-day delivery, grocery delivery, and Informed Delivery), USPS officials told us that senior leadership discussed the results and lessons of the pilots and made determinations regarding whether, how, and when to launch them more broadly, but that they did not document these decisions or the rationale for them. 29 By not documenting conclusions, USPS risks losing information that could affect the success of future related efforts and that could inform future USPS leadership of the rationale for maintaining investments in activities upon which pilots were based. Documenting conclusions for innovation pilots can be especially important in cases in which USPS decides to continue or expand pilot activities even when the pilots do not meet all of their intended objectives. For example, USPS’s same-day delivery and grocery delivery pilots had revenue objectives, along with associated performance measures; however, neither pilot achieved these objectives. For the same-day delivery pilot, costs exceeded revenue in 12 of the 13 fiscal year quarters in which the pilot was conducted, according to data USPS reported to PRC. Likewise, USPS data indicate it did not reach its annual revenue target for its grocery delivery pilot. Similarly, USPS’s pilot of Informed Delivery was intended to generate “statistically valid data” on how consumers respond to mailer marketing campaigns. However, according to a senior USPS official, the pilot did not generate the data as intended, because no such campaigns were conducted during the pilot. As discussed earlier in this report, USPS did not discontinue any of these three selected innovations when their pilots concluded. Although USPS may have had good reasons to continue with, or more broadly launch, these innovations despite the pilots not meeting all of their objectives, the lack of documentation regarding its reasoning and decisions limits information relevant to whether USPS is making judicious use of limited resources. Ensure appropriate two-way communication at all stages of the pilot with key internal and external stakeholders in order to understand and address their views: USPS’s policies do not fully reflect this leading practice. 29 In August 2017, USPS proposed purchasing over 1,400 additional keyless parcel locker units. However, according to a senior USPS official, USPS did not approve the purchase because the pilot had not demonstrated that the benefits of providing keyless parcel lockers to consumers outweighed installation, maintenance, and other costs. Page 19 GAO-19-293 Postal Innovations USPS’s policies require the involvement of key internal stakeholders in pilots. Specifically, USPS’s policies require the involvement of cross- functional teams—which include legal, finance, and other departments— and varying levels of review during the design and implementation of pilot proposals. However, USPS’s policies do not address communication with key external stakeholders. According to USPS officials, some pilot projects may be confidential or have limited or no direct effect on external stakeholders and, thus, communication with external stakeholders may not be appropriate. While external stakeholder communication may not be appropriate with some pilots, such communication, as appropriate, can help to ensure that issues critical to the success of a pilot activity are identified and addressed. Among the innovations that we selected for review, USPS officials explained various steps taken to involve internal stakeholders in the design and evaluation of the pilots, such as the involvement of cross functional teams to develop pilot proposals. Further, while USPS’s policies do not address external stakeholder communication, we found that USPS employed strategies for some of the innovations we selected for review to communicate with some external stakeholders—i.e., industry associations and mailers. For example, a representative of a postal association told us that USPS shared information and sought input about its Informed Delivery pilot during a quarterly meeting with industry groups. Similarly, a mailer we interviewed told us that USPS had shared information and sought input on the Informed Delivery pilot through direct outreach with the mailer. However, USPS did not consistently employ strategies to communicate with some key external stakeholders among the innovations that we selected for review. Specifically, USPS did not design or implement strategies to obtain feedback from consumers on its pilots for same-day delivery, grocery delivery, or keyless parcel lockers, despite the fact that each of these innovations directly affected consumers. In contrast, as previously discussed, for its Informed Delivery pilot, USPS planned and conducted a survey to obtain consumer feedback, the results of which helped USPS project managers support the proposal to expand the service nationally. Absent communication with all key stakeholders, USPS risks not having a complete understanding of perspectives that could inform the viability of its innovations. In recent years, USPS has sought to compete in a challenging business Conclusions environment by piloting innovations intended primarily to generate Page 20 GAO-19-293 Postal Innovations revenue and enhance customers’ experience. The policies that USPS uses for piloting key innovations fully reflect some leading practices for pilot design and evaluation, such as articulating a methodology for evaluating pilot performance. However, addressing gaps between USPS’s policies and leading practices related to linking objectives and performance measures, documenting conclusions, and communicating with key external stakeholders would enable USPS leadership to better assess the outcomes of its pilots, understand the rationale for conclusions about scalability based on pilot results, and gauge customers’ reactions to innovative products and services. Moreover, developing tools or training to ensure that USPS consistently implements its policy of documenting lessons learned from pilots would provide USPS with key information to inform future related efforts. We are making the following two recommendations to USPS: Recommendations for Executive Action The Postmaster General should direct the Vice President of Product Innovation to develop policies that fully reflect leading practices for pilot design and evaluation in areas such as linking objectives and performance measures; documenting conclusions about scalability based on pilot results; and communicating with key external stakeholders, as appropriate. (Recommendation 1) The Postmaster General should direct the Vice President of Product Innovation to develop tools, such as a template, or training to help ensure USPS consistently documents lessons learned at the conclusion of pilots, as required by USPS policies. (Recommendation 2) We provided a draft of this product to USPS and PRC for comment. Agency Comments USPS provided a written response, which is reproduced in appendix II of and Our Evaluation this report. In its response, USPS did not state whether it agreed with our recommendations, but described actions that it plans to take related to each. These actions, if fully implemented, would meet the intent of our recommendations. For example, USPS stated that it would develop policies specifically for pilot design, and would reflect leading practices for pilot design and evaluation based upon best practice research. USPS also noted that it would develop training to ensure consistent documentation of lessons learned from its pilots. USPS added that this planned training would cover best practices for pilot tests. Page 21 GAO-19-293 Postal Innovations Regarding our first recommendation USPS said that pilots are only one step in a larger process for developing innovations. We agree with this and noted in our report that piloting is one key element of USPS’s efforts to innovate. Nonetheless, given USPS’s financial position, effectively piloting innovations is a critical step to ensure that USPS invests its limited resources on innovations that are most likely to improve its long- term viability. USPS also stated that flexibility is important in innovation pilots, particularly as it pertains to linking pilot objectives with performance measures. We continue to believe that linking objectives with performance measures is key to effectively evaluating pilots. In so doing, however, there is flexibility to adjust pilot objectives and performance measures as new information is gleaned during the pilot. Finally, with regard to communication with external stakeholders during pilots, USPS said that it communicates consistently with external stakeholders regarding pilots at Mailers’ Technical Advisory Committee meetings (MTAC). 30 In our report, we noted that USPS employed strategies to communicate with some external stakeholders—i.e., industry associations and mailers. We continue to believe, however, in the importance of communication with all key external stakeholders, which may include stakeholders, such as consumers, that do not participate in MTAC meetings. USPS and PRC also provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committee, the Postmaster General, Chairman of PRC, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. 30 The Mailers’ Technical Advisory Committee is a venue for USPS to share technical information with mailers and to receive their advice and recommendations on matters concerning mail-related products and services. The committee holds meetings on a quarterly basis. Page 22 GAO-19-293 Postal Innovations If you or your staff have any questions about this report, please contact me at (202) 512-2834 or RectanusL@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. Sincerely yours, Lori Rectanus Director Physical Infrastructure Issues Page 23 GAO-19-293 Postal Innovations Appendix I: List of U.S. Postal Service Key Piloted Innovations, Fiscal Years 2013-2017 The U.S. Postal Service (USPS) piloted 24 key innovations from fiscal years 2013 through 2017 (see table 3). 1 Table 3: U.S. Postal Service (USPS) Key Piloted Innovations, Fiscal Years 2013 through 2017 Innovation Primary Innovation name Description goal Metro Post (i.e., same-day Package delivery product that delivers packages from participating retailers Generate revenue delivery) to customers in selected areas on the same day. Customized Delivery (i.e., Package delivery product that delivers groceries and other prepackaged Generate revenue grocery delivery) goods from a participating retailer to customers in selected areas. Standard Post / Retail Package delivery product that reduces customer costs by providing “zone Generate revenue Ground pricing” (i.e., lower prices for boxes shipped shorter distances) for less-than- urgent deliveries and oversized packages. Parcel Return Service pick- Offering that reduces the number of USPS locations shippers must visit to Generate revenue up locations pick-up returned packages. Priority Mail regional rate Alternative shipping product that reduces customer costs by providing “zone Generate revenue pricing” (i.e., lower prices for boxes shipped shorter distances) for some Priority Mail shipments. Informed Delivery Platform Service that provides consumers with a digital preview of their household Generate revenue mail arriving soon. Mailers can integrate digital campaign elements to engage the consumer. MyUSPS Portal that allows customers to see all packages destined to their address, Improve customers’ create return labels, and request mail holds or redelivery. This feature in experience now a part of the Informed Delivery platform so customers can see all mail and packages in one place. Expected Delivery A service that provides customers with: Improve customers’ a delivery timeframe (currently 2 hours) on the day of delivery; and experience predictions for the expected delivery date and delivery timeframe at least the day before the actual delivery day. Mobile Check Payment A service that allows commercial customers to make deposits using a Improve customers’ mobile device, thereby avoiding the need to visit a retail location, enhancing experience the customer experience, and expediting the clearance of funds. Keyless parcel lockers Lockers installed in some USPS post offices that serve as a “last mile” Improve customers’ delivery point for packages that cannot be delivered directly into P.O. Boxes. experience Identity Verification Service A service that would allow users to verify their digital identity either remotely Generate revenue (i.e., online) or in person. 1 USPS defines “key innovations” as those that have a significant effect on cost, the end consumer, the shipper, or the mailer. The 2006 Postal Accountability and Enhancement Act defines (a) postal “product” as a postal service with a distinct cost or market characteristic for which a rate or rates are, or may reasonably be, applied, and (b) “postal service” refers to the delivery of letters, printed matter, or mailable packages, including acceptance, collection, sorting, and transportation, or other ancillary function. USPS defines an “enhancement” as any improvement to the quality or value of a postal product or service. Page 24 GAO-19-293 Postal Innovations Innovation Primary Innovation name Description goal Electronic Postmark A service that assures the integrity, authenticity, and accuracy of electronic Generate revenue data content. Priority Mail redesign Enhanced Priority Mail product by providing expected delivery, free Generate revenue insurance up to $50, and new packaging. Parcel Select rural pricing Enhanced Parcel Select product by developing two-tiered (i.e., rural and Generate revenue non-rural) pricing to meeting shipper needs. Parcel Select next day Enhanced Parcel Select product by allowing shippers to drop packages at Generate revenue delivery USPS processing plants with lower sorting requirements. Parcel Select Return Service Enhanced Parcel Select product by expanding the number of USPS Generate revenue locations that accept package returns from shippers. Click-N-Ship Business Pro Developed software that allows mailers to pay postage electronically and Generate revenue provides enhanced reporting features for some shipments. Merchant Return scan base Streamlined package returns process by using electronic scan data to Generate revenue payment calculate payment in lieu of manually weighing and rating each package. Developed online tools for Developed software that allows mailers to, for example, print shipping Improve Internal shippers labels, track packages, schedule package pick-ups by accessing USPS Infrastructure data. Next Generation Mailbox Developed larger mailbox receptacle capable of accommodating more Improve customers’ package deliveries and locations with multiple delivery points. experience Informed Visibility / GEO Developed software that provides mailers with near real-time end-to-end Improve customers’ delivery tracking information for some mailpieces and mail aggregates (e.g., experience containers). USPS returns automated Developed capability to charge for returns based on captured attributes, Generate revenue such as capturing package weight from mail processing equipment, accurately assessing postage, and easily collecting postage due. Near real time package Developed capability of on-street devices used by carriers to transmit mail Improve customers’ delivery notification scan information at the time of the scanning. experience E-Commerce Program A service to enable printing and payment of shipping labels from customers’ Generate revenue home or office. Source: GAO analysis of USPS information. | GAO-19-293 Note: USPS did not discontinue any key piloted innovations from fiscal years 2013 through 2017. Page 25 GAO-19-293 Postal Innovations Appendix II: Comments from the U.S. Postal Appendix II: Comments from the U.S. Postal Service Service Page 26 GAO-19-293 Postal Innovations Appendix II: Comments from the U.S. Postal Service Page 27 GAO-19-293 Postal Innovations Appendix III: GAO Contact and Staff Appendix III: GAO Contact and Staff Acknowledgments Acknowledgments Lori Rectanus, (202) 512-2834 or RectanusL@gao.gov GAO Contact In addition to the individual named above, Derrick Collins (Assistant Staff Director); William Colwell and James Leonard (analysts in charge); Acknowledgments Barbara El Osta; Geoffrey Hamilton; Gina Hoover; Anthony Jackson; and Laurel Voloder made key contributions to this report. Page 28 GAO-19-293 Postal Innovations (102478) The Government Accountability Office, the audit, evaluation, and investigative GAO’s Mission arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO’s commitment to good government is reflected in its core values of accountability, integrity, and reliability. The fastest and easiest way to obtain copies of GAO documents at no cost is Obtaining Copies of through GAO’s website (https://www.gao.gov). Each weekday afternoon, GAO GAO Reports and posts on its website newly released reports, testimony, and correspondence. To have GAO e-mail you a list of newly posted products, go to https://www.gao.gov Testimony and select “E-mail Updates.” Order by Phone The price of each GAO publication reflects GAO’s actual cost of production and distribution and depends on the number of pages in the publication and whether helthe publication is printed in color or black and white. Pricing and ordering information is posted on GAO’s website, https://www.gao.gov/ordering.htm. Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537. Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information. Connect with GAO on Facebook, Flickr, Twitter, and YouTube. Connect with GAO Subscribe to our RSS Feeds or E-mail Updates. Listen to our Podcasts. Visit GAO on the web at https://www.gao.gov. Contact FraudNet: To Report Fraud, Website: https://www.gao.gov/fraudnet/fraudnet.htm Waste, and Abuse in Automated answering system: (800) 424-5454 or (202) 512-7700 Federal Programs Orice Williams Brown, Managing Director, WilliamsO@gao.gov, (202) 512-4400, Congressional U.S. Government Accountability Office, 441 G Street NW, Room 7125, Relations Washington, DC 20548 Chuck Young, Managing Director, email@example.com, (202) 512-4800 Public Affairs U.S. Government Accountability Office, 441 G Street NW, Room 7149 Washington, DC 20548 James-Christian Blockwood, Managing Director, firstname.lastname@example.org, (202) 512-4707 Strategic Planning and U.S. Government Accountability Office, 441 G Street NW, Room 7814, External Liaison Washington, DC 20548 Please Print on Recycled Paper.
U.S. Postal Service: Addressing Policy Gaps Could Improve Pilot Design and Evaluation for Postal Innovations
Published by the Government Accountability Office on 2019-03-14.
Below is a raw (and likely hideous) rendition of the original report. (PDF)