oversight

Checked Baggage Screening: TSA Has Deployed Optimal Systems at the Majority of TSA-Regulated Airports, but Could Strengthen Cost Estimates

Published by the Government Accountability Office on 2012-04-27.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

             United States Government Accountability Office

GAO          Report to Congressional Requesters




April 2012
             CHECKED BAGGAGE
             SCREENING
             TSA Has Deployed
             Optimal Systems at
             the Majority of TSA-
             Regulated Airports,
             but Could Strengthen
             Cost Estimates




GAO-12-266
                                               April 2012

                                               CHECKED BAGGAGE SCREENING
                                               TSA Has Deployed Optimal Systems at the Majority
                                               of TSA-Regulated Airports, but Could Strengthen
                                               Cost Estimates
Highlights of GAO-12-266, a report to
congressional requesters




Why GAO Did This Study                         What GAO Found
TSA’s EBSP, one of DHS’s largest               The Transportation Security Administration’s (TSA) Electronic Baggage
acquisition programs, aims to improve          Screening Program (EBSP) reports that 76 percent of the airports (337 of 446)
security and lower program life cycle          the agency regulates for security have a mix of in-line and stand-alone baggage
costs by optimizing checked baggage            screening configurations that best meet airport needs (i.e., optimal systems).
screening systems that best meet the           However, only 36 percent (10 of 28) of the nation’s larger airports—based on
needs of the nation’s airports. This           factors such as the total number of takeoffs and landings annually—have
includes, among other things, the              complete optimal systems. This is because the larger airports generally need
integration of baggage screening               more complex in-line systems and often require a significant amount of airport
equipment into baggage handling
                                               infrastructure modification and construction. In August 2011, TSA shifted its
systems, referred to as in-line systems.
                                               focus from installing optimal baggage screening systems to replacing aging
Installing in-line systems typically
requires airports to undertake costly
                                               machines (recapitalization). However, TSA plans to continue to optimize systems
facility modification projects, for which      during many of its recapitalization projects.
TSA will generally reimburse up to the         Using TSA cost estimates, GAO estimates that reducing the portion of costs that
applicable federal cost share. As              TSA pays for facility modifications associated with the installation of optimal
requested, GAO examined (1) the                baggage screening systems, from 90 percent to 75 percent, would lower the
status of TSA’s efforts to install optimal     federal government’s cost for airport modification projects it supports by roughly
checked baggage screening systems              $300 million from fiscal year 2012 through fiscal year 2030. Officials from all 10
in collaboration with airports, (2) how
                                               airports with whom GAO spoke stated that airports benefit from the installation of
reducing the federal cost share for
                                               integrated, in-line baggage screening systems. The primary benefit—cited by
eligible airport modification projects
from 90 percent to its previous level of       representatives from 9 of the airports GAO visited—is that passenger congestion
75 percent would affect the amount             is reduced by removing stand-alone machines from lobbies or ticketing areas.
that TSA pays for modifications, and           Other benefits cited by airports included a reduction in lost baggage and
(3) whether TSA’s methods for                  increased screening and passenger throughput. However, for a variety of
estimating and validating costs for the        reasons, representatives from 8 of 10 airports GAO visited opposed a reduction
EBSP are consistent with best                  in the federal cost share for related airport modifications.
practices. GAO reviewed EBSP
                                               TSA established cost estimates for the EBSP to help identify total program cost,
planning and status documents,
                                               recapitalization cost, and potential savings resulting from installing optimal
compared TSA’s cost estimation
approach against GAO best practices,
                                               systems, but its processes for developing these estimates do not fully comply
and visited 10 airports selected in part       with best practices. These include, among other things, ensuring that the
based on the status of the EBSP                estimates comprise all costs and are well documented. For example, TSA’s
optimization at these airports. Although       estimates were properly adjusted for inflation and were developed using relevant
the results from these visits are not          data, such as existing contracts for equipment purchases and maintenance
generalizable, they provided insights          costs. However, the estimates did not include all costs, for example, the costs
into the program.                              associated with detecting all security threats, and many assumptions and
                                               methodologies underlying the cost model were not clearly documented. As
What GAO Recommends                            highlighted in GAO’s past work, a high-quality, reliable cost estimation process
GAO recommends that TSA ensure                 provides a sound basis for making accurate and well-informed decisions about
that its life cycle cost estimates             resource investments and budgets and thus is critical to the success of a
conform to cost estimating best                program. Developing accurate cost estimates would help ensure that the
practices. DHS concurred with GAO’s            program does not experience unanticipated cost growth and program funding
recommendation.                                needs resulting from future recapitalization and facility modification activities. In
                                               addition, TSA is working with the Department of Homeland Security (DHS) to
                                               develop an approved acquisition program baseline, which according to DHS
                                               guidance is the contract between program and departmental oversight officials
View GAO-12-266. For more information,
contact Stephen M. Lord at (202) 512-4379 or   for what will be delivered, how it will perform, and what it will cost. TSA expects
lords@gao.gov.                                 the baseline to be approved in May 2012.

                                                                                        United States Government Accountability Office
Contents


Letter                                                                                   1
               Background                                                                8
               TSA Has Made Progress in Deploying Optimal Checked Baggage
                 Screening                                                              17
               Reducing the Federal Cost Share for Eligible Airport Facility
                 Modification Projects Could Reduce TSA’s Costs                         22
               TSA’s Cost Estimation Processes Do Not Fully Comply with Best
                 Practices, and TSA and DHS Are Working Together to Establish
                 an Acquisition Program Baseline                                        26
               Conclusions                                                              34
               Recommendation for Executive Action                                      34
               Agency Comments and Our Evaluation                                       34

Appendix I     Objectives, Scope, and Methodology                                       37



Appendix II    Status of Implementation Efforts for TSA’s EBSP                          42



Appendix III   Our Evaluation of EBSP’s Cost Estimate Process                           44



Appendix IV    Comments from the Department of Homeland Security                        46



Appendix V     GAO Contact and Staff Acknowledgments                                    48



Tables
               Table 1: Funding Available for the EBSP for Fiscal Years 2006 to
                        2012                                                            14
               Table 2: GAO’s Assessment of TSA’s Method for Developing EBSP
                        Life Cycle Cost Estimates                                       28
               Table 3: Checked Baggage Configurations for TSA-Regulated
                        Airports That Have Optimal Systems as of December 2011          42
               Table 4: TSA-Regulated Airports with at Least One In-line System
                        as of December 2011                                             42




               Page i                                  GAO-12-266 Checked Baggage Screening
          Table 5: Numbers of In-line Systems and Stand-alone EDS and ETD
                   Machines for TSA-Regulated Airports as of December 2011        43
          Table 6: Summary Assessment of TSA’s Electronic Baggage
                   Screening Program Cost Estimate Compared to Best
                   Practices                                                      44


Figures
          Figure 1: Transportation Security Officer Removes Checked
                   Baggage from Stand-alone EDS Machine                            9
          Figure 2: An ETD Machine Being Used to Manually Resolve a
                   Suspect Item                                                   10
          Figure 3: A Checked Baggage Inspection System That Includes a
                   Baggage Handling System with Three In-line EDS
                   Machines                                                       11
          Figure 4: A Diagram of a Checked Baggage Inspection System and
                   Three Levels of Screening                                      12
          Figure 5: Baggage Screening Configurations at the 337 Airports
                   with Optimal Systems                                           19
          Figure 6: Number of Airports with Optimal Baggage Screening
                   Systems Compared to All TSA-Regulated Airports                 20
          Figure 7: Age and Number of Deployed EDS Machines as of
                   December 2011                                                  22




          Page ii                                GAO-12-266 Checked Baggage Screening
Abbreviations

AIP                        Airport Improvement Program
ATSA                       Aviation and Transportation Security Act
CBRA                       Checked Baggage Resolution Area
DHS                        Department of Homeland Security
EBSP                       Electronic Baggage Screening Program
EDS                        explosives detection system
ETD                        explosives trace detection
FAA                        Federal Aviation Administration
LCCE                       Life Cycle Cost Estimate
LOI                        letter of intent
OMB                        Office of Management and Budget
OSR                        On-Screen Resolution
OTA                        other transaction agreements
TSA                        Transportation Security Administration



This is a work of the U.S. government and is not subject to copyright protection in the
United States. The published product may be reproduced and distributed in its entirety
without further permission from GAO. However, because this work may contain
copyrighted images or other material, permission from the copyright holder may be
necessary if you wish to reproduce this material separately.




Page iii                                          GAO-12-266 Checked Baggage Screening
United States Government Accountability Office
Washington, DC 20548




                                   April 27, 2012

                                   The Honorable John D. Rockefeller IV
                                   Chairman
                                   The Honorable Kay Bailey Hutchison
                                   Ranking Member
                                   Committee on Commerce, Science, and Transportation
                                   United States Senate

                                   The Honorable Susan M. Collins
                                   Ranking Member
                                   Committee on Homeland Security and Governmental Affairs
                                   United States Senate

                                   The Honorable Jim DeMint
                                   United States Senate

                                   The Honorable Henry Johnson
                                   House of Representatives

                                   As demonstrated by the attempted attack of Northwest Airlines Flight 253
                                   from Amsterdam to Detroit on Christmas Day in 2009, commercial
                                   aviation continues to be a target of terrorist activity. The 1988 bombing of
                                   a U.S. airliner over Lockerbie, Scotland, further demonstrates that
                                   explosive devices placed in checked baggage have long been an area of
                                   concern. The Aviation and Transportation Security Act (ATSA), enacted
                                   on November 19, 2001, created the Transportation Security
                                   Administration (TSA), now a component of the Department of Homeland
                                   Security (DHS), and mandated, among other things, that it provide for the
                                   screening of all checked baggage using explosive detection systems. 1 To
                                   satisfy this mandate, TSA has deployed two types of screening
                                   equipment to airports in the United States where screening is required:
                                   (1) explosives detection systems (EDS), which use X-rays with computer-
                                   aided imaging to automatically recognize the characteristic signatures of



                                   1
                                     See Pub. L. No. 107-71, §§ 101(a), 110(b), 115 Stat. 597, 597-604, 614-16 (2001)
                                   (codified as amended at 49 U.S.C. §§ 114(a), 44901(d)). Pursuant to the Homeland
                                   Security Act of 2002, enacted on November 25, 2002, TSA transferred from the
                                   Department of Transportation to the newly formed DHS. See Pub. L. No. 107-296, §§ 101,
                                   403(2), 425, 116 Stat. 2135, 2142, 78, 85-86 (2002).




                                   Page 1                                         GAO-12-266 Checked Baggage Screening
threat explosives, and (2) explosives trace detection (ETD) machines, in
which a human operator (baggage screener) uses chemical analysis to
manually detect traces of explosive materials’ vapors and residue.

As amended, ATSA mandated that TSA provide for the screening of all
checked baggage using EDS machines no later than December 31,
2003. 2 To meet the 100 percent checked baggage screening
requirement, TSA deployed screening equipment in a variety of
temporary screening solutions, including EDS machines placed in stand-
alone configurations in airport terminal lobbies, and solutions that relied
on ETD machines for primary screening. According to TSA, these initial
stand-alone EDS machines have higher operating costs and can screen
fewer bags per hour than EDS machines that are integrated into a single,
coordinated “in-line” baggage handling system. To improve the efficiency
and effectiveness of baggage screening at airports, TSA’s Electronic
Baggage Screening Program (EBSP) aims to, among other goals, (1)
replace, reconfigure, and deploy equipment to increase throughput,
system capacity, and effectiveness while reducing staffing requirements
and airport lobby installations; (2) increase equipment reliability, reduce
equipment downtime, and extend service life; and (3) achieve new
capabilities to better detect changing terrorist threats.

Pursuant to ATSA, providing for checked baggage screening is a federal
responsibility, but airports must be capable of accommodating the
screening systems to secure safe travel for the nation’s passengers. TSA
collaborates with airports and airlines to determine the most appropriate
screening configuration and install the systems that best fit the airports’ or
airlines’ needs. Such optimal baggage screening systems may have a
mix of detection systems (EDS or ETD) and configurations (in-line or
stand-alone systems), depending on the airport’s needs, and may help
streamline airport and TSA operations, reduce screening costs, and
enhance security. However, in-line solutions may not be appropriate or
cost effective in airports or terminals with fewer bags per week to screen.
From fiscal years 2008 through 2011, TSA reported that it obligated


2
  See 49 U.S.C. § 44901(d). ATSA further provides that TSA must take all necessary
action to ensure that all explosive detection systems deployed are fully utilized, and that if
explosive detection equipment at an airport is unavailable, all checked baggage is
screened by an alternative means authorized under the statute, see § 44901(d)-(e)
(authorizing a bag match program, a manual search, search by canine explosive detection
units in combination with other means, or other means or technology approved by the TSA
Administrator, as alternative means to using explosive detection equipment).




Page 2                                              GAO-12-266 Checked Baggage Screening
approximately $1.9 billion to support airport facility modification projects
required to install more efficient optimal baggage screening systems, and
estimates that it will obligate an additional $1.15 billion for this purpose
through fiscal year 2030. 3 TSA has determined that manual ETD
screening is an optimal configuration for the nation’s smaller airports at
which TSA performs or oversees screening.

TSA has generally supplemented the costs of airport facility modification
projects necessary to install such optimal baggage screening systems by
entering into reimbursable agreements with airports. 4 In general, an
airport must be willing and financially capable of undertaking the facility
modification project; TSA may not compel an airport to undertake such a
project. Although TSA will procure and install the screening equipment,
the airport, and not TSA, is ultimately responsible for any facility
modification costs. Pursuant to a reimbursable agreement and subject to
the availability of appropriations, however, TSA will generally reimburse
an airport undertaking such a project at the applicable percentage of
allowable project costs. According to TSA, since 2003, it has entered into
12 letter of intent (LOI) agreements at the statutorily established federal
cost shares of 75 or 90 percent and at least 150 other transaction
agreements (OTA) at cost shares that have ranged from 50 to 100




3
  The $1.9 billion includes over $700 million made available to TSA in fiscal year 2009
through the American Recovery and Reinvestment Act of 2009, see Pub. L. No. 111-5,
123 Stat. 115, 163 (2009).
4
  According to TSA’s Planning Guidelines and Design Standards for Checked Baggage
Inspections Systems, version 4.1, September 15, 2011, the project sponsor is assumed to
be an airport owner/operator or an airline (if the system is for an airline-owned terminal).
Some of the key responsibilities of the project sponsor include initiation and execution of
planning and design, application for TSA funding, initiation and execution of construction,
as well as testing and commissioning of the checked baggage inspection system and
operation and maintenance of the baggage handling system portion of the overall checked
baggage inspection system. TSA is responsible for, among other things, developing the
planning guidelines and design standards, working with airports and airlines to develop
and review designs, providing the EDS and ETD screening equipment, and final approval
of the checked baggage system.




Page 3                                             GAO-12-266 Checked Baggage Screening
percent. 5 Currently, TSA enters into reimbursable agreements through
which it generally funds 90 percent of the costs of modifying airports for
the installation of an optimal system, with airports or airlines funding the
remaining 10 percent of the projects’ cost.

A foundation for making informed budget decisions for acquisition
programs at DHS is the acquisition program baseline, which according to
DHS’s guidance is its contract with a component, such as TSA, on a
program’s critical costs, schedule, and program performance goals.
Establishing such a baseline at the program start is important for defining
its scope, assessing whether all life cycle costs are properly calculated,
and measuring how well the program is meeting its goals. By tracking and
measuring actual program performance against this baseline,
management can be alerted to potential problems, such as cost growth or
changing requirements, and has the ability to take early corrective action.

You asked us to assess the EBSP’s current status, costs, and alternative
cost sharing options. Accordingly, this report addresses the following
questions:

•   What is the status of TSA’s efforts to install optimal checked baggage
    screening systems in collaboration with airports?
•   How would reducing the current federal cost share for eligible airport
    modification projects from 90 percent to its previous level of 75
    percent affect the amount that TSA pays for these modifications, and
    what benefits, if any, do airports report receiving from in-line baggage
    screening systems?
•   To what extent are TSA’s cost estimation procedures consistent with
    best practices and is TSA’s acquisition baseline consistent with DHS
    guidance?



5
  Pursuant to an LOI, TSA may commit to obligate from future budget authority an amount
not to exceed the federal government’s share of a project’s cost for the purpose of
reimbursing an airport or other responsible party that has undertaken an eligible airport
facility improvement project. See, e.g., 49 U.S.C. § 44923(d). An LOI is subject to the
availability of appropriations and does not constitute a binding commitment of federal
funding. According to TSA, OTAs are administrative vehicles used by the agency to fund
airport modification projects without undertaking a long-term commitment. OTAs take
many forms and are generally not required to comply with federal laws and regulations
that apply to contracts, grants, or cooperative agreements, and they enable the federal
government and others entering into these agreements to freely negotiate provisions that
are mutually agreeable.




Page 4                                           GAO-12-266 Checked Baggage Screening
To determine the status of TSA’s efforts to install optimal checked
baggage screening systems, we obtained the most current data available
from TSA as of December 2011 and January 2012 for the following: 1) on
the current number of airports with at least one in-line system, 2) the
number of airports that have optimal checked baggage screening
systems, 3) and the number of operational in-line systems and EDS and
ETD machines deployed to TSA-regulated airports. We reviewed
documentation from TSA’s EBSP, including the EBSP strategic plans for
fiscal years 2006, 2008, and 2009, and the 2011 EDS and ETD
Recapitalization and Optimization Plan.

To determine how reducing the current federal cost share for eligible
airport facility modification projects may affect the amount that TSA pays
for these modifications, we calculated estimates based on TSA’s August
2011 projections of how much airport modifications will cost in the future.
To estimate what cost shifts could occur in the future, we obtained TSA’s
projections of its airport modification costs as stated in its final August
2011 Life Cycle Cost Estimate (LCCE) report. 6 We used TSA’s LCCE
projections of airport modification costs for each fiscal year, 2012 through
2030, and estimated the amount of expenditures for airport modifications
that could be shifted from TSA to airports and airlines if the federal
government’s cost share were reduced from the current 90 percent to
previous federal cost share of 75 percent.

We assessed the reliability of the various data TSA provided about
airports, including the number of TSA-regulated airports (by category) and
the numbers of airports with the different configurations of baggage
screening (in-line or stand-alone) systems, and the investment and
budget expenditure dollar values in the LOIs and OTAs by questioning
cognizant TSA officials and obtaining documentation about these various




6
  Life cycle costs are to be comprehensive, which includes all resources and associated
cost elements required to develop, produce, deploy, and sustain a particular program from
initial concept through operations, support, and disposal. Acquisition costs include costs
for all supplies and services for a designated investment.




Page 5                                            GAO-12-266 Checked Baggage Screening
data. 7 We found these data to be sufficiently reliable for the purposes of
our report.

To assess the extent to which TSA’s cost estimation methods are
consistent with best practices and how its acquisition program baseline
efforts are consistent with DHS guidance, we analyzed the TSA August
2011 LCCE for EBSP. We used best practices in GAO’s Cost Estimating
and Assessment Guide to evaluate TSA’s estimating methodologies,
assumptions, and results to assess whether the official cost estimates
were comprehensive (i.e., include all costs), accurate, well documented,
and credible. 8 We defined a characteristic of the LCCE as not met if the
agency provided no evidence that satisfied any portion of the criterion,
minimally met if the agency provided evidence that satisfied less than
one-half of the criterion, partially met if the agency provided evidence that
satisfied about one-half of the criterion, substantially met if the agency
provided evidence that satisfied more than one-half of the criterion, and
met if the agency provided complete evidence that satisfied the entire
criterion. Our analyses examined TSA’s practices in developing its LCCE.
Regarding the development of an acquisition program baseline, we
examined the EBSP Acquisition Review Board decisions and relevant
acquisition decision memos during the period 2005 through 2011, as well
as other documentation related to DHS and TSA efforts to develop an
acquisition program baseline. 9 We also interviewed TSA and DHS
officials, including officials in the TSA Chief Financial Officer’s Office and
DHS’s Program Accountability and Risk Management Office, to identify




7
  TSA classifies the over 400 commercial airports in the United States into one of five
airport security categories (X, I, II, III, and IV) based on various factors, such as the total
number of takeoffs and landings annually, the extent to which passengers are screened at
the airport, and other special security considerations. In general, category X airports have
the largest number of passenger boardings and category IV airports have the smallest.
8
 GAO, GAO Cost Estimating and Assessment Guide: Best Practices for Developing and
Managing Capital Program Costs, GAO-09-3SP (Washington, D.C.: March 2009).
9
  DHS established the Acquisition Review Board as the department’s highest review body
and charged it with reviewing and approving all programs at key milestone decision points
that are above $300 million in life cycle costs. The Acquisition Review Board reviews
provide an opportunity to determine a program’s readiness to proceed to the next life cycle
phase. DHS’s Acquisition Management Directive 102-01 of January 2010 superseded
similar interim directives, and requires the Acquisition Review Board chairperson to
approve key acquisition documents critical to establishing the program, operational
requirements, acquisition baseline, testing, and support.




Page 6                                              GAO-12-266 Checked Baggage Screening
what procedures they have put in place to approve the acquisition
program baseline.

To gain a better understanding of issues across all of our objectives, we
conducted site visits to airports in California, New York, Massachusetts,
Washington, D.C., and Florida to interview local airport officials, regional
TSA officials, and airline representatives. To get a range of airports, we
chose these airports based on the size of airport, type of checked
baggage screening system installed, and whether the airport’s checked
baggage screening system had been optimized. Because we selected a
nonprobability sample of airports, the information we obtained from these
interviews and visits cannot be generalized to all airports. However, we
believe that observations obtained from these visits provided us with a
greater understanding of airport officials’ perspectives on TSA’s
collaboration with airports that seek to have optimal baggage screening
systems installed and allowable costs reimbursed, as well as the
perspective of local TSA officials. In addition, we interviewed officials from
the largest industry associations, which represent airport executives,
airports, and airlines (the American Association of Airport Executives, the
Airports Council International North America, and the Air Transport
Association). We also interviewed an official from the Association for
Airline Passenger Rights. These results cannot be generalized to the
entire industry but provided industry perspectives.

We conducted this performance audit from October 2010 through April
2012 in accordance with generally accepted government auditing
standards. 10 Those standards require that we plan and perform the audit
to obtain sufficient, appropriate evidence to provide a reasonable basis
for our findings and conclusions based on our audit objectives. We
believe that the evidence obtained provides a reasonable basis for our
findings and conclusions based on our audit objectives.




10
  We issued preliminary results for this work in 2011 and 2012 as part of GAO’s effort to
identify opportunities that federal agencies or the Congress could consider for reducing
the cost of government operations or enhancing revenue collections for the Treasury. See
GAO, Opportunities to Reduce Potential Duplication in Government Programs, Save Tax
Dollars, and Enhance Revenue, GAO-11-318SP (Washington, D.C.: Mar. 1, 2011), and
2012 Annual Report: Opportunities to Reduce Duplication, Overlap and Fragmentation,
Achieve Savings, and Enhance Revenue, GAO-12-342SP (Washington D.C.: Feb. 28,
2012).




Page 7                                           GAO-12-266 Checked Baggage Screening
             With the passage of ATSA in November 2001, TSA assumed from the
Background   Federal Aviation Administration (FAA) responsibility for securing the
             nation’s civil aviation system. In accordance with ATSA, TSA is
             responsible for the procurement, installation, and maintenance of
             explosive detection systems, including EDS and ETD, used to screen
             checked baggage for explosives (see figs. 1 and 2) at TSA-regulated
             airports. 11 EDS machines identify suspicious items or anomalies in
             checked baggage that could indicate the presence of explosives or
             detonation devices. At airports with EDS, EDS machines are generally
             employed for primary screening of checked baggage while ETD machines
             are used for secondary screening to help resolve questions raised by
             EDS screening. At airports without EDS, ETD machines are used as the
             primary method for screening checked baggage.




             11
                “TSA-regulated airports” refers to all airports that implement TSA-approved security
             programs pursuant to 49 C.F.R. part 1542, which includes airports regularly serving air
             carrier operations regulated pursuant to 49 C.F.R. §§ 1544.101 and 1546.101 (what may
             be characterized as U.S and foreign-flagged commercial air carriers), and at which TSA
             performs or oversees the performance of screening activities.




             Page 8                                           GAO-12-266 Checked Baggage Screening
Figure 1: Transportation Security Officer Removes Checked Baggage from Stand-
alone EDS Machine




Page 9                                     GAO-12-266 Checked Baggage Screening
Figure 2: An ETD Machine Being Used to Manually Resolve a Suspect Item




TSA deploys EDS machines in stand-alone and in-line configurations. In
a stand-alone configuration, checked baggage is manually loaded and
unloaded by screeners (see fig. 1). 12 In contrast, an in-line configuration
integrates EDS machines with a baggage handling system—a conveyor
system that transports and sorts baggage from the ticket counter through
the baggage screening system. (See fig. 3, which shows an in-line
configuration with three EDS machines.)




12
   The term screeners refers to both transportation security officers, who are TSA
employees, and to private sector screeners employed by companies under contract to
TSA and who carry out the screening function with TSA oversight at airports participating
in TSA’s Screening Partnership Program. See 49 U.S.C. §§ 44901(a), 44920.




Page 10                                           GAO-12-266 Checked Baggage Screening
Figure 3: A Checked Baggage Inspection System That Includes a Baggage Handling
System with Three In-line EDS Machines




Generally, an in-line checked baggage inspection system employs three
levels of screening (see fig. 4). EDS machines perform automated (Level
1) screening. If the EDS machine is unable to clear a bag, it sends an
alarm to a screener who performs a secondary (Level 2) inspection
known as On-Screen Resolution by reviewing an image of the contents of
the bag via computer monitor. If the screener cannot resolve the alarm
using on-screen resolution tools, the bag goes to the Checked Baggage
Resolution Area (Level 3) where another screener will perform manual
inspection of the bag assisted by an ETD machine.




Page 11                                    GAO-12-266 Checked Baggage Screening
Figure 4: A Diagram of a Checked Baggage Inspection System and Three Levels of Screening




                                       TSA officials stated that deployment of an integrated, centralized in-line
                                       system of EDS machines can enhance security, increase screening
                                       efficiencies, and lower screening costs by, among other things, reducing
                                       the number of screeners needed to conduct baggage screening and
                                       reducing work-related injuries caused by lifting heavy bags. Installing an
                                       in-line system can require modification of an airport terminal, including
                                       removal of the existing system, installation of a new baggage handling
                                       system and EDS machines, and the use of an interim solution to screen


                                       Page 12                                     GAO-12-266 Checked Baggage Screening
checked baggage while the in-line system is built. TSA estimates that
depending on the size and complexity of an in-line project, installing an in-
line system can take one to four years at larger (category X and I)
airports.

In 2005, we reported that although TSA made substantial progress in
installing EDS machines, the agency had not conducted a systematic,
prospective analysis to determine which airports could achieve long-term
savings and improve efficiencies and security by installing in-line systems
or, where in-line systems may not be economically justifiable, by making
greater use of stand-alone EDS rather than relying on the labor-intensive
and less efficient ETD screening processes. 13 We recommended that
TSA systematically evaluate baggage screening needs at airports,
including identifying and prioritizing the airports where the benefits—such
as cost savings of screening operations and improved security—of
replacing stand-alone baggage screening machines with in-line systems
are likely to exceed the costs of the systems. TSA concurred and in
response released its Strategic Planning Framework in February 2006,
which identified and prioritized airports based on an analysis of several
factors, including security risk and the amount of estimated cost savings.

In March 2011, we reported that by continuing to replace or modify older
baggage screening systems with more efficient solutions, including in-line
screening systems, TSA could continue to eliminate baggage screener
positions. 14 TSA agreed that the deployment of more efficient systems
offers potential personnel cost savings to the federal government.

In July 2011, we reported on TSA’s efforts to enhance explosives
detection requirements for checked baggage screening technologies and
efforts to ensure that newly acquired and currently deployed explosives
detection technologies meet the enhanced requirements. 15 We
recommended that TSA, among other things, develop a plan to ensure
that new machines, as well as those machines currently deployed in


13
  GAO, Aviation Security: Systematic Planning Needed to Optimize the Deployment of
Checked Baggage Screening Systems, GAO-05-365 (Washington, D.C.: Mar. 15, 2005).
14
     GAO-11-318SP.
15
   GAO, Aviation Security: TSA Has Enhanced Its Explosives Detection Requirements for
Checked Baggage, but Additional Screening Actions Are Needed, GAO-11-740
(Washington, D.C.: July 11, 2011).




Page 13                                        GAO-12-266 Checked Baggage Screening
                                          airports, will be operated at the levels in established requirements and we
                                          recommended that TSA develop a reliable schedule for the EBSP. DHS
                                          concurred with these recommendations and has initiated actions to
                                          implement them.

                                          In February 2012, we reported that we continue to believe that TSA might
                                          achieve savings in screening personnel costs by continuing to replace or
                                          modify older checked baggage screening systems with more efficient
                                          solutions, including in-line screening systems, to the extent possible. 16
                                          TSA reported that since the issuance of GAO’s 2011 report, it had
                                          replaced 60 stand-alone checked baggage screening machines with more
                                          efficient in-line screening systems.

                                          Since fiscal year 2006, about $6.8 billion has been made available to TSA
                                          for activities related to checked baggage screening (see table 1), making
                                          the EBSP one of DHS’s largest acquisition programs.

Table 1: Funding Available for the EBSP for Fiscal Years 2006 to 2012

Dollars in millions
Year                              FY06            FY07        FY08            FY09            FY10          FY11         FY12            Total
               a
EBSP funding                       $670         $1,036         $771          $1,542         $1,294          $781         $732          $6,826
                                          Source: TSA.
                                          a
                                           Amounts available as reported by TSA for activities related to EBSP planning, procurement,
                                          installation (including facility modification), and maintenance and utilities. These amounts do not
                                          include other related costs, such as the cost of screening personnel. Each fiscal year includes $250
                                          million from the Aviation Security Capital Fund (see 49 U.S.C. § 44923) as well as supplemental
                                          appropriations, including over $700 million made available to TSA in fiscal year 2009 through the
                                          American Recovery and Reinvestment Act of 2009, Pub. L. No. 111-5, 123 Stat. at 163. Although
                                          TSA reports $732 million in funding available for the EBSP in fiscal year 2012, the Department of
                                          Homeland Security Appropriations Act, 2012, appropriated approximately $543 million for explosive
                                          detection systems that with the $250 million from the Aviation Security Capital Fund, amounts to
                                          approximately $793 million made available in fiscal year 2012. See Pub. L. No. 112-74, Div. D, 125
                                          Stat. 786, 950-51 (2011).


Cost Share Option for                     Following the terrorist attacks of September 11, 2001, and the enactment
Funding Airport                           of ATSA, airports relied upon various sources of funding to support
Modifications                             security-related capital improvement projects. For example, as enacted,
                                          ATSA authorized the use of Airport Improvement Program (AIP) funds for



                                          16
                                            GAO, Follow-up on 2011 Report: Status of Actions Taken to Reduce Duplication,
                                          Overlap, and Fragmentation, Save Tax Dollars, and Enhance Revenue, GAO-12-453SP
                                          (Washington, D.C.: Feb. 28, 2012).




                                          Page 14                                                 GAO-12-266 Checked Baggage Screening
projects to support the installation of explosive detection systems. 17
Subsequently enacted statutes, such as the Vision 100—Century of
Aviation Reauthorization Act (Vision 100), however, either limited or
precluded the use of AIP to fund projects related to the installation of in-
line systems. 18 TSA, which is solely responsible for procuring and
deploying equipment to screen checked baggage for explosives, also
provides funding in support of related facility modifications. The
Consolidated Appropriations Resolution, 2003, first authorized the use of
LOIs by TSA for airport facility modification projects related to the
installation of in-line baggage screening systems. 19 Although not a
binding commitment of federal funding, LOIs are agreements providing
that TSA will reimburse airports or airlines for a specified percentage of
an eligible project’s cost, subject to the availability of appropriations. This
in turn enables an airport to proceed with a project because the airport
and any investors are aware that the agreed-upon percentage of
allowable costs will likely be reimbursed. The airport or airline is
responsible for its share of the total funding needed to complete the
project and generally must be capable of funding the project in its entirety.

From fiscal years 2003 through 2007, TSA entered into 8 LOI agreements
covering 9 airports. 20 Pursuant to the law then in effect, these LOI
agreements provided for a 75 percent federal cost share of allowable




17
   See Pub. L. No. 107-71, § 119(a)(1), 115 Stat. at 628 (amending 49 U.S.C. § 47102(3))
to authorize use of AIP funding for any required security-related activities after September
11, 2001). During fiscal years 2002 and 2003, six airports used AIP grants to fund facility
modification projects related to installing in-line EDS screening processes.
18
   See Pub. L. No. 108-176, §§ 142, 159(b), 117 Stat. 2490, 2503, 2510-11 (2003)
(amending 49 U.S.C. § 47102(3) to limit the availability of AIP funds for security-related
projects). Subsequent prohibitions on the use of AIP funds for projects related to the
installation of in-line EDS machines may be found in FAA’s annual appropriations statutes
beginning in fiscal year 2004. See, e.g., Pub. L. No. 108-199, 118 Stat. 3, 283; Pub. L. No.
108-447, 118 Stat. 2809, 3203 (2004); and Pub. L. No. 112-55, 125 Stat. 552, 648 (2011).
19
  See Pub. L. No. 108-7, § 367, 117 Stat. 11, 423-24 (2003) (authorizing TSA to enter into
LOIs to replace baggage conveyor systems or to reconfigure terminal baggage areas to
install explosive detection systems if the project would increase security or improve airport
efficiencies without lessening security).
20
 TSA entered into one LOI agreement that covered Los Angeles International Airport and
Ontario International Airport, both of which are operated by Los Angeles World Airports.




Page 15                                            GAO-12-266 Checked Baggage Screening
project costs. 21 Beginning in fiscal year 2008 and in accordance with
Vision 100, any LOI entered into by TSA was to reflect a 90 percent
federal cost share. 22 Since 2008, TSA has entered into 4 more LOIs at a
90 percent federal cost share. As of December 13, 2011, TSA reported
that its net cumulative obligations for all 12 LOIs were $1.46 billion. TSA
also uses OTAs to support airport facility modification projects related to
the installation of checked baggage screening equipment. TSA describes
OTAs, which have become the primary administrative vehicles through
which TSA financially supports such projects, generally as single-year
reimbursable agreements (in contrast to the multiyear LOI agreements).
According to TSA, OTAs take many forms and are generally not required
to comply with federal laws and regulations that apply to contracts or
cooperative agreements, such as the Federal Acquisitions Regulations,
thus enabling the parties to negotiate provisions that are mutually
agreeable. According to TSA, the federal cost share applied to OTAs may
be negotiated, but since fiscal year 2008 TSA has generally followed the
federal cost share applicable to LOIs. As of the end of fiscal year 2011,
TSA had used at least 150 OTAs to support airport facility modifications
related to the installation of in-line systems. According to TSA, these
OTAs have reflected federal cost shares ranging from 50 to100 percent.
As of December 13, 2011, TSA reported that its net cumulative
obligations for all OTAs were $1.74 billion.




21
   The 75 percent federal cost share only applied to projects at medium and large hub
airports; a 90 percent federal cost share applied to projects at any other airport. See Pub.
L. No. 108-7, § 367(e), 117 Stat. at 424. A “medium hub airport” is a commercial service
airport with at least 0.25 percent but less than 1.0 percent of passenger boardings based
on the prior calendar year, while a “large hub airport” is a commercial service airport with
at least 1.0 percent of passenger boardings. 49 U.S.C. § 47102(10), (12). Small and non-
hub airports are commercial service airports with at least .05 percent but less than 0.25
percent, and less than 0.05 percent of passenger boardings, respectively. § 47102 (13),
(23). A “commercial service airport” is a public airport in a state that the Secretary of
Transportation determines has at least 2,500 passenger boardings each year and is
receiving scheduled passenger aircraft service. See § 47102(7).
22
   See 49 U.S.C. § 44923(e) (providing that the federal share for projects at medium or
large hub airports would be 90 percent and 95 percent for projects at any other airports).
Notwithstanding Vision 100, which was enacted in December 2003, the fiscal years 2005
through 2007 DHS appropriations acts maintained the 75 percent federal cost share for
projects under any LOI. See, e.g., Pub. L. No. 109-295, 120 Stat. 1355, 1362-63 (2006).




Page 16                                            GAO-12-266 Checked Baggage Screening
TSA Has Made
Progress in Deploying
Optimal Checked
Baggage Screening
TSA Has Collaborated with     TSA reports that 337 of 446 airports (76 percent) the agency regulates for
Airports to Install Optimal   security have optimal baggage screening systems. The remaining 109 of
Baggage Screening             446 airports (24 percent) do not yet have optimal baggage screening
                              systems in all screening areas. 23 To be considered an airport with an
Systems at 76 Percent of
                              optimal baggage screening system, as TSA considers it and as we define
TSA-Regulated Airports        it for purposes of this report, an airport must have completed installation
                              and activation of the in-line or stand-alone systems that best fits the
                              airport’s screening needs without relying on temporary stand-alone
                              systems. 24 Thus, an airport with an optimal baggage screening system
                              may have a mix of explosive detection systems (EDS or ETD) and
                              configurations (in-line or stand-alone systems), depending on the airport’s
                              needs. TSA officials told us that they plan to deploy equipment for an
                              additional 201 in-line systems in the future, which will include the




                              23
                                 Airports that do not have optimal solutions in place include those with in-line systems
                              under construction and that are not yet screening checked baggage, and those that, for
                              example, continue to rely on temporary stand-alone technology in the lobby. According to
                              TSA, all TSA-regulated airports, regardless of whether they have an optimal solution,
                              remain capable of screening 100 percent of checked baggage for explosives using EDS or
                              ETD machines.
                              24
                                 According to TSA officials, although an airport will not be considered as having an
                              optimal baggage screening system if it relies on temporary stand-alone systems for
                              screening checked baggage, some exceptions exist. For example, temporary stand-alone
                              systems could be installed at airports for testing, special events, or other needs, without
                              affecting the airport’s status related to having an optimal baggage screening system.




                              Page 17                                           GAO-12-266 Checked Baggage Screening
purchase of an estimated 685 new EDS machines for installation at
airports that are not screening with optimal configurations. 25

TSA aims to complete its efforts to deploy optimal screening systems by
using EDS machines as the primary means for screening checked
baggage at all category X, I, II, and III airports while continuing to use
ETD machines as the primary means at category IV airports. 26
Additionally, TSA plans to deploy EDS machines in in-line configurations
at all category X and I airports and in-line or stand-alone configurations at
category II and III airports. At each of the 10 airports we visited, we
observed distinct checked baggage screening needs, based on an
airport’s terminal configuration and the number of passenger boardings.
We discussed an airport’s willingness and financial ability to pay for
facility modifications required to install in-line systems. For each of these
airports, officials and engineers provided documentation (for example,
drawings and blueprints) on distinct facility modification projects to
accommodate baggage screening system upgrades. According to airport
officials, TSA and airport officials work together to determine the most
appropriate baggage screening configuration based on an airport’s
needs. Of the 337 airports where baggage screening systems are optimal
and no longer using temporary solutions, 55 airports use EDS in-line
systems exclusively for their primary checked baggage screening needs,
while 92 airports use EDS stand-alone machines only, and 167 use ETD
systems exclusively. The remaining 23 airports have a mix of systems. 27


25
   Data are based on the most current numbers available as of December 2011. TSA
officials told us that the accuracy of this estimate at any given time depends on an
airport’s willingness and financial readiness. Additionally, TSA officials stated that this
estimate is based on several assumptions and constraints that may affect the number of
new in-line systems deployed in the future, including (1) number of new terminals, terminal
renovations, or other capital projects at airports; (2) changing cost assumptions; and (3) if
an airport is added to, removed from, or changed on the list of TSA-regulated airports. As
of December 2011, TSA had installed 1,933 stand-alone EDS machines, 4,819 ETD
machines, and 261 in-line systems among the 446 TSA-regulated airports (see app. II for
further details).
26
   TSA has a goal of deploying EDS machines for primary screening at all category X, I, II,
and III airports with a minimum peak throughput of 1,500 bags per week. Where this
throughput requirement is not met, primary ETD screening is more cost effective and will
be allowed. Currently, as shown in app. II, table 4, 30 category III airports have ETD
machines for their primary baggage screening, while 20 category IV airports have some
form of EDS screening.
27
   Data are based on the most current numbers available as of December 2011. Airports
that use EDS machines also use ETD machines to assist in resolving alarmed bags.




Page 18                                            GAO-12-266 Checked Baggage Screening
See figure 5 for additional details on airport screening configurations.
Also, see appendix II for more details on the status of efforts to optimize
checked baggage screening systems.

Figure 5: Baggage Screening Configurations at the 337 Airports with Optimal
Systems




Note: Percentages do not add to 100 percent because of rounding.


Of the 337 airports with optimal baggage screening systems, larger
airports were less likely to have completed optimal solutions than smaller
airports. 28 Specifically, 36 percent (10 of 28) of the category X airports
and 49 percent (28 of 57) of the category I airports were considered to
have optimal solutions, whereas 60 percent (46 of 77) of the category II
airports, 76 percent (96 of 127) of the category III airports, and all of the
157 of the category IV airports were considered to be optimally
configured, as shown in figure 6. 29 According to TSA and airport officials,


28
   Data are based on the most current numbers available as of December 2011. TSA also
tracks airports that have begun installing optimal systems but whose systems are not yet
complete.
29
  In addition to the 10 category X airports with optimal systems in all screening areas,
TSA reports that 15 others have optimal systems in some screening areas.




Page 19                                              GAO-12-266 Checked Baggage Screening
this is because the larger airports generally need to install more complex
in-line systems, which are more time and resource intensive to install and
often require a significant amount of airport infrastructure modification and
construction, while the smaller airports, particularly the category IV
airports that rely on the smaller ETD machines, require far less time and
resources to install these systems. Moreover, TSA officials stated that the
in-line systems that best meet the screening needs of larger airports take
longer to plan and build because of the added complexity and scale of the
upgrades and the coordination required among multiple stakeholders.

Figure 6: Number of Airports with Optimal Baggage Screening Systems Compared
to All TSA-Regulated Airports




Page 20                                    GAO-12-266 Checked Baggage Screening
TSA Has Shifted Its Focus   TSA anticipates that in the next 5 years about 60 percent (1,153 of 1,933)
to Replacing Aging          of the EDS machines will reach the end of their useful life of about 10
Machines                    years and will need to be replaced, as shown in figure 7. 30 As a result, to
                            ensure that 100 percent of checked bags continue to be screened as
                            required by ATSA, TSA revised its focus from replacing stand-alone EDS
                            in airport lobbies with in-line systems to replacing its aging fleet of EDS
                            and ETD machines, a process it calls recapitalization. However, TSA
                            reported that it will continue to collaborate with airports or airlines to install
                            optimal in-line systems if it coincides with efforts to recapitalize aging
                            EDS machines or if the existing in-line systems do not meet current TSA
                            standards. In August 2011, TSA issued its EDS and ETD Recapitalization
                            and Optimization Plan, which establishes the method and criteria for
                            prioritizing when and how EBSP recapitalization and optimization will
                            occur. The plan notes that many in-line recapitalization projects will
                            include an optimization component. For example, a number of early in-
                            line screening systems are likely to require optimization, among other
                            things, to improve performance, increase efficiency, and reduce operating
                            costs. At one airport we visited with an early in-line system, we observed
                            a baggage handling system that needed to be replaced because it had
                            sharp curves and steep grades that led to an excessive number of errors
                            and jams. 31 The airport was involved in TSA’s recapitalization pilot
                            program, and airport officials anticipated TSA supporting the optimization
                            of this system as part of recapitalization.




                            30
                                Data are based on the most current numbers available as of December 2011. In
                            addition to age, TSA considers a variety of factors to decide which machines to replace
                            first, such as repair rate and downtime.
                            31
                                TSA is also replacing and modernizing aging ETD machines at all TSA-regulated
                            airports. TSA’s EDS and ETD Recapitalization and Optimization Plan notes that because
                            of the significant number of ETDs deployed during the initial TSA rollout in 2002 and 2003,
                            many of those units have reached or will exceed the 8 years of service in fiscal years 2010
                            and 2011 and are now due to be recapitalized. According to the recapitalization plan,
                            because of the high volume, it was not possible to recapitalize all the ETDs at the 8-year
                            mark in 2010 and 2011. As necessary, TSA will focus on replacing poorly performing units
                            first.




                            Page 21                                           GAO-12-266 Checked Baggage Screening
Figure 7: Age and Number of Deployed EDS Machines as of December 2011




Reducing the Federal
Cost Share for
Eligible Airport
Facility Modification
Projects Could
Reduce TSA’s Costs

Reducing the Federal Cost             Consistent with current law, TSA enters into reimbursable agreements
Share                                 through which it generally funds 90 percent of the cost of an eligible
                                      airport facility modification project to support the installation of an optimal
                                      system, with an airport or airline funding the remaining 10 percent of the




                                      Page 22                                     GAO-12-266 Checked Baggage Screening
project’s cost. 32 If the federal cost share for airport facility modification
projects is reduced, TSA may be able to use available funding to install a
greater number of optimal solutions than currently anticipated. From fiscal
year 2003 through fiscal year 2007, in accordance with law then in effect,
TSA entered into LOIs at a 75 percent federal cost share. 33 Looking
forward, we used TSA’s projections of airport modification costs for each
fiscal year, 2012 through 2030, as presented in its latest LCCE, and
estimated the amount of expenditures for airport modifications that could
be shifted from TSA to airports if the federal government’s cost share
were reduced from the current 90 percent to 75 percent. As discussed
later in this report, we found that TSA’s LCCE data are of questionable
reliability for a precise estimate. However, the data can serve to provide a
rough indication of how much TSA could save if the cost share were
adjusted. Thus, we estimate that if the federal cost share for such projects
returned to the 75 percent TSA applied to many of the reimbursable
agreements it entered into prior to fiscal year 2008, rather than the
current federal share of 90 percent, TSA’s anticipated expenditures for
these modifications would be roughly $300 million less through fiscal year
2030. 34

TSA had previously determined that assigning costs among industry
stakeholders and the nation as a whole is difficult because operational



32
   TSA uses LOIs and OTAs to support facility modification projects related to checked
baggage screening. In general, LOIs are multiyear but nonbinding agreements through
which TSA agrees, subject to the availability of appropriations, to reimburse an airport or
other responsible party for a percentage of allowable airport facility modification costs.
See 49 U.S.C. § 44923(d). OTAs, as described by TSA, are generally single-year
reimbursable agreements used by TSA to fund airport facility modification projects without
the long-term commitment associated with LOIs.
33
   TSA has used OTAs more regularly than LOIs to fund such projects. TSA is able to
negotiate OTA cost shares, and has used cost shares as low as 50 percent and as high
as 100 percent. TSA began implementing a business practice in fiscal year 2008 that
applies the same cost share rates to OTAs as are statutorily required for LOIs.
34
   Based on the level of modification costs currently projected in TSA’s LCCE estimate,
we calculated that TSA could save $339 million through fiscal year 2030. To estimate the
decrease in TSA’s payments for airport modification costs (and the corresponding
increase in those costs paid by airports and, in a few cases, airlines), we totaled the
airport modification costs for the life of the program, fiscal years 2012 through 2030, as
identified in TSA’s LCCE. We then estimated what TSA’s expenditures would be if it pays
75 percent of these costs. All estimates are in 2011 dollars, based on the discount rate
(3.9 percent) that TSA applied in its September 2011 analysis of alternatives submission
to the Office of Management and Budget (OMB).




Page 23                                           GAO-12-266 Checked Baggage Screening
                           improvements to the baggage handling systems and national security
                           benefits are difficult to quantify. This, in turn, makes it difficult to develop a
                           cost share formula that would allow TSA to allocate costs in proportion to
                           benefits. Consistent with the Intelligence Reform and Terrorism
                           Prevention Act of 2004, TSA commissioned the 2006 Baggage Screening
                           Investment Study Working Group to prepare a report for the Aviation
                           Security Advisory Committee, which examined what an appropriate
                           federal government/airport cost share should be for the installation of
                           checked baggage screening equipment. 35 The working group, which
                           consisted of over 60 members representing, among others, TSA, FAA,
                           airports, airlines, designers of baggage handling systems, and financial
                           institutions, were unable to develop a consensus on an appropriate cost
                           share formula, in large part because of the difficulties of measuring
                           benefits, differing views on the federal responsibility for funding capital
                           investments related to baggage screening, and the competing demands
                           on the federal budget. As a result, potential cost share options were not
                           submitted to the Congress as part of DHS’s fiscal year 2006 budget
                           submission.

Airport Officials          Representatives of all 10 airports we visited told us that they benefit from
Acknowledge Benefits       the installation of integrated, in-line baggage screening systems.
from In-line Systems but   Specifically,
Oppose Increasing Their    •   officials from 9 of the 10 airports cited the reduction of passenger
Cost Share                     congestion in airport terminals because stand-alone EDS machines
                               were removed from the lobby or ticketing areas,
                           •   officials from half of the airports noted that in-line systems reduce the
                               number of lost or stolen bags by creating a streamlined process for
                               moving checked baggage directly from where baggage is checked by
                               the passenger and airline to the aircraft, and
                           •   officials from 3 of 10 airports noted that in-line systems facilitate
                               airport growth.




                           35
                              See Pub. L. No. 108-458, § 4019(d), 118 Stat. 3638, 3722 (2004). The Aviation
                           Security Advisory Committee is composed of private sector organizations representing
                           key constituencies affected by aviation security requirements, including, among others,
                           law enforcement, aviation consumer groups, airport operators, airline management, airline
                           labor, and airline security equipment manufacturers. Its mission is to examine areas of
                           civil aviation security as tasked by TSA with the aim of developing recommendations for
                           improving civil aviation security methods, equipment, and procedures.




                           Page 24                                          GAO-12-266 Checked Baggage Screening
However, for various reasons, officials representing 8 of the 10 airports
opposed a reduction in the federal cost share that would increase
airports’ share of modifications costs. 36 Specifically, officials from half of
the airports stated the following four concerns:

•     Assuming a larger share of airport modification costs would pose
      hardships because of current fiscal or funding constraints.
•     Airports incur additional (that is, nonallowable) costs that are
      necessary to building an in-line system, but which TSA will not
      reimburse. Examples of the nonallowable costs the airports cited
      include the costs of designing an in-line system and constructing
      rooms in which screeners manually screen bags that have not
      previously been cleared. 37 As a result, officials from 5 of the 10
      airports we spoke with told us that after necessary, but nonallowable,
      costs were included, the airports were already paying for more than
      10 percent of the modification costs associated with in-line systems. 38
•     Airports have a backlog of capital projects or would rather fund
      projects that will produce additional revenue, such as parking garages
      or larger areas for concessions, than projects that are related to TSA’s
      security responsibilities.
•     TSA will be the primary beneficiary of in-line baggage screening
      systems because the integration and consolidation of these systems
      will enable TSA to reduce the number of baggage screeners and
      provide TSA with other operational efficiencies.




36
     Two airports had no comments.
37
   TSA’s guidance to project sponsors has been evolving. As explained in TSA’s Planning
Guidelines and Design Standards for Checked Baggage Inspection Systems, v. 4.1, app.
F, TSA will provide reimbursement for up to the agreed-upon percentage of allowable
costs; the document also lists the reimbursable and nonreimbursable costs. For example,
TSA does not reimburse costs associated with the building shell or exterior enclosure.
TSA does not reimburse the cost of construction of terminal expansions, whether
necessary to support TSA operations or for other purposes. TSA also does not support
costs related to the construction of new or replacement terminal buildings and
infrastructure unless such construction is proven to be cost-beneficial to TSA.
38
   TSA officials told us that provided a design OTA is in place between TSA and the
airport when design commences, TSA does pay for allowable costs. However, TSA
agreed that an airport’s cost share for the overall project may be higher if the design
includes unallowable costs or if the airport began design activities prior to having an OTA
in place with TSA.




Page 25                                            GAO-12-266 Checked Baggage Screening
TSA’s Cost Estimation
Processes Do Not
Fully Comply with
Best Practices, and
TSA and DHS Are
Working Together to
Establish an
Acquisition Program
Baseline

TSA Estimates That the     TSA’s August 25, 2011, life cycle cost estimate identified a total program
EBSP Will Cost Close to    cost for EBSP of $49.2 billion through fiscal year 2030. 39 The $49.2 billion
$50 Billion through 2030   includes $2.65 billion for EBSP program operation and management;
                           $11.03 billion for capital costs, including, among other things,
                           recapitalization and facility modifications for optimization; $14.89 billion
                           for operations and maintenance of equipment; $18.42 billion for screener
                           salaries; and $2.22 billion for research and development and other
                           miscellaneous related costs. 40 Additionally, TSA officials reported that the
                           program is expected to provide life cycle cost savings of $537 million. 41
                           However, we found that the cost estimates are of questionable reliability
                           for a precise estimate. TSA continues to revise its life cycle cost
                           estimates. For example, its August 2011 EBSP life cycle cost estimate
                           report stated that new requirements, including recapitalization and



                           39
                              Costs are in then-year dollars and include costs to the federal government, airports, and
                           airlines.
                           40
                              An LCCE encompasses all past (or sunk), present, and future costs for every aspect of
                           the program, regardless of funding source. TSA officials noted that some of these costs,
                           such as screener salaries, are EBSP related but are not paid for by the EBSP.
                           41
                              This total is based on life cycle cost savings as submitted in OMB Exhibit 300 for EBSP.
                           TSA’s EBSP net cost savings estimate is in present value dollars and based on the DHS
                           alternative analysis presented in Exhibit 300, which was completed in August 2011 and
                           submitted to OMB. The preferred alternative—a combination of upgrading and
                           recapitalizing checked baggage screening equipment—was estimated to cost $537 million
                           less than the baseline option—an approach where no additional investment is made in
                           optimized systems and equipment is recapitalized as needed.




                           Page 26                                            GAO-12-266 Checked Baggage Screening
                             upgrading the efficiency of early in-line systems, will likely lead to a gap
                             between anticipated program needs and anticipated funding during fiscal
                             years 2012 to 2017, totaling up to $436 million. However, in December
                             2011 TSA officials told us the DHS Acquisition Review Board had
                             requested that TSA revise the EBSP funding plans and projections to
                             more accurately reflect current budget constraints and reduced funding
                             available for the program. 42 According to TSA officials, they plan to
                             complete the revised EBSP planning estimates and funding projections to
                             help eliminate the potential funding gap before the next Acquisition
                             Review Board meeting in May 2012. EBSP senior program officials
                             explained that TSA will address the potential funding gap by
                             (1) controlling the costs associated with engineering initiatives and
                             improvements in technology performance, (2) delaying funding of some
                             new in-line systems and recapitalization projects, and (3) extending the
                             useful life of equipment beyond 10 years in cases where replacement
                             could be delayed.


TSA’s Life Cycle Cost        Although TSA’s methods for developing its LCCE reflect features of best
Estimates Are Not Fully      practices, its methods do not fully adhere to these practices. As
Consistent with Best         highlighted in our past work, a high-quality, reliable cost estimation
                             process provides a sound basis for making accurate and well-informed
Practices for Reliable and   decisions about resource investments, budgets, assessments of
Credible Estimates           progress, and accountability for results and thus is critical to the success
                             of a program. 43 According to the Office of Management and Budget
                             (OMB), federal agencies must maintain current and well-documented
                             estimates of program costs, and these estimates must encompass the
                             program’s full life cycle. 44 Without such an estimate, agencies are at
                             increased risk of making poorly informed investment decisions, securing



                             42
                                The Acquisition Review Board reviews provide an opportunity to determine a program’s
                             readiness to proceed to the next life cycle phase. The directive also requires the
                             Acquisition Review Board chairperson to approve key acquisition documents critical to
                             establishing the program, operational requirements, acquisition baseline, and document
                             testing and support plans.
                             43
                                  See GAO-09-3SP.
                             44
                                OMB Circular No. A-11, Preparation, Submission, and Execution of the Budget; (August
                             2011); OMB Circular No. A-130 Revised, Management of Federal Information Resources
                             (August 2000); and OMB, Capital Programming Guide: Supplement to Office of
                             Management and Budget Circular A-11: Planning, Budgeting, and Acquisition of Capital
                             Assets (Washington, D.C.: August 2011).




                             Page 27                                         GAO-12-266 Checked Baggage Screening
                                           insufficient resources to effectively execute defined program plans and
                                           schedules, and experiencing program cost and schedule overruns and
                                           performance shortfalls. As highlighted in our Cost Estimating and
                                           Assessment Guide, a reliable cost estimate has four characteristics—it is
                                           comprehensive, well documented, accurate, and credible. We reviewed
                                           TSA’s cost estimation procedures for the EBSP and assessed the extent
                                           to which the agency met the four characteristics, as shown in table 2.

Table 2: GAO’s Assessment of TSA’s Method for Developing EBSP Life Cycle Cost Estimates

Characteristic    Description                                                                                                Overall assessmenta
Comprehensive     A comprehensive cost estimate should include all government and contractor costs over                      Partially met
                  the program’s full life cycle, provide sufficient detail to ensure that the cost elements are
                  neither omitted nor double counted, and document all cost-influencing ground rules and
                  assumptions.
Well documented   A well-documented cost estimate should capture in writing such things as the source and Partially met
                  significance of the data used, the calculations performed and their results, and the
                  rationale for choosing a particular estimating method or reference. A well-documented
                  estimate can be easily reconstructed by an outside source and should be reviewed and
                  accepted by management.
Accurate          An accurate cost estimate should be, among other things, based on historical data                          Partially met
                  reflecting most likely costs, adjusted properly for inflation, and validated against an
                  independent cost estimate. An accurate estimate should be updated regularly to reflect
                  material changes in the program and actual cost experience on the program, and steps
                  should be taken to minimize mathematical mistakes.
Credible          A credible cost estimate should discuss any limitations in the analysis caused by                          Minimally met
                  uncertainty or biases surrounding the data and assumptions. Major assumptions should
                  be varied and other outcomes computed to determine how sensitive the estimate is to
                  changes in the assumptions. Risk and uncertainty inherent in the estimate should be
                  assessed and disclosed.
                                           Source: GAO analysis based on the GAO Cost Estimating and Assessment Guide.
                                           a
                                            We defined a characteristic as not met if the agency provided no evidence that satisfied any portion
                                           of the criterion, minimally met if the agency provided evidence that satisfied less than one-half of the
                                           criterion, partially net if the agency provided evidence that satisfied about one-half of the criterion,
                                           substantially met if the agency provided evidence that satisfied more than one-half of the criterion,
                                           and met if the agency provided complete evidence that satisfied the entire criterion.


                                           Our assessment showed that TSA’s EBSP estimates partially met three
                                           characteristics and minimally met one characteristic of a reliable cost
                                           estimate. Specifically, TSA’s cost estimate was as follows:

                                           •     Partially comprehensive because the estimate defines the program,
                                                 reflects the current schedule, is technically reasonable, includes
                                                 assumptions identified by a team of personnel and engineers, and
                                                 provides risks related to detection standards. However, the cost
                                                 estimate is not considered fully comprehensive because it does not
                                                 incorporate costs associated with all security threats, lacks a detailed


                                           Page 28                                                             GAO-12-266 Checked Baggage Screening
    product-oriented work breakdown structure that covers the entire
    scope of work, and lacks a single technical baseline. Without fully
    accounting for life cycle costs, management may have difficulty
    successfully planning program resource requirements and making
    wise decisions. Further, the program lacks a defined end date. A
    reasonable criterion is that the estimate capture at least 10 years of
    costs beyond the planned full operational capability date—the date at
    which optimal systems are fully deployed and operating at all
    locations. However, we cannot determine whether the time frame is
    sufficient because we have not received documentation to support the
    program’s official, planned full operational capability date. According
    to TSA, the EBSP does not have a defined end date for procurement
    because maintaining compliance with the 100 percent screening
    mandate established by ATSA requires TSA to continuously procure
    and replace equipment as it reaches the end of its useful life. TSA
    also believes that it is following DHS acquisition guidance outlined
    under the acquisition decision memorandum dated January 13, 2005,
    for estimating threshold dates. Nevertheless, the EBSP still lacks a
    defined, official full operational capability date, without which we can
    neither determine whether the time frame used in the LCCE is
    sufficient nor verify that the life cycle cost estimate is fully
    comprehensive.
•   Partially documented because TSA used relevant data to help
    develop the estimate. For example, TSA’s estimated price for the
    equipment is based on existing contracts for EBSP equipment
    purchases, maintenance costs, LOI agreements, and OTAs. TSA also
    provided narratives, briefings, and documents to describe the program
    requirements, purpose, technical characteristics, and acquisition
    strategy, and explained how calculations were performed. However,
    TSA did not adequately document many assumptions or
    methodologies underlying its cost model to the extent that would allow
    someone unfamiliar with the cost estimate, using only the available
    documentation, to easily re-create the estimate. For example,
    equipment purchase and hardware upgrade costs were based in part
    on estimates from engineers and contract specialists rather than
    historical or analogous data. Unless ground rules and assumptions
    are clearly documented, the cost estimate will not have a basis for
    areas of potential risk to be resolved. In addition, TSA also provided
    little or no evidence that the assumptions and methodologies




Page 29                                   GAO-12-266 Checked Baggage Screening
     underlying the cost estimate were approved by management. 45 TSA
     officials agreed that additional documentation could improve the
     outside reviewers’ ability to evaluate the estimate. According to TSA,
     the EBSP plans to follow DHS guidance to implement software
     dedicated to the estimation, documentation, and reporting of costs
     during all phases of the EBSP program life cycle to help address
     documentation concerns.
•    Partially accurate because while the estimate is properly adjusted for
     inflation, differences between planned and actual costs are not fully
     documented, explained, or reviewed. In addition, we cannot determine
     whether the cost estimate is unbiased—that is, neither overly
     conservative nor overly optimistic—because the program did not
     perform an uncertainty analysis that meets best practices. While TSA
     agreed that costs should be documented, it could not explain why the
     differences between planned and actual costs were not being fully
     documented, explained, or reviewed.
•    Minimally credible because while TSA identified changes in cost for
     each scenario against the baseline and developed a limited risk
     analysis, TSA did not complete other relevant activities to ensure that
     the estimate accounts for bias and uncertainty. For example, the
     agency did not (1) document in detail the assumptions and
     parameters associated with its sensitivity analysis, such as detailed
     calculations on how each parameter was varied between its minimum
     and maximum values; (2) conduct a fully objective uncertainty
     analysis that derives the point estimate percentile rather than
     assumes it; (3) cross-check major cost elements to see whether
     results are similar; and (4) use an independent cost estimate to
     validate the cost estimate. Given the important role that an
     independent cost estimate provides in developing an objective and
     unbiased assessment of whether the program estimate can be
     achieved, developing and using an independent cost estimate would
     provide decision makers with insight into a program’s potential costs
     and reduce the risk of underfunding a program. TSA officials told us
     they did not perform a complete uncertainty analysis because it was
     too costly and time consuming. TSA officials concurred that an
     independent cost estimate was not done for the EBSP and agreed
     that completing an independent estimate would be helpful.


45
  Among other things, TSA presented no evidence that a formal overview of the
program’s technical foundation, time-phased life cycle cost estimate, ground rules and
assumptions, estimating methodology, sensitivity analysis, or risk and uncertainty analysis
were presented to and reviewed by management.




Page 30                                           GAO-12-266 Checked Baggage Screening
Regarding the credibility of TSA’s estimates, our past work has shown
that program cost estimates that are independently validated help
improve the confidence that the estimate is credible, are needed for
making timely and informed budget decisions, and help reduce the
likelihood of unanticipated program cost growth. 46 Our prior reviews of
several DHS programs, including the EBSP, have also shown that if cost
estimates are not validated in accordance with DHS acquisition
management directive requirements at the start of an acquisition
program, it is difficult to assess whether a program is being deployed
within planned budgets. 47 Further, DHS’s Acquisition Management
Directive requires major program cost estimates to be validated early in
the decision-making process, before programs can receive authorization
for acquisition contracts at the DHS Acquisition Review Board meeting. 48
However, we found that since 2008 the DHS Acquisition Review Board
allowed the EBSP to proceed with acquisition contracts before the LCCE
was independently validated by DHS, which is inconsistent with DHS
policy. 49

In May 2010, the DHS Cost Analysis Division reviewed the EBSP life
cycle cost estimate and found that it needed more comprehensive data
and that its accuracy could not be determined. As of December 2011, the


46
  An independent cost estimate is a comparison with the program estimate to determine
whether it is accurate and realistic. According to our best practices, independent cost
estimates that are validated are generally conducted by an organization outside the
acquisition chain, using the same detailed technical information as the program estimate.
See GAO-09-3SP.
47
  In addition, we previously reported that the EBSP faced recurring programwide costs in
excess of the budget, and that program costs could grow significantly as cost estimates
are revised to include new requirements. GAO-10-588SP.
48
  DHS Acquisition Management Directive 102-01, and DHS Acquisition
Instruction/Guidebook, 102-01-001 (2008). Specifically, according to DHS guidance, cycle
cost estimates are required to be validated before programs are approved for acquisition.
49
   According to DHS, the Director, Cost Analysis Division, located in the Office of the Chief
Procurement Officer, serves as the focal point within DHS for cost analysis and estimating
policy, process, and procedure. In November 2011, DHS officials stated that the Cost
Analysis Division was reorganized and split into the Program Support and Risk Analysis
Division. According to DHS officials, the process for the future oversight of cost estimates
is unclear and is likely to be performed by the newly established Risk Analysis Division,
which will include some key Cost Analysis Division staff. While DHS officials supported the
former Cost Analysis Division’s positive assessment of the EBSP life cycle cost estimates,
they also stated that DHS never completed a comprehensive independent cost estimate
for the EBSP.




Page 31                                            GAO-12-266 Checked Baggage Screening
estimate had not been independently validated by DHS. 50 According to
TSA, DHS guidance that required validation of cost estimates was not in
place until November 2008. The interim DHS directive for LCCEs and
validation was established in November 2008, and the Cost Analysis
Division was designated the authority responsible for independent cost
estimates on January 24, 2010. However, the DHS Acquisition Review
Board did not request a validated LCCE until February 25, 2011. TSA
officials also stated that since the program’s budget circumstances
changed over time, the cost estimate needed to be revised to reflect a
constrained budget for fiscal year 2013 and other program changes. For
example, TSA officials stated that new requirements for the EBSP,
particularly the shift in focus to recapitalization of the aging EDS and ETD
fleet and upgrades to in-line baggage screening systems and threat
detection levels, contributed to continued program cost growth and
delayed efforts to validate the latest LCCEs. As a result, the federal
government’s portion of the cost estimates for the EBSP has increased
from approximately $20.5 billion in fiscal year 2010 to $25.4 billion in
fiscal year 2011—a 24 percent increase. 51 TSA is currently working with
DHS on the validation of the cost estimate for the next Acquisition Review
Board meeting scheduled for May 2012.

According to our 2009 Cost Estimating and Assessment Guide, endorsed
by the OMB and DHS, cost estimates are integral to determining and
communicating a realistic view of likely cost outcomes that can be used to
plan the work necessary to develop, produce, and support a program. 52
Taking steps to ensure that its cost estimates for the EBSP conform to
cost estimating best practices will help provide TSA with a sound basis for
understanding how the program can be sustained in future years.


50
  The DHS guidance uses information in our GAO Cost Estimating and Assessment
Guide as criteria to define a valid estimate as one that is well documented,
comprehensive, accurate, and credible. DHS policy explains that cost estimates should be
validated to ensure that the single best estimate is provided for all resources required to
develop, acquire, field, sustain, and dispose of required capability over the program’s life
cycle. According to DHS officials, a validated LCCE is one that is approved and reviewed
by the former DHS Cost Analysis Division.
51
  This amount is in then-year dollars in billions. These estimates are based on annual
submissions to OMB (DHS Exhibit 300). These cost estimates represent only the historical
and projected life cycle costs to the federal government for the EBSP, whereas the
complete life cycle cost estimate of $49.2 billion includes significant additional costs for
screener salaries, as well as the costs to industry.
52
     GAO-09-3SP.




Page 32                                            GAO-12-266 Checked Baggage Screening
TSA and DHS Are Working   Another foundation for making informed budget decisions is the
to Develop an EBSP        acquisition program baseline, which is to document a program’s critical
Acquisition Program       cost elements, including acquisition costs and life cycle costs. According
                          to DHS’s acquisition guidance, the program baseline is the contract
Baseline                  between the program and departmental oversight officials and must be
                          established at program start to document the program’s expected cost,
                          deployment schedule, and technical performance. Establishing such a
                          baseline at program start is important for defining the program’s scope,
                          assessing whether all life cycle costs are properly calculated, and
                          measuring how well the program is meeting its goals. As we have
                          previously reported, establishing realistic original baseline estimates is
                          important for minimizing the risks of poorly defined requirements and
                          achieving better program outcomes. 53 By tracking and measuring actual
                          program performance against this formal baseline, management can be
                          alerted to potential problems, such as cost growth or changing
                          requirements, and has the ability to take early corrective action. 54

                          However, since the inception of the program more than eight years ago,
                          the EBSP has not had a DHS approved acquisition program baseline and
                          DHS did not require TSA to complete an acquisition program baseline
                          until November 2008. An approved baseline will provide DHS with
                          additional assurances that TSA’s approach is appropriate and that the
                          capabilities being pursued are worth the expected costs. DHS officials
                          told us that several reorganizations of DHS offices responsible for
                          approving the baseline and a lack of functional expertise within the
                          agency contributed to further delays in approving the EBSP acquisition
                          program baseline. According to TSA officials, they have twice submitted
                          an acquisition program baseline to DHS for approval. In November 2009
                          and February 2011, TSA requested approval of a program baseline, but
                          according to DHS officials TSA did not have a fully developed life cycle
                          cost estimate. In November 2011, DHS told TSA that it needed to revise
                          the life cycle cost estimates as well as its procurement and deployment
                          schedules to reflect budget constraints. DHS officials told us that they
                          could not approve the acquisition program baseline as written because


                          53
                               GAO-10-588SP.
                          54
                             According to DHS Acquisition Management Directive 102-01, the baseline requirements
                          must also include a threshold value that is the minimum acceptable value that in the user’s
                          judgment, is necessary to satisfy the need. Failure to achieve a threshold would require
                          creating a new baseline or termination of the program. For more information, see
                          GAO-10-588SP.




                          Page 33                                           GAO-12-266 Checked Baggage Screening
                     TSA’s estimates were significantly over budget. TSA officials stated that
                     TSA is currently working with DHS to amend the draft program baseline
                     for approval. TSA officials stated that they plan to resubmit the revised
                     acquisition program baseline before the next Acquisition Review Board
                     meeting in May 2012. Establishing and approving a program baseline, as
                     DHS and TSA currently plan to do for the EBSP, could help DHS assess
                     the program’s progress in meeting its goals and achieve better program
                     outcomes.


                     TSA’s EBSP is aimed at increasing airport screening efficiencies and
Conclusions          addressing the continuing threat of explosives concealed in checked
                     baggage, at a total estimated cost to the federal government and the
                     private sector of close to $50 billion through fiscal year 2030. 55 Given the
                     size of the federal investment, it is vital that TSA ensures effective
                     stewardship over these resources and conveys useful information to the
                     Congress about the scope and cost of the program. However, the
                     limitations we identified in TSA’s EBSP cost estimates raise questions
                     about their reliability. Taking steps to ensure that its cost estimates meet
                     the four characteristics for high-quality and reliable cost estimates would
                     provide TSA with increased assurance about the reliability of the
                     estimated total cost of the program and better position it to account for all
                     resources and associated costs required to develop, implement, and
                     sustain the EBSP.


                     In order to strengthen the credibility, comprehensiveness, and reliability of
Recommendation for   TSA’s cost estimates and related savings estimates for the EBSP, we
Executive Action     recommend that the Administrator of TSA ensure that its life cycle cost
                     estimates conform to cost estimating best practices.


                     On March 30, 2012, we provided a draft of this report to DHS for its
Agency Comments      review and comment. DHS provided written comments on April 19, 2012,
and Our Evaluation   which are reprinted in appendix IV. In its written comments, DHS
                     concurred with our recommendation that TSA ensure that its life cycle


                     55
                       This includes $2.65 billion for EBSP operation and management; $11.03 billion for
                     capital costs, including, among other things, recapitalization and facility modifications for
                     optimization; $14.89 billion for operations and maintenance of equipment; $18.42 billion
                     for screener salaries; and $2.22 billion for miscellaneous related costs.




                     Page 34                                              GAO-12-266 Checked Baggage Screening
cost estimates conform to cost estimating best practices and discussed
efforts under way to address it. DHS further acknowledged the
importance of producing life cycle cost estimates that are comprehensive,
well documented, accurate, and credible so that they can be used to
support DHS funding and budget decisions. DHS also noted that after
conducting an internal review, TSA is implementing a management
directive that applies DHS guidance and the best practices from the GAO
Cost Estimating and Assessment Guide. As part of this effort, TSA is
(1) establishing a working group and executive board to review program
cost estimates to validate whether the estimates are credible and
affordable, (2) requiring all life cycle cost estimates to be approved by
DHS to ensure consistency and quality across TSA programs,
(3) purchasing and training its employees on specialized cost estimating
software, and (4) initiating hiring actions to hire additional cost estimating
personnel. TSA believes that this will institutionalize cost estimating best
practices within the organization and ultimately allow TSA and the
Department to make better-informed investment decisions. These are
positive steps; however, additional time will be needed to assess whether
they have been fully and consistently implemented in accordance with
GAO best practices.


We are sending copies of this report to the Secretary of Homeland
Security, the Assistant Secretary of the Transportation Security
Administration, and appropriate congressional committees. In addition,
this report is available at no charge on the GAO website at
http://www.gao.gov.

If you or your staff have any questions about this report, please contact
me at (202) 512-8777 or lords@gao.gov. Contact points for our Offices of
Congressional Relations and Public Affairs may be found on the last page




Page 35                                    GAO-12-266 Checked Baggage Screening
of this report. GAO staff who made major contributions to this report are
listed in appendix V.




Stephen M. Lord
Director
Homeland Security and Justice Issues




Page 36                                  GAO-12-266 Checked Baggage Screening
Appendix I: Objectives, Scope, and
              Appendix I: Objectives, Scope, and
              Methodology



Methodology

              We examined the Department of Homeland Security’s (DHS)
              Transportation Security Administration’s (TSA) operation of the Electronic
              Baggage Screening Program (EBSP) to assess the program’s current
              status, alternative cost sharing options, and cost estimates. Specifically,
              we addressed the following questions:

              •   What is the status of TSA’s efforts to install optimal checked baggage
                  screening systems in collaboration with airports?
              •   How would reducing the current federal cost share for eligible airport
                  modification projects from 90 percent to its previous level of 75
                  percent affect the amount that TSA pays for these modifications, and
                  what benefits, if any, do airports report receiving from in-line baggage
                  screening systems?
              •   To what extent are TSA’s cost estimation procedures consistent with
                  best practices and is TSA’s acquisition baseline consistent with DHS
                  guidance?
              To determine the status of TSA’s efforts to install optimal checked
              baggage screening systems, we obtained data as of December 2011 and
              January 2012 from TSA, such as the current number of airports with at
              least one in-line system, and the number of airports with optimal systems.
              We also obtained data on the number of airports configured exclusively
              with in-line screening, the number of airports configured with a mix in-line
              and stand-alone explosives detection systems (EDS), and the number of
              airports using only stand-alone EDS for the same time period. We also
              collected data on the overall number of operational in-line systems and
              EDS and explosives trace detection (ETD) machines as of December
              2011. We reviewed documentation from TSA’s EBSP, including the EBSP
              strategic plans for fiscal years 2006, 2008, and 2009, and the 2011 EDS
              and ETD Recapitalization and Optimization Plan.

              We assessed the reliability of the various data TSA provided about
              airports, including the number of TSA-regulated airports (by category) and
              the numbers of airports with the different configurations of baggage
              screening systems (in-line or stand-alone), and the investment and
              budget expenditure dollar values in the letters of intent and other
              transaction agreements by questioning cognizant TSA officials and
              obtaining extensive documentation about these various data. We found
              these data to be sufficiently reliable for the purposes of our report.

              To determine how reducing the federal cost share from the current 90
              percent to the previous federal cost share of 75 percent for eligible airport
              modification projects may affect the amount that TSA pays for these



              Page 37                                    GAO-12-266 Checked Baggage Screening
Appendix I: Objectives, Scope, and
Methodology




modifications, we calculated estimates based on TSA’s August 2011
projections of how much airport modifications will cost in the future. These
projections represent TSA’s best estimate for how much it will spend on
airport modifications for in-line systems each year from fiscal years 2012
through 2030. We also reviewed the reliability of the cost estimate by
evaluating how well TSA followed best practices detailed in the GAO Cost
Estimating and Assessment Guide (see below). While TSA’s process for
estimating costs only partially meets the characteristics of a reliable cost
estimate, the data can serve to provide a rough indication of how much
could be saved by reducing the federal cost share for optimization. To
assess how the installation of in-line baggage handling systems may
benefit the airports that receive them, we visited a nonrandom sample of
10 airports. We chose these airports based on the size of airport, type of
checked baggage screening systems installed, and status of airport
facility modification completion. We discussed cost share with officials at
each airport representing the airport authority, tenant airlines, and TSA’s
Federal Security Director. In addition, we interviewed officials from the
largest industry associations that represent airport executives, airports
and airlines (the American Association of Airport Executives, the Airports
Council International North America, and the Air Transport Association).
We also interviewed an official from of the Association for Airline
Passenger Rights. In addition, we discussed potential benefits of
optimization with aviation security experts. These results cannot be
generalized to the entire industry, but did provide broader perspectives on
the issues and costs associated with the EBSP.

To assess the extent to which TSA’s methods for estimating costs for
EBSP are consistent with best practices and its acquisition program
baseline is consistent with DHS guidance, we analyzed TSA’s most
recent life cycle cost estimate and recapitalization report finalized in
August 2011. Specifically, we used best practices in the GAO Cost
Estimating and Assessment Guide to evaluate TSA’s estimating
methodologies, assumptions, and results to assess whether the official
cost estimates were comprehensive (i.e., includes all costs), accurate,
well documented, and credible. 1 Our Cost Estimating and Assessment
Guide considers an estimate to be




1
 GAO, Cost Estimating and Assessment Guide: Best Practices for Developing and
Managing Program Costs, GAO-09-3SP (Washington, D.C.: March 2009).




Page 38                                       GAO-12-266 Checked Baggage Screening
Appendix I: Objectives, Scope, and
Methodology




•   comprehensive if its level of detail ensures that all pertinent costs are
    included and no costs are double counted;
•   accurate if it is not overly conservative, is based on an assessment of
    the most likely costs, and is adjusted properly for inflation;
•   well documented if the estimate can be easily repeated or updated
    and can be traced to original sources through auditing; and
•   credible if the estimate has been cross-checked with an independent
    cost estimate and a level of uncertainty associated with the estimate
    has been identified.
We also interviewed the TSA EBSP office’s cost estimating team and its
consultants to obtain a detailed understanding of their methodology, the
cost model, and data. In doing so, we interviewed cognizant program
officials, including the Program Manager and cost analysis team,
regarding their respective roles, responsibilities, and actions in developing
the cost estimate, reviewing it, or both.

We examined data reliability of the cost estimate by doing the following:

•   Obtaining cost estimates and reviewing how each major element was
    calculated with an emphasis on the basis for the estimate and
    strength and quality of the supporting documentation.
•   Verifying that the parameters used to create each estimate were valid
    and applicable by comparing to available cost estimating references,
    posing questions to the cost estimators for clarification, and relying on
    other technical sources for cross-checking.
•   Verifying that calculations were correct for each major element.
•   Verifying that escalation was properly applied and elements rolled up
    accurately to the overall program cost estimate.
We reviewed TSA’s EBSP cost estimates to determine whether the
characteristic was (1) not met if the agency provided no evidence that
satisfied any portion of the criterion, (2) minimally met if the agency
provided evidence that satisfied less than one-half of the criterion,
(3) partially met if the agency provided evidence that satisfied about one-
half of the criterion, (4) substantially met if the agency provided evidence
that satisfied more than one-half of the criterion, and (5) met if the agency
provided complete evidence that satisfied the entire criterion. One analyst
assigned a value ranging from 1 to 5 indicating the extent to which the
agencies met each best practice and averaged the values for the
practices that were associated with each characteristic. A second analyst
independently verified the results. We also interviewed program officials
from TSA and DHS responsible for each cost estimate about the
estimate’s derivation. In doing so, we independently assessed the cost



Page 39                                    GAO-12-266 Checked Baggage Screening
Appendix I: Objectives, Scope, and
Methodology




estimates for the current EBSP, as provided to us in August 2011, against
our best practices.

To understand how TSA is working to make better informed budget
decisions and complying with DHS guidance to develop an acquisition
program baseline, we reviewed DHS guidance on acquisitions and
documents related to TSA’s efforts to coordinate with DHS on developing
an acquisition program baseline for EBSP, which DHS considers the
contract between the program and departmental oversight officials to
document the program’s expected cost, deployment schedule, and
technical performance. We also reviewed EBSP Acquisition Review
Board decisions and relevant acquisition decision memos during the
period 2005 through 2011. Additionally, we interviewed TSA and DHS
officials, including officials in the TSA Chief Financial Officer’s office and
DHS’s Program Accountability and Risk Management Office to identify
what procedures have been put in place to approve the acquisition
program baseline.

To gain a better understanding of issues across all of our objectives,
including the development of optimal systems, status of implementation,
funding challenges, impact of a change in the cost share formula at the
airport level, and cost estimation, we conducted site visits to California,
New York, Massachusetts, Washington, D.C., and Florida to interview
local airport officials, regional TSA officials, and airline representatives. 2
To get a range of airports for our site visits, we made our selections
based on the size of airport, type of checked baggage screening systems
installed, and status of airport facility modification completion. We also
considered recommendations from TSA and industry association officials
about which airports to visit. Because we selected a nonprobability
sample of airports, the information we obtained from these interviews and
visits cannot be generalized to all airports. However, we believe that
observations obtained from these visits provided us with a greater
understanding of the airport officials’ perspectives. On these site visits,
we interviewed airport, airline, and TSA officials responsible for financing,
operating, and installing the checked baggage systems within their
respective airports.



2
  We visited the following airports: Ronald Reagan Washington National, Washington
Dulles International, General Edward Lawrence Logan International, John F. Kennedy
International, Miami International, Monterrey Peninsula, Mineta San Jose International,
San Francisco International, Southwest Florida International, and Westchester County.




Page 40                                           GAO-12-266 Checked Baggage Screening
Appendix I: Objectives, Scope, and
Methodology




We conducted this performance audit from October 2010 through April
2012 in accordance with generally accepted government auditing
standards. 3 Those standards require that we plan and perform the audit
to obtain sufficient, appropriate evidence to provide a reasonable basis
for our findings and conclusions based on our audit objectives. We
believe that the evidence obtained provides a reasonable basis for our
findings and conclusions based on our audit objectives.




3
  We issued preliminary results for this work in 2011 and 2012 as part of GAO’s effort to
identify opportunities that federal agencies or the Congress could consider for reducing
the cost of government operations or enhancing revenue collections for the Treasury. See
GAO, Opportunities to Reduce Potential Duplication in Government Programs, Save Tax
Dollars, and Enhance Revenue, GAO-11-318SP (Washington, D.C.: Mar. 1, 2011), and
2012 Annual Report: Opportunities to Reduce Duplication, Overlap and Fragmentation,
Achieve Savings, and Enhance Revenue, GAO-12-342SP (Washington, D.C.: Feb. 28,
2012).




Page 41                                          GAO-12-266 Checked Baggage Screening
Appendix II: Status of Implementation
              Appendix II: Status of Implementation Efforts
              for TSA’s EBSP



Efforts for TSA’s EBSP

              TSA, through its EBSP, has deployed EDS and ETD machines in a
              variety of in-line and stand-alone configurations at airports to streamline
              airports and TSA operations, reduce screening costs, and enhance
              security. The following three tables provide information on the status of
              checked baggage screening systems. Table 3 highlights the different
              system configurations for airports that have optimal checked baggage
              systems by airport category. Table 4 shows the number of TSA-regulated
              airports with at least one in-line system. Table 5 provides the numbers of
              in-line systems and stand-alone EDS and ETD machines at TSA-
              regulated airports.

              Table 3: Checked Baggage Configurations for TSA-Regulated Airports That Have
              Optimal Systems as of December 2011

                                                                Number of airports (by airport
                                                                         category)
               Types of checked baggage
               configurations                                  X       I      II    III    IV Total
               In-line screening only                          8      25     12      9     1      55
               Mix of EDS in-line and stand-alone              2       3      3      0     0       8
               EDS stand-alone only                            0       0     31     56     5      92
               Mix of stand-alone EDS and primary ETD          0       0      0      1    14      15
               Primary ETD                                     0       0      0     30 137       167
               Total                                          10      28     46     96 157       337
              Source: TSA.




              Table 4: TSA-Regulated Airports with at Least One In-line System as of December
              2011

                                                               Number of airports (by airport
                                                                        category)
                                                               X       I      II    III    IV Total
               Number of TSA-regulated airports               28      57     77    127    157    446
               Number of TSA-regulated airports with at       23      38     18     12      1    92
               least one in-line system
              Source: TSA.




              Page 42                                         GAO-12-266 Checked Baggage Screening
Appendix II: Status of Implementation Efforts
for TSA’s EBSP




Table 5: Numbers of In-line Systems and Stand-alone EDS and ETD Machines for
TSA-Regulated Airports as of December 2011

                                                    Number of baggage screening
                                                  configurations (by airport category)
    Types of baggage screening
    configurations                                 X           I     II    III   IV   Total
    In-line systemsa                             133         88     27    12      1    261
    In-line EDS                                  694        284     60    27      2   1,067
    Stand-alone EDS                              383        172    202    91     18    866
    ETD machines                                2,195      1,324   592    405 304     4,820
Source: TSA.
a
One in-line system may include one or more EDS machines.




Page 43                                            GAO-12-266 Checked Baggage Screening
Appendix III: Our Evaluation of EBSP’s Cost
                                      Appendix III: Our Evaluation of EBSP’s Cost
                                      Estimate Process



Estimate Process

                                      In determining that TSA’s processes for developing EBSP cost estimates
                                      do not fully comply with best practices, we evaluated TSA’s cost
                                      estimation methods against our 2009 Cost Estimating and Assessment
                                      Guide. (See table 6.) We applied the following scale across the four
                                      categories of best practices:

                                      •   Not met: TSA provided no evidence that satisfies any portion of the
                                          criterion.
                                      •   Minimally met: TSA provided evidence that satisfies less than one-half
                                          of the criterion.
                                      •   Partially met: TSA provided evidence that satisfies about one-half of
                                          the criterion.
                                      •   Substantially met: TSA provided evidence that satisfies more than
                                          one-half of the criterion.
                                      •   Met: TSA provided complete evidence that satisfies the entire
                                          criterion.

Table 6: Summary Assessment of TSA’s Electronic Baggage Screening Program Cost Estimate Compared to Best Practices

                  Overall                                                                                       Individual
Characteristic    assessment        Best practice                                                               assessment
Comprehensive     Partially met     The cost estimate includes all life cycle costs.                            Partially met
                                    The cost estimate completely defines the program, reflects the current      Partially met
                                    schedule, and is technically reasonable.
                                    The cost estimate work breakdown structure is product oriented,             Partially met
                                    traceable to the statement of work/objective, and at an appropriate level
                                    of detail to ensure that cost elements are neither omitted nor double
                                    counted.
                                    The estimate documents all cost-influencing ground rules and                Partially met
                                    assumptions.
Well documented   Partially met     The documentation should capture the source data used, the reliability      Partially met
                                    of the data, and how the data were normalized.
                                    The documentation describes in sufficient detail the calculations           Partially met
                                    performed and the estimating methodology used to derive each
                                    element’s cost.
                                    The documentation describes step-by-step how the estimate was               Partially met
                                    developed so that a cost analyst unfamiliar with the program could
                                    understand what was done and replicate it.
                                    The documentation discusses the technical baseline description, and         Substantially met
                                    the data in the baseline are consistent with the estimate.
                                    The documentation provides evidence that the cost estimate was              Minimally met
                                    reviewed and accepted by management.
Accurate          Partially met     The cost estimate results are unbiased, not overly conservative or          Partially met
                                    optimistic and based on an assessment of most likely costs.
                                    The estimate has been adjusted properly for inflation.                      Met




                                      Page 44                                            GAO-12-266 Checked Baggage Screening
                                   Appendix III: Our Evaluation of EBSP’s Cost
                                   Estimate Process




                 Overall                                                                                    Individual
Characteristic   assessment      Best practice                                                              assessment
                                 The estimate contains few, if any, minor mistakes.                         Partially met
                                 The cost estimate is regularly updated to reflect significant changes in   Partially met
                                 the program so that it is always reflecting current status.
                                 Variances between planned and actual costs are documented,                 Not met
                                 explained, and reviewed.
                                 The estimate is based on a historical record of cost estimating and        Partially met
                                 actual experiences from other comparable programs.
Credible         Minimally met   The cost estimate includes a sensitivity analysis that identifies a range Partially met
                                 of possible costs based on varying major assumptions, parameters, and
                                 data inputs.
                                 A risk and uncertainty analysis was conducted that quantified the          Partially met
                                 imperfectly understood risks and identified the effects of changing key
                                 cost driver assumptions and factors.
                                 Major cost elements were cross-checked to see whether results were         Not met
                                 similar.
                                 An independent cost estimate was conducted by a group outside the          Not met
                                 acquiring organization to determine whether other estimating methods
                                 produce similar results.
                                   Source: GAO analysis of TSA’s data.




                                   Page 45                                            GAO-12-266 Checked Baggage Screening
Appendix IV: Comments from the
             Appendix IV: Comments from the Department
             of Homeland Security



Department of Homeland Security




             Page 46                                     GAO-12-266 Checked Baggage Screening
Appendix IV: Comments from the Department
of Homeland Security




Page 47                                     GAO-12-266 Checked Baggage Screening
Appendix V: GAO Contact and Staff
                  Appendix V: GAO Contact and Staff
                  Acknowledgments



Acknowledgments

                  Stephen M. Lord, (202) 512-4379 or lords@gao.gov
GAO Contact
                  In addition to the contact named above, Glenn Davis, Assistant Director,
Staff             and Daniel Rodriguez, Analyst-in-Charge, managed this assignment.
Acknowledgments   Wendy Dye, Daren Sweeney, and Yee Wong made major contributions to
                  the planning and all other aspects of the work. David Alexander and
                  Richard Hung assisted with design, methodology, and data reliability.
                  Chuck Bausell and Jack Wang assisted with economic analysis. Jason
                  Lee, Stacey Steele, and Karen Richey assisted with life cycle cost
                  estimate analysis. Nathan Tranquilli assisted with acquisition and
                  contracting issues. Linda Miller provided assistance in report preparation.
                  Thomas Lombardi provided legal support.




(440887)
                  Page 48                                  GAO-12-266 Checked Baggage Screening
GAO’s Mission         The Government Accountability Office, the audit, evaluation, and
                      investigative arm of Congress, exists to support Congress in meeting its
                      constitutional responsibilities and to help improve the performance and
                      accountability of the federal government for the American people. GAO
                      examines the use of public funds; evaluates federal programs and
                      policies; and provides analyses, recommendations, and other assistance
                      to help Congress make informed oversight, policy, and funding decisions.
                      GAO’s commitment to good government is reflected in its core values of
                      accountability, integrity, and reliability.

                      The fastest and easiest way to obtain copies of GAO documents at no
Obtaining Copies of   cost is through GAO’s website (www.gao.gov). Each weekday afternoon,
GAO Reports and       GAO posts on its website newly released reports, testimony, and
                      correspondence. To have GAO e-mail you a list of newly posted products,
Testimony             go to www.gao.gov and select “E-mail Updates.”

Order by Phone        The price of each GAO publication reflects GAO’s actual cost of
                      production and distribution and depends on the number of pages in the
                      publication and whether the publication is printed in color or black and
                      white. Pricing and ordering information is posted on GAO’s website,
                      http://www.gao.gov/ordering.htm.
                      Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
                      TDD (202) 512-2537.
                      Orders may be paid for using American Express, Discover Card,
                      MasterCard, Visa, check, or money order. Call for additional information.
                      Connect with GAO on Facebook, Flickr, Twitter, and YouTube.
Connect with GAO      Subscribe to our RSS Feeds or E-mail Updates. Listen to our Podcasts.
                      Visit GAO on the web at www.gao.gov.
                      Contact:
To Report Fraud,
Waste, and Abuse in   Website: www.gao.gov/fraudnet/fraudnet.htm
                      E-mail: fraudnet@gao.gov
Federal Programs      Automated answering system: (800) 424-5454 or (202) 512-7470

                      Katherine Siggerud, Managing Director, siggerudk@gao.gov, (202) 512-
Congressional         4400, U.S. Government Accountability Office, 441 G Street NW, Room
Relations             7125, Washington, DC 20548

                      Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800
Public Affairs        U.S. Government Accountability Office, 441 G Street NW, Room 7149
                      Washington, DC 20548




                        Please Print on Recycled Paper.