oversight

Supply Chain Security: CBP Needs to Conduct Regular Assessments of Its Cargo Targeting System

Published by the Government Accountability Office on 2012-10-25.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

               United States Government Accountability Office

GAO            Report to Congressional Requesters




October 2012
               SUPPLY CHAIN
               SECURITY
               CBP Needs to
               Conduct Regular
               Assessments of Its
               Cargo Targeting
               System




GAO-13-9
                                               October 2012

                                               SUPPLY CHAIN SECURITY
                                               CBP Needs to Conduct Regular Assessments of Its
                                               Cargo Targeting System
Highlights of GAO-13-9, a report to
congressional requesters




Why GAO Did This Study                         What GAO Found
The U.S. economy is dependent on the           U.S. Customs and Border Protection (CBP), within the Department of Homeland
expeditious flow of millions of tons of        Security (DHS), employs a risk-based approach that uses the Automated
cargo each day. Cargo containers are           Targeting System (ATS) and other tools to identify (target) maritime cargo
an important instrument of global trade        shipments for further examination. ATS is a web-based enforcement and
but also can present security concerns.        decision support system that includes a set of rules to assess the risk level for
CBP is responsible for administering           each arriving cargo shipment. This set of rules is referred to as the maritime
container security programs, and its           national security weight set (weight set) because each rule in the set has a
strategy for securing maritime cargo           specific weighted value assigned to it. CBP classifies the risk scores from the
containers includes analyzing                  weight set as low, medium, or high risk. CBP policy states that a shipment’s risk
information to identify shipments that         score is to determine, in part, actions taken by CBP officers (targeters) at the
may contain terrorist weapons or other         ports. Specifically, targeters are generally required to review shipment data for all
contraband. Because CBP has                    medium-risk and high-risk shipments and hold high-risk shipments for
insufficient resources to examine every        examination. The risk score, however, is not the sole factor that determines
container, targeters use ATS to target
                                               whether a targeter reviews the data for a shipment or whether CBP examines a
which container shipments should be
                                               shipment. In particular, targeters at each of the six ports GAO visited explained
examined. GAO was asked to assess
CBP’s targeting efforts. This report
                                               that they use the ATS risk score as a starting point for the targeting process but
addresses (1) how ATS supports                 that their decisions regarding which shipments to examine are ultimately based
CBP’s targeting of maritime cargo              on additional research. Targeters at the six ports GAO visited said they also use
container shipments for national               tools outside of ATS, such as web searches, to research shipments.
security purposes and (2) the extent to
which CBP assesses the effectiveness           CBP efforts to assess the weight set’s effectiveness in identifying the risk of
of ATS’s national security targeting           shipments have been limited. CBP has performance measures—represented by
rules. GAO analyzed fiscal year 2011           the percentage of shipments targeted as high risk that contain a threat and the
CBP data on shipments and containers           percentage of shipments targeted as high risk that do not contain a threat—that
arriving at U.S. ports and containers          enable CBP to determine the accuracy of the weight set, given a particular
scanned at these ports. GAO also               workload or examination rate. However, CBP did not assess the weight set to
visited six CBP units selected on the          verify its effectiveness when implementing an updated version in early 2011.
basis of the percentage of maritime            Prior to implementing the updated version of the weight set, CBP assessed the
shipments that were scored as high             potential impact of the update on CBP’s workload but did not conduct an
risk or medium risk for national security      assessment to determine whether the updated version of the weight set would be
purposes at these locations in fiscal          more effective in identifying high-risk shipments than the previous version or
year 2011, among other factors. GAO            other alternatives. Assessing the potential effectiveness of alternative versions of
also analyzed documents, such as
                                               the weight set prior to selecting one for implementation could help CBP make
CBP’s ATS performance measures.
                                               more informed decisions about future updates. Doing so could also provide CBP
What GAO Recommends                            reasonable assurance that the version it selects is the most effective of the
                                               alternatives and is more effective than the previous version it replaces.
GAO recommends that CBP (1) ensure             Furthermore, since implementing the updated version of the weight set in early
that future updates to the weight set          2011, CBP has not regularly assessed the weight set to monitor its performance
are based on assessments of its                and to help determine when changes are needed. For example, CBP conducted
performance and (2) establish targets
                                               the first assessment of the current version of the weight set, using the
for performance measures and use
                                               performance measures, in the summer of 2012—18 months after the weight set’s
those measures to regularly assess
effectiveness of the weight set. DHS
                                               implementation in early 2011. Regular assessments of the weight set’s
concurred with these                           effectiveness could help CBP determine when updates are needed in a timelier
recommendations.                               manner and ensure that targeters have the best information available to make
                                               targeting decisions. Moreover, CBP has not established targets for the
View GAO-13-9. For more information, contact   performance measures so that it is not clear whether a particular change in the
Stephen L. Caldwell at (202) 512-9610 or
caldwells@gao.gov.
                                               weight set’s performance is significant enough to suggest that changes are
                                               needed to improve the effectiveness of the weight set.


                                                                                        United States Government Accountability Office
Contents


Letter                                                                                            1
                       Background                                                                 8
                       CBP Uses ATS and Other Tools to Target Maritime Cargo
                         Shipments for National Security Purposes                               15
                       CBP’s Efforts to Assess the Weight Set Have Been Limited                 19
                       Conclusions                                                              25
                       Recommendations for Executive Action                                     26
                       Agency Comments                                                          26

Appendix I             Federal Strategy for Ensuring the Security of Maritime Cargo
                       Container Shipments                                                      28



Appendix II            Past Audit Findings and Recommendations to Improve the Targeting
                       Process                                                                  30



Appendix III           Comments from the Department of Homeland Security                        38



Appendix IV            GAO Contact and Staff Acknowledgments                                    40



Related GAO Products                                                                            41



Tables
                       Table 1: Description of CBP’s Core Cargo Security Programs               28
                       Table 2: Status of GAO Recommendations from February 2004                31


Figure
                       Figure 1: Key Steps for Targeting High-Risk Shipments throughout
                                the Maritime Supply Chain Process                               10




                       Page i                                         GAO-13-9 Supply Chain Security
Abbreviations

A&T               Analysis and Targeting Division
ATS               Automated Targeting System
ATU               Advance Targeting Unit
CBP               U.S. Customs and Border Protection
CERTS             Cargo Enforcement Reporting and Tracking System
DHS               Department of Homeland Security
FMFIA             Federal Managers' Financial Integrity Act
FPR               false positive rate
NII               nonintrusive inspection
NTC-C             National Targeting Center–Cargo
OIG               Office of Inspector General
OMB               Office of Management and Budget
TPR               true positive rate
WMD               weapons of mass destruction



This is a work of the U.S. government and is not subject to copyright protection in the
United States. The published product may be reproduced and distributed in its entirety
without further permission from GAO. However, because this work may contain
copyrighted images or other material, permission from the copyright holder may be
necessary if you wish to reproduce this material separately.




Page ii                                                    GAO-13-9 Supply Chain Security
United States Government Accountability Office
Washington, DC 20548




                                   October 25, 2012

                                   The Honorable Susan M. Collins
                                   Ranking Member
                                   Committee on Homeland Security and Governmental Affairs
                                   United States Senate

                                   The Honorable Cliff Stearns
                                   Chairman
                                   The Honorable Diana DeGette
                                   Ranking Member
                                   Subcommittee on Oversight and Investigations
                                   Committee on Energy and Commerce
                                   House of Representatives

                                   The economic well-being of the United States is dependent on the
                                   expeditious flow of millions of tons of cargo each day. According to the
                                   U.S. Department of Transportation, the majority of U.S. imports arrive by
                                   ocean vessel, and much of that is transported in cargo containers. 1 In
                                   fiscal year 2011, for example, about 13.4 million cargo containers arrived
                                   at U.S. seaports. Cargo containers are an important segment of the
                                   global supply chain—the flow of goods from manufacturers to retailers—
                                   and can present significant security concerns. Within the federal
                                   government, U.S. Customs and Border Protection (CBP), part of the
                                   Department of Homeland Security (DHS), is responsible for administering
                                   container security and reducing the vulnerabilities associated with the
                                   supply chain. Balancing security concerns with the need to facilitate the
                                   free flow of commerce, part of CBP’s mission, remains an ongoing
                                   challenge for the public and private sectors alike. 2 CBP officials believe
                                   that the likelihood of terrorists smuggling weapons of mass destruction
                                   (WMD) into the United States in cargo containers is relatively low;
                                   however, the consequences of such an event could be catastrophic.


                                   1
                                    U.S. Department of Transportation, Research and Innovative Technology Administration,
                                   Bureau of Transportation Statistics, America’s Container Ports: Linking Markets at Home
                                   and Abroad (Washington, D.C.: January 2011).
                                   2
                                    In addition to its priority mission of keeping terrorists and their weapons out of the United
                                   States, CBP is also responsible for securing the border, facilitating international trade and
                                   travel, collecting duties, and enforcing numerous U.S. laws and regulations pertaining to
                                   immigration and illicit drugs, among other things.




                                   Page 1                                                       GAO-13-9 Supply Chain Security
Although there have been no known incidents of cargo containers being
used to transport WMD, ensuring the security of cargo containers remains
an important role for CBP given that criminals have exploited containers
for other illegal purposes, such as smuggling weapons, people, and illicit
substances.

Since September 11, 2001, Congress has passed various laws to
address concerns about the security of cargo containers in the global
supply chain. The enactment of the Maritime Transportation Security Act
of 2002 3 called for the establishment of a program to evaluate and certify
secure systems of international intermodal transportation, including
standards and procedures for screening and evaluating cargo containers
prior to loading onto vessels and for securing and monitoring cargo while
in transit. 4 In 2006, the Security and Accountability For Every Port Act
was enacted, 5 which required, among other things, that pilot projects be
established at three ports to test the feasibility of scanning 100 percent of
U.S.-bound cargo containers at foreign ports. 6 Subsequently, the
Implementing Recommendations of the 9/11 Commission Act of 2007
(9/11 Act) 7 required, among other things, that by July 2012, 100 percent
of U.S.-bound cargo containers be scanned at foreign ports with both
radiation detection and nonintrusive inspection (NII) equipment before
being placed on U.S.-bound vessels. 8 In May 2012, the Secretary of
Homeland Security authorized a 2-year extension—until July 2014—of
the deadline for implementing the requirement that containers not enter




3
Pub. L. No. 107-295, 116 Stat. 2064.
4
See 46 U.S.C. § 70116.
5
Pub. L. No. 109-347, 120 Stat. 1884.
6
 6 U.S.C. § 981. A similar requirement was enacted that same year by the Department of
Homeland Security Appropriations Act, 2007 (Pub. L. No. 109-295, 120 Stat. 1355 (2006))
and is codified at 6 U.S.C. § 981a. Both statutes specify scanning as examination with
both radiation detection equipment and nonintrusive imaging equipment. 6 U.S.C.
§§ 981(a), 981a(a)(1).
7
Pub. L. No. 110-53, § 1701(a), 121 Stat. 266, 489-90 (amending 6 U.S.C. § 982(b)).
8
 Radiation detection equipment detects radiation being emitted from a container, and
through an NII scan, CBP can identify anomalies in a container’s image that could, among
other things, indicate the presence of shielding material.




Page 2                                                   GAO-13-9 Supply Chain Security
the United States unless they were scanned at foreign ports prior to being
loaded on vessels. 9

We reported in October 2009 that CBP faced numerous challenges in
implementing 100 percent scanning at pilot ports, and on the basis of
work we have completed since then we know that CBP has not yet
achieved 100 percent scanning. 10 In October 2009, we recommended,
among other things, that CBP conduct a feasibility analysis of
implementing the 100 percent scanning requirement and provide the
results to Congress along with any suggestions of cost-effective
alternatives to implementing the 100 percent scanning requirement, as
appropriate. DHS stated that CBP concurred with these
recommendations, but CBP has not yet taken action to address them. In
its report to Congress in May 2012 on the planned deadline extension,
DHS stated that it recognizes the need to proceed with container security
programs in a manner that maximizes the security of maritime cargo and
facilitates its movement and reported that it plans to continue working with
other federal agencies and international partners to develop technology
and enhance risk management processes, in addition to continuing
existing programs that enhance cargo security. 11 However, given that the
feasibility of 100 percent scanning remains in doubt and DHS and CBP
have not identified alternatives that could achieve the same goals as 100
percent scanning, uncertainty persists regarding the scope of DHS’s and
CBP’s existing container security programs and how these programs will
collectively affect the movement of goods between trading partners.




9
 The 9/11 Act scanning provision includes possible extensions for containers loaded at a
port or ports for which DHS certifies that at least two out of a list of specific conditions
exist. Among others, these conditions include the following: (1) adequate scanning
equipment is not available or cannot be integrated with existing systems, (2) a port does
not have the physical characteristics to install the equipment, or (3) use of the equipment
will significantly affect trade capacity and the flow of cargo. See 6 U.S.C. § 982(b)(4).
10
  GAO, Supply Chain Security: Container Security Programs Have Matured, but
Uncertainty Persists over the Future of 100 Percent Scanning, GAO-12-422T
(Washington, D.C.: Feb. 7, 2012), and Supply Chain Security: Feasibility and Cost-Benefit
Analysis Would Assist DHS and Congress in Assessing and Implementing the
Requirement to Scan 100 Percent of U.S.-Bound Containers, GAO-10-12 (Washington,
D.C.: Oct. 30, 2009).
11
  DHS, Scanning of Maritime Cargo Containers: Fiscal Year 2012 Report to Congress
(Washington, D.C.: May 3, 2012).




Page 3                                                       GAO-13-9 Supply Chain Security
CBP’s strategy for securing the maritime supply chain consists of
programs that intersect with key points in the supply chain. These
programs include, among other things, analyzing information to identify
shipments that may be at high risk of transporting WMD or other
contraband; working with foreign governments to examine U.S.-bound,
high-risk shipments at foreign ports; and examining U.S.-bound, high-risk
shipments that were not examined overseas upon their arrival at a
domestic port. 12 To aid in this process, CBP uses the Automated
Targeting System (ATS), which is an intranet-based enforcement and
decision support system that compares traveler, cargo, and conveyance
information against intelligence and other enforcement data. Among other
things, ATS uses a set of rules that assess different factors in data
provided by supply chain parties, such as importers, to determine the risk
level for a shipment. CBP officers (targeters) use information in ATS to
identify (target) which shipments should be held for an examination,
which may include an NII scan or a physical inspection. 13 Because CBP
does not scan 100 percent of U.S.-bound cargo containers, the
effectiveness of CBP’s security strategy depends on CBP’s ability to use
ATS, among other tools, to effectively target shipments in the supply
chain that pose the greatest security risks.




12
  CBP refers to the automated process of analyzing data and classifying shipments by risk
level as screening, and according to CBP, it screens (but does not scan) all U.S.-bound
cargo shipments before they are loaded onto vessels at foreign ports. In this report, we
discuss the screening process in terms of assessing the risk of a shipment. An
examination refers to either (1) the scanning of a container or other cargo conveyance
using large-scale NII technology, which may use X-rays or gamma rays to create an
image of the contents of the container or other conveyance, or (2) a physical inspection of
a container or other cargo conveyance. If the results of an NII scan indicate that a threat
may be present, CBP may choose to conduct a physical inspection. In addition to an NII
exam, scanning can also refer to the use of radiation detection equipment, such as
radiation portal monitors. According to CBP, 99 percent of containers are scanned through
radiation portal monitors prior to leaving a domestic port.
13
  In this report, we use the term “targeting” to refer to the synthesis and use of information
from a variety of sources, including the results of screening, to identify shipments that may
be a potential security risk.




Page 4                                                       GAO-13-9 Supply Chain Security
In response to your request, we analyzed certain aspects of CBP’s
maritime national security targeting efforts. Specifically, this report
addresses the following objectives:

•    How does ATS support CBP’s process for targeting maritime cargo
     container shipments for national security purposes?
•    To what extent does CBP assess the effectiveness of the national
     security targeting rules in ATS?

To address the first objective, we obtained data from CBP for each of the
115 U.S. seaports for fiscal year 2011 on the number of (1) shipments it
placed in each of three categories—high risk, medium risk, and low risk—
arriving at each of the ports; 14 (2) container arrivals at these ports; and (3)
containers scanned at these ports using NII equipment. We selected and
visited six CBP units responsible for targeting at domestic ports.
Specifically, these ports were selected from among the largest ports in
the United States using the following criteria: 15 (1) the percentage of
maritime shipments that were scored as high risk or medium risk for
national security purposes, (2) the percentage of cargo containers that
were examined using NII equipment, and (3) whether a CBP official from
the port participated in the most recent CBP conference to discuss
changes to ATS cargo targeting rules. 16 The six targeting units we visited


14
  CBP collects data on the number of shipments as well as the number of containers
arriving in the United States. A shipment is the tender of one lot of cargo at one time from
one shipper to one recipient. In some cases, a shipment will refer to all the contents in a
single container. In other cases, a shipment may refer to the cargo in multiple containers.
Additionally, a single container could hold multiple shipments from different supply chain
parties.
15
   We limited our selection to the 20 ports with the largest volume of arriving shipments in
fiscal year 2011 because the consideration of our other criteria—specifically the
percentage of high-risk or medium-risk shipments and the percentage of containers
scanned using NII equipment—could be disproportionately influenced by ports with
smaller volumes.
16
  According to CBP officials, prior to making an update to the weight set, CBP hosts a
rules conference to discuss potential updates to the rules in the weight set. In February
2009, CBP hosted a conference (the Importer Security Filing rules conference) in which
CBP officials explained policies regarding the receipt of new data from importers and
vessel carriers and how the data were to be used in targeting. Targeters from various
ports attended this conference as representatives of their ports and subject matter
experts. According to CBP officials, this conference was different from rules conferences
in the past. However, CBP could not provide attendee lists for the rules conference that
preceded the Importer Security Filing conference, and therefore we used attendance at
the Importer Security Filing conference as part of our criteria.




Page 5                                                      GAO-13-9 Supply Chain Security
were responsible for targeting efforts at 15 ports that collectively received
about 60 percent of the maritime shipments that arrived in the United
States in fiscal year 2011. As part of these site visits, we observed port
operations, including the scanning of containers. At each location, we
also interviewed CBP targeters, including those who participated in the
most recent CBP conference to discuss changes to cargo targeting rules,
and observed their use of ATS and other tools to conduct cargo targeting
activities. The results from our visits to these six targeting units cannot be
generalized to ports nationwide; however, visits to these locations allowed
us to directly observe the targeting process and provided insights into
how ATS assigns risk scores to maritime shipments and how CBP
integrates the scores into its targeting process. We synthesized the
information from these site visits to describe the targeting process and
also analyzed CBP policies and guidance, such as CBP’s National
Maritime Targeting Policy and course materials from CBP’s Sea Cargo
Targeting Training. We also visited CBP’s National Targeting Center–
Cargo (NTC-C) to interview targeters responsible for conducting national-
level targeting and to observe their targeting activities. 17

To address the second objective, we analyzed CBP’s performance
measures related to CBP’s national security targeting rules in ATS that
are used to assess potential risks in maritime cargo container shipments.
These performance measures include the true positive rate and false
positive rate, as discussed later in this report. In particular, we reviewed a
consulting firm’s evaluation of ATS from 2006 and CBP’s project plan for
implementing the most recent update to the targeting rules, including
CBP’s plans for monitoring ATS through the use of performance
measures and an established methodology. We also reviewed our past
work on ATS and its effectiveness, as well as an audit report from the
DHS Office of Inspector General. 18 The Office of Inspector General’s
assessment of CBP’s modifications to the ATS national security targeting
rules was based on a series of interviews with agency officials and a



17
  NTC-C analyzes advance cargo tactical and strategic information using ATS before
shipments reach the United States. NTC-C also promotes information sharing with other
federal agencies and foreign governments to detect and seize threats at U.S. and foreign
ports.
18
  DHS Office of Inspector General, Cargo Targeting and Examinations, OIG-10-34
(Washington, D.C.: Jan. 6, 2010). Because the reports we issued in February 2004 and
August 2006 regarding CBP’s targeting practices contain sensitive information, they are
not publicly available.




Page 6                                                    GAO-13-9 Supply Chain Security
review of relevant documentation and was sufficient to address the issue
of documentation of changes to the ATS rules that we present in this
report. We analyzed documentation of CBP’s most recent update to the
national security targeting rules, which CBP implemented in early 2011,
and the extent to which this documentation addressed effectiveness.
Specifically, we evaluated CBP’s impact assessments, which provide
information on the number of shipments the updated rules would assess
as high risk, thereby affecting CBP’s examination workload. In addition,
we evaluated the extent to which CBP used its methodology to assess
the current national security targeting rules and analyzed the results of
CBP’s assessments. To assess the reliability of the results of CBP’s
assessments, we reviewed documentation on the methodology created
for CBP by a consulting firm in 2006. We interviewed knowledgeable CBP
officials about any adjustments to this methodology since the contract
expired and CBP analysts began conducting the performance
assessments. On the basis of this information we determined that the
results of the assessments are sufficiently reliable for the purposes of our
report. To determine the extent to which CBP conducts such
assessments on a regular basis, we analyzed documentation of recent
assessments of the national security targeting rules conducted in spring
2011 and summer 2012. We compared this information with key elements
for a risk management approach and Standards for Internal Control in the
Federal Government. 19 We also reviewed our prior work on risk
management practices and compared our analysis of CBP’s actions and
assessments with those practices. Finally, we interviewed officials at CBP
headquarters who are responsible for maintaining and updating ATS to
obtain information on past efforts to assess and update the national
security targeting rules.

DHS deemed some of the information in a draft version of this report as
sensitive, and therefore, this report omits sensitive details regarding
specific information available in ATS, examples of how targeters may use


19
  GAO, Standards for Internal Control in the Federal Government, GAO/AIMD-00-21.3.1
(Washington, D.C.: November 1999). These standards, issued pursuant to the
requirements of the Federal Managers’ Financial Integrity Act of 1982 (FMFIA), provide
the overall framework for establishing and maintaining internal control in the federal
government. Also pursuant to FMFIA, the Office of Management and Budget (OMB)
issued Circular A-123, revised December 21, 2004, to provide the specific requirements
for assessing the reporting on internal controls. Internal control standards, and the
definition of internal control in OMB Circular A-123, are based on GAO’s Standards for
Internal Control in the Federal Government.




Page 7                                                   GAO-13-9 Supply Chain Security
                       that information, and specific dates associated with changes to CBP’s
                       targeting criteria. These omissions did not affect the presentation of the
                       key information and findings that support our conclusions and
                       recommendations.

                       We conducted our work from October 2011 through October 2012 in
                       accordance with generally accepted government auditing standards.
                       Those standards require that we plan and perform the audit to obtain
                       sufficient, appropriate evidence to provide a reasonable basis for our
                       findings and conclusions based on our audit objectives. We believe that
                       the evidence obtained provides a reasonable basis for our findings and
                       conclusions based on our audit objectives.

Background
CBP’s Maritime Cargo   CBP’s security strategy for maritime cargo uses a risk-based approach to
Container Security     focus limited resources on targeting and examining cargo shipments that
Strategy               pose a potential risk while allowing other cargo shipments to proceed
                       without unduly disrupting commerce into or out of the United States. The
                       strategy is based, in part, on obtaining advance cargo information. In
                       particular, through what is referred to as the 24-hour rule, CBP generally
                       requires vessel carriers to electronically transmit cargo manifests to CBP
                       24 hours before cargo is loaded onto U.S.-bound vessels at foreign
                       ports. 20 Through the Importer Security Filing and Additional Carrier
                       Requirements (known as the 10+2 rule), CBP requires importers and
                       vessel carriers to provide data elements for improved identification of
                       containerized cargo shipments that may pose a risk for terrorism. 21
                       Importers are responsible for supplying CBP with 10 shipping data
                       elements—such as country of origin—24 hours prior to loading, while
                       vessel carriers are required to provide 2 data elements—container status




                       20
                         19 C.F.R. § 4.7(b). Cargo manifests are prepared by the ocean carrier and are
                       composed of bills of lading for each shipment loaded onto a vessel to describe the
                       contents of the shipments. Bills of lading are documents issued by a carrier describing the
                       goods, the details of the intended voyage, and the conditions of transportation.
                       21
                         Importer Security Filing and Additional Carrier Requirements, 73 Fed. Reg. 71,730 (Nov.
                       25, 2008) (codified at 19 C.F.R. pts. 4, 12, 18, 101, 103, 113, 122, 123, 141, 143, 149,
                       178, & 192).




                       Page 8                                                     GAO-13-9 Supply Chain Security
messages and stow plans 22—that are not required by the 24-hour rule.
Other aspects of CBP’s maritime cargo container security strategy include
using technology, such as radiation detection equipment, to detect
potential threats and partnering with foreign governments and the trade
industry to examine containers prior to their arrival in the United States
and implement security measures throughout the supply chain process,
respectively. A brief description of the core programs that constitute
CBP’s security strategy for cargo containers is provided in appendix I.
Throughout the supply chain process, and underlying many of the
programs within CBP’s security strategy for cargo containers, CBP
assesses the national security risks posed by shipments throughout the
supply chain process, as shown in figure 1.




22
  Container status messages report terminal container movements, such as loading and
discharging the vessel, and report the change in the status of containers, such as if they
are empty or full. Container status messages also report conveyance movements, such as
vessel arrivals and departures. A vessel stow plan includes information such as the vessel
operator, voyage number, the stow position of each container, hazardous material code (if
applicable), and the port of discharge.




Page 9                                                    GAO-13-9 Supply Chain Security
Figure 1: Key Steps for Targeting High-Risk Shipments throughout the Maritime Supply Chain Process




                                        a
                                         The Container Security Initiative places CBP staff at participating foreign ports to work with host
                                        country customs officials to target and examine high-risk cargo containers for WMD before they are
                                        shipped to the United States. CBP officials identify the containers that may pose a risk for terrorism
                                        and request that their foreign counterparts examine the contents of the containers.
                                        b
                                        Host government officials at foreign ports that do not participate in the Container Security Initiative
                                        may also scan containers using radiation portal monitors as part of their operations.



                                        Page 10                                                              GAO-13-9 Supply Chain Security
Role of ATS in CBP’s        According to CBP, ATS is the cornerstone of CBP’s targeting efforts that
Maritime Cargo Container    underlie the other programs that constitute CBP’s security strategy for
Security Strategy           maritime cargo containers. CBP targeters review shipment records in
                            ATS prior to the cargo being loaded onto U.S.-bound vessels, during
                            shipment transit, and upon arrival at domestic ports to identify potential
                            threats and determine if additional action, such as an examination, is
                            required. When shipment data are updated with additional or amended
                            information, CBP targeters using ATS might identify new risks or mitigate
                            previously identified risks. These targeting efforts, and their reliance on
                            ATS, are key to the operations of CBP’s other security programs. For
                            example, through ATS, CBP determines which shipments may be
                            scanned overseas as part of the Container Security Initiative in an effort
                            to prevent potentially dangerous cargo from being loaded onto U.S.-
                            bound vessels. Additionally, through ATS, CBP is able to reduce the
                            likelihood of examinations for members of the Customs-Trade Partnership
                            Against Terrorism. The Customs-Trade Partnership Against Terrorism is
                            a program through which CBP provides facilitated processing, such as
                            reduced likelihood of security-based examinations, for members that
                            implement the program’s minimum security criteria or guidelines and best
                            practices.

                            The overall effectiveness of CBP’s strategy for securing maritime cargo
                            containers depends largely on the effectiveness of ATS. We have
                            previously reported on CBP’s efforts to assess the effectiveness of ATS
                            and have made recommendations to help CBP achieve the objectives of
                            its overall maritime cargo targeting strategy. In response to these
                            recommendations, CBP has made improvements to ATS and its targeting
                            process, but in some cases CBP’s implementation efforts have been
                            slow, leaving CBP without the benefits of these improvements for several
                            years. For more information about our prior work and recommendations
                            regarding ATS, see appendix II.


ATS and National Security   CBP targeters use ATS in the targeting process to help them determine
Targeting Rules             whether to take further security actions, such as holding the shipment for
                            examination, for cargo shipments they are reviewing. ATS consolidates
                            data from various sources to create a single, comprehensive record for
                            each U.S.-bound shipment. For example, carriers and importers provide
                            data in compliance with the 24-hour rule and the 10+2 rule through other
                            CBP systems, and these systems automatically feed the data into ATS.




                            Page 11                                         GAO-13-9 Supply Chain Security
ATS assesses and presents these data for every cargo shipment,
including containerized shipments and bulk shipments. 23 For
containerized shipments, a single shipment may consist of one or more
containers, or multiple shipments may be consolidated into a single
container for transport. Because ATS collects and presents data on
shipments, CBP targets shipments—rather than individual containers—for
examination. For a shipment that includes multiple containers, CBP may
select some containers to examine or may examine all the containers in
the shipment. If a targeted shipment is packed into a container with other
shipments, CBP may examine the entire container. 24

ATS performs a risk-based assessment of cargo shipments that CBP
uses to focus its resources for conducting examinations and enhance its
ability to identify potential violations of U.S. law, possible terrorist threats,
and other threats to border security. To do so, ATS incorporates two
types of targeting rules—strategic and tactical—to identify risk factors in
shipment data.

•    Strategic rules: Rules that identify general intelligence or threats or
     that identify relationships among different data elements within a
     single record or across multiple records. The process to update
     strategic rules involves iterations of testing to ensure that rules have
     their intended effect.
•    Tactical rules: Rules that identify risks posed by specific intelligence
     or threats and are typically based on specific entries for one or more
     shipment data elements. Tactical rules can generally be updated in
     time to react to specific intelligence.

ATS has many rules, and one set of rules within ATS is programmed to
check for information or patterns that could be indicative of suspicious or
terrorist activity. This set of rules is referred to collectively as the maritime



23
  Bulk cargo is shipped loose in the hold of a ship, not in packages or containers. For
example, grain, coal, oil, and chemicals are usually bulk cargo. Manifest data for bulk
cargo must be submitted electronically to CBP 24 hours prior to arrival in the United
States, rather than 24 hours prior to cargo being loaded onto U.S.-bound vessels at
foreign ports, as is required for containerized cargo. See 19 C.F.R. § 4.7(b)(4).
24
  Examinations using NII equipment are conducted on containers (i.e., the resulting image
from the exam will depict all contents of the container). If the examination includes a
physical inspection, CBP may focus its efforts on the portion of the container that holds
the targeted shipment.




Page 12                                                     GAO-13-9 Supply Chain Security
national security weight set (the weight set) because each rule in the set
has a specific weight value assigned to it, and for each risk factor that
rules identify, the weight values are added together to calculate an overall
risk score for the shipment. For example, some rules in the weight set
determine if any of the supply chain parties have possible matches to
known terrorists or previous violations of U.S. law, and other rules in the
weight set evaluate the completeness of the data, for example, whether
addresses are provided as required for supply chain parties listed in the
data, such as the importer. The weight set includes both strategic and
tactical rules.

CBP classifies the risk scores from the weight set as low, medium, or high
risk. Shipments with connections to known or suspected terrorists as well
as those that include invalid information are more likely to be classified as
high risk, and shipments from trusted shippers who participate in CBP’s
Customs-Trade Partnership Against Terrorism are more likely to be
classified as low risk.

CBP can make updates to the weight set when the need arises.
According to CBP, some updates, such as responding to specific
intelligence information, can be implemented in a short amount of time.
Substantial updates to the weight set, such as integrating the 10+2 rule
data, can take longer to develop and implement. As part of this process,
CBP may add new rules, change existing rules, or adjust the weights
assigned to rules. To make such substantial updates to the weight set,
CBP uses a multistep process that includes (1) identifying new
requirements, (2) designing alternative versions of the weight set that
may address the new requirements in different ways, (3) testing and
evaluating those alternative versions, (4) selecting and implementing the
new version to replace the existing version of the weight set, and (5)
monitoring the performance of the new version of the weight set after
implementation. According to CBP’s Rules Development Concept of
Operations, which describes the processes for updating the weight set, a
substantial update provides the opportunity for a full assessment and
evaluation of alternatives before CBP implements a new version of the




Page 13                                           GAO-13-9 Supply Chain Security
                             weight set. CBP completed the most recent substantial update to the
                             weight set in early 2011. 25


National Security            Within CBP, targeters are stationed at different locations with varying
Targeting Responsibilities   responsibilities and focuses depending on their location, as described
within CBP                   below.

                             •    Container Security Initiative ports: CBP places targeters at
                                  designated foreign ports to work with their foreign counterparts to
                                  identify shipments that may pose a high risk for containing WMD or
                                  other terrorist contraband before the shipments are loaded onto U.S.-
                                  bound vessels. 26 When CBP targeters at a Container Security
                                  Initiative port determine through advance information that a U.S.-
                                  bound cargo shipment poses a high risk, CBP typically requests that
                                  the host government scan the container(s) with radiation detection
                                  and NII equipment. If these scans indicate the potential presence of
                                  WMD or other contraband, CBP is to request that the host
                                  government conduct a physical inspection of the shipment. If the host
                                  government declines, CBP can either issue a “do not load” order to
                                  prevent the shipment from being loaded onto a U.S.-bound vessel or
                                  flag the shipment for further examination upon arrival at the domestic
                                  port.
                             •    Advance Targeting Units (ATU): ATUs are located at select
                                  domestic ports, and their targeting efforts are focused on shipments
                                  destined for ports within their respective regions. An ATU may be
                                  responsible for targeting shipments arriving at multiple ports in a
                                  region. For example, targeters at the Port of Houston are also
                                  responsible for targeting shipments that are bound for Freeport,
                                  Texas. CBP targeters at ATUs can review data as soon as carriers
                                  and importers submit the required data in accordance with the 24-
                                  hour rule and the 10+2 rule, and the data are available in ATS. After



                             25
                               In February 2009, CBP convened a conference of subject matter experts, including
                             targeters at domestic ports, to discuss integrating the 10+2 rule data into the weight set.
                             Following the conference, CBP designed and evaluated five alternative versions of the
                             weight set. In early 2011, CBP selected one of these alternative versions and
                             implemented it, making it the current version of the weight set that CBP targeters use in
                             their targeting efforts.
                             26
                               The Container Security Initiative has been operational at 58 foreign ports since fiscal
                             year 2007.




                             Page 14                                                     GAO-13-9 Supply Chain Security
                              reviewing the data, if the shipment data indicate a substantial risk that
                              cannot be mitigated through an overseas examination at a Container
                              Security Initiative port or by obtaining additional information about the
                              shipment through additional research, the targeters may seek
                              approval for a “do not load” order from the NTC-C Director before a
                              shipment is loaded onto a U.S.-bound vessel. Once a shipment is
                              loaded onto a vessel, targeters continue to review shipment data in
                              ATS and use other sources, such as public records, to assess
                              whether the shipment could pose a risk, in which case a targeter may
                              target the shipment for examination upon arrival.
                        •     NTC-C: In contrast to targeters at Container Security Initiative ports
                              and ATUs, who focus their reviews on those shipments that are
                              transiting from or to their respective ports, targeters at NTC-C review
                              shipments for security risks from a national perspective. For example,
                              if there is specific intelligence regarding an attempt to smuggle a
                              terrorist weapon in a container, NTC-C targeters can use ATS to
                              identify whether any shipments destined for the United States match
                              the intelligence information, regardless of the port of arrival. NTC-C
                              targeters also serve as a resource for other CBP targeters stationed
                              at foreign and domestic ports because targeters at NTC-C may have
                              access to research tools, such as classified databases, that may not
                              be available to CBP targeters at these other locations.

                        ATS is the primary system that CBP targeters use to review maritime
CBP Uses ATS and        cargo shipments for national security purposes, and targeters we spoke
Other Tools to Target   with were generally satisfied with how ATS and its weight set of national
                        security rules have assisted in their targeting efforts. For example,
Maritime Cargo          targeters at one ATU we visited said that because ATS filters information
Shipments for           and presents key information to the targeters, the targeters are able to
                        better focus their targeting efforts than before they had ATS. 27 Those
National Security       targeters as well as the ATU supervisor at another ATU noted that, in
Purposes                particular, the risk scores that the current version of the weight set
                        produces are helpful in balancing their targeting workload. The risk score,
                        however, is not the sole factor that determines whether a targeter reviews
                        the data for a shipment or whether the shipment is selected for a security
                        examination. In particular, targeters at each of the six ATUs we visited
                        explained that they use the ATS risk score as a starting point for the
                        targeting process, but that their decisions are ultimately based on


                        27
                            CBP introduced ATS in 1999.




                        Page 15                                             GAO-13-9 Supply Chain Security
additional research. To conduct this research, they may use information
within ATS or other tools and information outside of ATS. On the basis of
the ATS risk score and the research conducted, targeters make a
qualitative assessment of the risk and determine whether to hold a
shipment for examination. Targeters at one of the ATUs we visited
emphasized the important role that targeters’ expertise and experience
play in the risk assessment process, stating that although they believe
ATS’s capabilities are helpful, they believe that there could be negative
effects from further automating the targeting process, such as decreased
use of targeters’ expertise regarding the different types of shipments that
arrive at their respective ports. 28

According to CBP policy, the risk scores assigned by the weight set in
ATS determine, in part, what actions CBP officers at the ports are to take
to address potential threats. Targeters at ATUs are required to review
data in ATS for all medium-risk and high-risk shipments that arrive at their
respective ports. For example, a targeter may review individual data
elements, such as the name of the importer or other supply chain parties.
A targeter may also review the rules that detected potential threats and,
therefore, contributed to the calculation of the risk score. 29 ATU targeters
are also required to hold high-risk shipments for examination unless they
can mitigate the risk through additional research or analysis of available
information. 30 CBP targeters at each of the six ATUs we visited
demonstrated how they implement this policy at their respective ports. In
addition to actions targeters take in accordance with CBP policy, targeters
have discretion over which low-risk and medium-risk shipments to select



28
  We have previously reported similar findings related to CBP’s targeting for intellectual
property violations. In particular, we reported in April 2007 that CBP officials at several
ports we visited expressed the view that there is no substitute for the skills and experience
of a well-trained CBP officer; however, other officials noted that automated systems can
assist with targeting because they can better handle data for vast volumes of shipments.
For more information, see GAO, Intellectual Property: Better Data Analysis and Integration
Could Help U.S. Customs and Border Protection Improve Border Enforcement Efforts,
GAO-07-735 (Washington, D.C.: Apr. 26, 2007).
29
  If a rule detects a potential threat based on the data for a shipment, the weighted value
for that rule contributes to the calculation for the shipment’s risk score. If a rule does not
detect a potential threat in the data, the rule does not contribute any points to the risk
score.
30
  Although it is possible to get a waiver to exempt a high-risk shipment from examination
based on information collected, CBP policy states that such waivers should be used
judiciously and only when based upon articulable reasons.




Page 16                                                       GAO-13-9 Supply Chain Security
for security examinations, and CBP expects targeters to select shipments
based on discretionary factors. 31

Targeters use various features within ATS to assist them in their research
into shipments of interest. Officials at each of the six ATUs we visited
discussed or demonstrated the following features and how targeters use
these features when targeting:

•   Queries: Through ATS’s querying capabilities, targeters are able to
    search for shipments that meet specific criteria, such as shipments
    from a particular country. For example, targeters at each ATU we
    visited use queries to identify shipments for review, although the set of
    queries used varied at each ATU. Five of the six ATUs we visited use
    queries to ensure that all shipments, regardless of risk score, are
    reviewed prior to arrival. For example, targeters at one ATU run a
    query for each arriving vessel to ensure that all shipments on the
    vessel have been reviewed. Targeters at another ATU have a query
    for each risk level (high, medium, or low) and targeters reviewing the
    results of each query are to ensure that all shipments of a particular
    risk level have been reviewed. In addition, targeters at five of the six
    ATUs we visited said that they run additional queries of interest for
    discretionary targeting after completing their assigned duties. Such
    discretionary targeting could be for national security purposes or for
    other efforts, such as counternarcotics. For example, targeters at one
    ATU may independently create queries to identify items of interest,
    such as all shipments of a particular commodity or those coming from



31
   In DHS’s Annual Performance Report for fiscal years 2008 through 2010, DHS reported
that CBP did not meet its target for the percentage of maritime cargo containers scanned
for contraband in fiscal year 2008. DHS reported that one of the reasons for not meeting
the target was that updates to ATS targeting rules resulted in fewer mandatory
examinations based on ATS’s assessment of a shipment as high risk, and that CBP did
not compensate for this decrease by increasing the number of discretionary (CBP officer
targeted) exams. DHS reported that CBP planned to increase the number of discretionary
exams. According to a CBP official responsible for tracking this information, CBP
exceeded its target in fiscal years 2009 through 2011. For example, in fiscal year 2011,
CBP scanned 4.13 percent of maritime cargo containers, which exceeded CBP’s goal of
3.80 percent. The containers scanned included those that were targeted based on their
risk score or other risk factors identified by targeters, as well as for other reasons. For
example, CBP officials at all six ATUs we visited said that their ports examine some
containers regardless of the risk assessed by ATS or a targeter (e.g., scan every seventh
container being unloaded from a vessel). CBP also has a Compliance Measurement
Program that supplements ATS by randomly selecting shipments to be inspected to
determine whether the shipments comply with supply chain security and trade laws.




Page 17                                                    GAO-13-9 Supply Chain Security
    a particular country of origin. In addition to queries that targeters at
    the ATUs run, NTC-C targeters run nationwide queries daily to identify
    shipments with the potential for containing chemical, biological,
    radiological, nuclear, or conventional weapons, among other things.
•   Targeters’ notes: ATS has a feature that allows a targeter to
    annotate a shipment with the targeter’s conclusion based on research
    regarding whether the shipment is considered a potential threat. The
    notes feature within ATS facilitates the sharing of research findings
    with CBP targeters at other locations.
•   Targeters’ reviews: ATS also indicates whether the shipment data
    have been reviewed by a targeter at the targeter’s own location or at
    another CBP targeting location, such as a Container Security Initiative
    port or NTC-C.

CBP targeters also use tools outside of ATS to conduct research. During
our interviews at the six domestic port ATUs we visited, targeters
explained that they use web-based and other research tools to aid in their
assessments of shipments. Such tools include web searches, which
targeters use to find general information on a company or address; a
third-party database of public and proprietary records, which targeters
use to research business names and associated information such as a
business’s locations, officers, and assets (e.g., registered vehicles); and
the State Department’s Consular Consolidated Database, which targeters
may use to obtain visa and passport information for foreign individuals
involved in a shipment. Targeters review and analyze all of the
information collected to make a decision as to whether a shipment should
be examined. On the basis of such research and analysis, a targeter
could select a low-risk shipment for examination. A targeter could also
determine that an examination is not necessary for a medium-risk
shipment—for example, the weight set may assign a medium-risk score
to a shipment based on the data available, but the targeter could
determine through research that the score is based on a clerical error in
the data provided.

Targeters’ experience may also inform targeting decisions. For example,
targeters at ATUs may have information about recent seizures and can
look for recurring patterns to identify future shipments that may be part of
a trend of illegal shipments. Targeters may also share such information
with other targeting units to help inform targeting decisions. Also,
targeters at all six of the ATUs we visited said they communicate regularly
with targeters at NTC-C regarding shipments of interest.




Page 18                                          GAO-13-9 Supply Chain Security
CBP’s Efforts to
Assess the Weight Set
Have Been Limited
CBP Developed               We have previously reported that ensuring controls to assess ATS’s
Performance Measures to     effectiveness in identifying high-risk shipments was important for
Assess the Weight Set but   providing CBP with the best information to inform its targeting efforts. 32 In
                            2005, in response to our work and an external peer review 33 of ATS
Is Continuing to Update
                            conducted in 2005, CBP contracted with a consulting firm to develop
the Methodology             performance measures and a methodology to determine the effectiveness
                            of the weight set in identifying high-risk shipments. The resulting
                            performance measures and methodology, which the consulting firm
                            provided to CBP in April 2006, balanced targeting accuracy with
                            examination workload and enabled CBP to compare the weight set’s
                            performance with the effectiveness of examinations conducted through a
                            random selection program. We then reported in August 2006 that the
                            performance measures and methodology developed by the consulting
                            firm were sufficient to assess the performance of the weight set and
                            provide a baseline against which future assessments may be
                            conducted; 34 however, we also reported that data limitations and
                            uncertainties existed, and we noted that CBP must interpret the
                            evaluations cautiously. 35 We also reported in August 2006 that CBP



                            32
                              Because the report we issued in August 2006 regarding CBP’s targeting practices
                            contains sensitive information, it is not publicly available.
                            33
                              An external peer review is a process that includes an assessment of the model by
                            independent and qualified external experts.
                            34
                              In 2006, CBP referred to its two performance measures as the estimated accuracy rate
                            and the estimated inspection rate, respectively. In more recent assessments, it refers to
                            these measures as the true positive rate and the false positive rate, respectively. In 2006,
                            CBP also calculated the relative performance factor, which provides an assessment of
                            how much better or worse ATS performs at targeting oceangoing cargo shipped in
                            containers than a random selection method.
                            35
                              To determine which shipments to use for the calculations, the consulting firm developed
                            a proxy positive definition of shipments that contain indications of materials or behaviors
                            that are believed to be similar in physical attributes, magnitude, and intent to the materials
                            and behavior of shipments associated with terrorist threats. The firm had to develop this
                            definition because CBP inspections of cargo shipped in containers at that time had not
                            resulted in the identification of any direct terrorism threats. Because CBP had not
                            identified any terrorist-related materials in shipments at the time this report was written,
                            there was no way to validate the proxy positive shipments.




                            Page 19                                                      GAO-13-9 Supply Chain Security
planned to continue using the consulting firm’s methodology in making
future adjustments to the weight set. For more information about GAO’s
past audit findings and recommendations to improve the targeting
process, see appendix II.

Currently, CBP assesses the performance of the weight set using the
following performance measures:

•   True positive rate (TPR) which reflects the percentage of maritime
    shipments that ATS assessed as high risk within the population of
    shipments in which CBP identified a threat during an examination.
•   False positive rate (FPR) which reflects the percentage of maritime
    shipments that ATS assessed as high risk within the population of
    shipments in which CBP did not identify a threat during an
    examination.
Taken in combination, the TPR and FPR measures enable CBP to
determine the effectiveness of the weight set by providing information
about the accuracy of the weight set and its impact on examination
workload. The TPR enables CBP to determine the accuracy of the weight
set in identifying high-risk shipments. The TPR and FPR measures also
enable CBP to determine the workload or examination rate for ports
based on the results of the weight set scores. For example, a high FPR
would unnecessarily increase the workload or number of examinations at
ports because officials would be required to examine a higher number of
shipments that do not contain an actual threat. Using data from the
version of the weight set CBP was using in 2005, the consulting firm used
its methodology to conduct a performance assessment, which involved
calculating the TPR and FPR to ultimately indicate the effectiveness of
the weight set at that time. Since the contract with the firm ended in 2011,
CBP has taken on the role of assessing the effectiveness of the weight
set. In its project plan for the most recent update to the weight set
implemented in early 2011, as it had previously done, CBP planned to
assess the performance using these measures to compare the targeting
effectiveness of the weight set with other measurements.

CBP officials stated that they face ongoing challenges with the
performance measures and the methodology by which they are
calculated. In particular, the FPR and TPR may not accurately reflect the
weight set’s performance in identifying national security threats because
they rely on indirect measures given that no true security threat has been
found in a cargo container. We reported in 2006 that CBP planned to take
steps to improve the process for assessing ATS performance. Since the



Page 20                                          GAO-13-9 Supply Chain Security
                              most recent update to the weight set in early 2011, CBP formed working
                              groups and has begun taking steps to ensure that the methodology it
                              uses to approximate threats accurately reflects what CBP considers to be
                              a national security threat (to the extent possible). For example, these
                              working groups plan to (1) create a new definition of “national security” for
                              the purposes of clarifying what the weight set should target and (2) revisit
                              the current definitions of threats in containers to provide consistency with
                              the new definition of national security. CBP officials stated that they
                              expect the working groups’ activities to be completed by April 2013.


CBP Does Not Have             Prior to implementing the current version of the weight set in early 2011,
Reasonable Assurance          CBP did not conduct an assessment to determine whether the updated
That the Updated Weight       version of the weight set would be more effective than the previous
                              version of the weight set or other alternatives that were considered during
Set Is More Effective than    the update process. For the 2011 update, CBP developed and evaluated
Alternative Versions or the   five alternative versions of the weight set. CBP’s consideration for which
Version It Replaced           alternative to select focused on two of the five versions because,
                              according to CBP officials, these two versions incorporated the newly
                              required 10+2 data and reflected current threat information about
                              countries of interest. CBP ultimately implemented one of these two
                              versions of the weight set, but CBP could not provide any documentation
                              to demonstrate that the version selected was more effective than either
                              the other alternative or the version it was to replace.

                              CBP’s process for updating the weight set involves assessing the impact
                              of alternative versions of the weight set. For example, for the most recent
                              update to the weight set, CBP’s impact assessment provides information
                              on how many shipments would be assessed as high risk under each
                              alternative version of the weight set and would, therefore, affect CBP
                              targeters’ workload at ports of arrival because such high-risk shipments,
                              under CBP policy, are to be held for examination, for example, through
                              the use of NII equipment. CBP officials stated that they believed the
                              impact assessment that CBP conducted during the update process
                              indicates the reasons for selecting the chosen version of the weight set.

                              While, according to CBP officials, the impact assessment provides CBP’s
                              reason for replacing the prior version of the weight set, we found that the
                              impact assessment primarily evaluates how the chosen alternative
                              version of the weight set could affect targeter workload and does not
                              address measures of accuracy in identifying high-risk shipments.
                              Therefore, the impact assessment does not fully account for the
                              effectiveness of each alternative of the weight set. Although managing


                              Page 21                                           GAO-13-9 Supply Chain Security
resources is an element of risk management, effectiveness in reducing
risks is also an important consideration when evaluating alternatives to
manage risk. CBP’s impact assessment does not address the balance
between targeting accuracy and workload.

Assessing the potential effectiveness of alternative versions of the weight
set prior to selecting one for implementation would provide CBP with
more information to make an informed decision. In January 2010, the
DHS Office of Inspector General recommended that CBP enhance its
documentation efforts to ensure that each stage of the process for
analyzing and developing ATS rules is documented, and CBP concurred
with this recommendation. 36 As part of this recommendation, the DHS
Office of Inspector General recommended documenting the rationale for
making changes to ATS rules but did not specify what types of analyses
could demonstrate or support the rationale for making changes to the
rules. On the basis of our analysis, the rationale for updates to the weight
set could be further strengthened through assessments of effectiveness
beyond workload. For example, determining the expected TPR and FPR
for an alternative version of the weight set and comparing these
measures against the TPR and FPR for the existing version of the weight
set could enable CBP to determine if the alternative version of the weight
set could be expected to result in improved effectiveness, based on these
performance measures. This would enable CBP to quantitatively compare
the effectiveness of the alternative versions of the weight set being
considered prior to selecting one for implementation. Doing so, in addition
to the impact assessment, would provide CBP with reasonable assurance
that the version of the weight set it selects for implementation is the most
effective of the alternatives considered after taking into account any
resource constraints. Furthermore, assessing the alternative versions of
the weight set in the future would provide CBP with better assurance that
the version it selects for implementation is more effective than the
previous version of the weight set. CBP officials stated that they plan to
calculate and document measures of effectiveness during the planned
update to the weight set that will begin in the fall of 2012.




36
  DHS Office of Inspector General, Cargo Targeting and Examinations, OIG-10-34
(Washington, D.C.: Jan. 6, 2010).




Page 22                                                GAO-13-9 Supply Chain Security
CBP Has Not Regularly       Since implementing the current version of the weight set in early 2011,
Assessed the Weight Set     CBP has not regularly assessed the weight set against established
against Performance         performance targets to monitor its performance and obtain information to
                            determine when updates to the weight set are necessary. We reported in
Targets to Determine when   August 2006 that CBP intended to establish targets for the performance
Updates Are Needed          measures to assess future performance of ATS, but CBP did not
                            establish such targets for those measures.

                            Targets could help CBP determine when updates are needed to improve
                            targeting effectiveness. For example, according to CBP’s analysis, the
                            TPR for summer 2011 through spring 2012 shows that, among shipments
                            CBP found to contain a potential threat during an examination, the weight
                            set accurately identified 6.3 percent as high risk, meaning that the weight
                            set classified 93.7 percent of shipments that carried a potential threat as
                            either medium risk or low risk. Furthermore, the FPR for that time period
                            shows that, among all the arriving shipments that CBP examined during
                            that time that did not pose a threat, the weight set identified 3.6 percent
                            as high risk. However, because CBP did not establish targets for either
                            TPR or FPR, it is not clear whether 6.3 percent for the TPR is sufficiently
                            low or 3.6 percent for the FPR is sufficiently high to suggest that changes
                            are needed to improve the performance of the weight set.

                            CBP’s project plan calls for conducting periodic performance
                            assessments by determining recurring measures of TPR and FPR.
                            Furthermore, according to CBP officials, the performance assessments
                            are to be conducted as part of quarterly reporting responsibilities.
                            However, CBP did not calculate these measures at the end of each
                            quarter, but instead calculated them as part of a single assessment in the
                            summer of 2012 and divided the results into quarters. 37 Accordingly, CBP
                            was not aware of the ongoing performance of the weight set from its
                            implementation in early 2011 through spring 2012, and CBP was



                            37
                               We compared the results of assessments for the previous version of the weight set with
                            the results of assessments for the current version of the weight set. For the previous
                            version of the weight set, the TPR was about 20 percent from the end of fiscal year 2009
                            through fiscal year 2010, whereas the TPR for the current weight set has generally been
                            below 10 percent from the third quarter of fiscal year 2011 through the third quarter of
                            fiscal year 2012. According to CBP officials, the measures for each quarter are calculated
                            using data from the prior 1-year period. However, according to CBP officials, the
                            methodologies used to assess each weight set were different, and because it is not clear
                            to what extent this change in methodology accounts for any changes in the performance
                            measures, it is not possible to make a meaningful comparison between the results.




                            Page 23                                                    GAO-13-9 Supply Chain Security
therefore unable to determine for 18 months whether the weight set was
performing at a level that could require changes or updates to improve its
effectiveness. According to CBP officials, the summer 2012 assessment
was conducted at that time in preparation for a conference to discuss
updates to the weight set planned for the fall of 2012. CBP had decided to
hold this conference before CBP conducted the assessment, meaning its
decision for when to update the weight set was not based on information
about the weight set’s effectiveness from ongoing monitoring of CBP’s
performance measures for the weight set.

Ongoing monitoring is a key element of a risk management approach,
and CBP’s project plan calls for such periodic performance monitoring to
determine targeting effectiveness. In addition, standard practices for
internal control indicate that (1) ongoing monitoring should occur in the
course of normal operations and can be accomplished by periodic review
of performance measures and (2) in the process of ongoing performance
monitoring, actions should include continuous comparison of performance
data against planned targets and analysis of any differences to take
corrective actions as necessary.

CBP officials stated that personnel have not been consistently available
to conduct performance assessments since the initial contract with the
consulting firm ended in July 2011 and that resource concerns, such as
funding, the availability of subject matter experts, and the availability of
programmers may affect the timing of weight set updates. Nevertheless,
given the importance of the weight set to CBP’s process for targeting
cargo containers, regular performance assessments of the weight set that
include evaluating results against established performance targets could
help CBP determine when updates are needed in a timelier manner and




Page 24                                          GAO-13-9 Supply Chain Security
              help it better prioritize the resources it needs to complete the updates. 38
              Furthermore, CBP officials stated that they intend to continue adjusting
              the methodology for calculating the performance measures to mitigate
              data limitations and more accurately reflect the performance of the weight
              set. Such steps could help CBP ensure that its targeters have the best
              information available regarding the risk of maritime cargo container
              shipments arriving in the United States.


              CBP recognizes the importance of and challenges to ensuring the
Conclusions   security of the global supply chain while facilitating the flow of legitimate
              commerce. Although no events have occurred to date, terror-related
              attacks on the supply chain could have devastating effects on the nation’s
              security and economic well-being, and it is imperative that CBP use the
              best information and tools available to continually mitigate potential
              threats and address vulnerabilities. DHS and CBP face difficulties in
              achieving 100 percent scanning of cargo containers prior to loading at
              foreign ports and have, instead, advocated a risk-based approach to
              target and scan those cargo containers that pose the highest risk. Given
              the critical role that ATS plays as part of this risk-based approach, it is
              important to ensure that ATS is performing effectively.

              CBP plans to continue enhancing risk management processes, including
              the use of ATS and its associated targeting rules. CBP’s determination of
              which containerized shipments to review or to hold for examination is
              based, in part, on the risk score. Thus, updating the weight set in ATS
              that calculates this risk score is important for ensuring that targeters are



              38
                In the past, CBP’s decision to update the performance of the weight set was dependent
              upon feedback from targeters or external reasons—such as the receipt of new data
              because of the 10+2 rule. However, feedback from targeters is not systematic and may
              not provide a reliable assessment of the weight set’s ability to identify high-risk containers.
              CBP utilizes ATU targeters at rules conferences to assist in rule and weight set updates
              because they use ATS for daily operations. CBP receives weight set input from ATU
              targeters from a variety of ports nationally. CBP headquarters officials consider all input
              and determine the best way to use the information to indicate risk. In some cases,
              changes are made to rules that will affect targeting nationwide, whereas in other cases,
              rule changes may not be implemented because they might have a negative effect across
              all ports. Further, in some cases, rules that affect only a single port or region may be
              added to a weight set. According to CBP officials, they do not rely on a particular amount
              of input to initiate the process of updating the weight set; rather, they subjectively
              determine the point at which the feedback is substantial enough to warrant an evaluation
              of the weight set.




              Page 25                                                       GAO-13-9 Supply Chain Security
                      using the most effective tools in making targeting decisions. CBP has
                      assessed workload impacts when making updates to the weight set, but it
                      did not fully assess the weight set’s effectiveness as part of the most
                      recent update. As a result, CBP does not have reasonable assurance that
                      the implemented version is the most effective. Further, CBP did not
                      conduct periodic assessments as part of ongoing monitoring efforts.
                      Specifically, CBP did not conduct an assessment of the weight set until
                      18 months after CBP implemented the new weight set. We believe it is
                      important that CBP more regularly assess the performance of the weight
                      set in ATS that produces the risk scores and compare the results of this
                      assessment against established performance targets. Such steps could
                      help CBP determine when changes may be needed and ensure that its
                      targeters have the best information available regarding the risk of
                      maritime cargo container shipments arriving in the United States.


                      To enhance its targeting of maritime cargo containers and better position
Recommendations for   CBP to provide reasonable assurance of the effectiveness of ATS, we
Executive Action      recommend that the Commissioner of CBP take the following two actions:

                      •   ensure that future updates to the weight set are based on results of
                          assessments that demonstrate that the chosen version of the weight
                          set is more effective than other alternatives, including the existing
                          version, and
                      •   establish targets for CBP’s performance measures and use those
                          measures to assess the effectiveness of the weight set on a regular
                          basis to better determine when updates to the weight set are needed.

                      On October 17, 2012, DHS provided written comments on a draft of this
Agency Comments       report, which are reprinted in appendix III. DHS concurred with the two
                      recommendations. Specifically, DHS concurred with the recommendation
                      to ensure that future updates to the weight set are based on the results of
                      assessments and stated that CBP plans to conduct analyses to ensure
                      that future versions of the weight set result in increased effectiveness.
                      DHS also noted that CBP is to conduct these analyses during the
                      development and deployment of future versions of the weight set.
                      According to DHS, these analyses would include performance measures,
                      subject matter expert input, current threat information, and other
                      intelligence. DHS stated that it expects these actions to be completed by
                      April 2013. Such actions should address the intent of the
                      recommendation to ensure improvements in the effectiveness of future
                      versions of the weight set. DHS also concurred with the recommendation



                      Page 26                                          GAO-13-9 Supply Chain Security
to establish targets for CBP’s performance measures and stated that CBP
is working to improve the current performance measures methodology.
DHS stated that, following approval of this methodology, CBP plans to
conduct quarterly reviews of the weight set to inform decision making.
DHS stated that it expects these actions to be completed by September
2013. If CBP takes these steps as planned and includes targets for any
performance measures that are part of the updated methodology, this
should address the intent of our recommendation. DHS also provided
technical comments, which we incorporated as appropriate.


As agreed with your office, unless you publicly announce its contents
earlier, we plan no further distribution of this report until 30 days after its
issue date. At that time, we will send copies of this report to the Secretary
of Homeland Security, appropriate congressional committees, and other
interested parties. In addition, the report will be available at no charge on
the GAO Web site at http://www.gao.gov.

If you or your staffs have any questions on this report, please contact me
at (202) 512-9610 or caldwells@gao.gov. Contact points for our Offices of
Congressional Relations and Public Affairs may be found on the last page
of this report. Staff acknowledgments are provided in appendix IV.




Stephen L. Caldwell
Director
Homeland Security and Justice




Page 27                                            GAO-13-9 Supply Chain Security
Appendix I: Federal Strategy for Ensuring the
                                             Appendix I: Federal Strategy for Ensuring the
                                             Security of Maritime Cargo Container
                                             Shipments


Security of Maritime Cargo Container
Shipments
                                             This appendix describes the core programs related to U.S. Customs and
                                             Border Protection’s (CBP) strategy for ensuring the security of maritime
                                             cargo container shipments. CBP has developed this strategy to mitigate
                                             the risk of weapons of mass destruction, terrorist-related material, or
                                             other contraband being smuggled into the United States in cargo
                                             containers. CBP’s strategy is based on related programs that attempt to
                                             focus resources on potentially risky cargo shipped in containers while
                                             allowing other cargo containers to proceed without unduly disrupting
                                             commerce into the United States. The strategy includes obtaining
                                             advanced cargo information to identify high-risk containers, using
                                             technology to inspect cargo containers, and partnering with foreign
                                             governments and the trade industry. Table 1 provides a brief description
                                             of the core programs that compose this security strategy.


Table 1: Description of CBP’s Core Cargo Security Programs

Program and year introduced                       Description
Obtaining advanced information to identify high-risk containers
Automated Targeting System (ATS), 1999             ATS is an intranet-based enforcement and decision support system that compares
                                                   traveler, cargo, and conveyance information against intelligence and other
                                                   enforcement data by incorporating risk-based targeting scenarios and
                                                   assessments. ATS assigns a risk score to arriving cargo shipments based on
                                                   shipping information to help CBP identify and prevent potential terrorists and
                                                   terrorist weapons from entering the United States.
24-hour rule, 2002                                 CBP generally requires vessel carriers to electronically transmit cargo manifests to
                                                   CBP’s Automated Manifest System 24 hours before U.S.-bound cargo is loaded
                                                   onto a vessel at a foreign port. The information is used by ATS in its calculation of
                                                   risk scores. The cargo manifest information is submitted by vessel carriers for all
                                                   arriving cargo shipments.
Importer Security Filing and Additional Carrier    CBP requires importers and vessel carriers to provide data elements for improved
Requirements (also known as 10+2), 2009            identification of containerized shipments that may pose a risk for terrorism. The
                                                   importer is responsible for supplying CBP with 10 shipping data elements, such as
                                                   country of origin, 24 hours prior to loading, while the vessel carrier is required to
                                                   provide two data elements, container status messages and stow plans, not
                                                   required by the 24-hour rule.a
Domestic scanning technology deployments
Nonintrusive inspection (NII) equipment, 2001      CBP uses NII equipment to actively scan both randomly selected containers and
                                                   those identified by ATS as high risk. NII uses X-rays or gamma rays to scan a
                                                   container and create images of the container’s contents without opening it.
                                                   According to CBP, as of August 2012, it had deployed 93 NII systems to U.S. ports
                                                   to scan containers. In fiscal year 2011, CBP scanned 4.13 percent of containers
                                                   arriving at U.S. ports.
Radiation portal monitors, 2002                    CBP’s program to scan 100 percent of containers arriving in the United States with
                                                   radiation detection equipment prior to leaving a domestic port. As of July 2012,
                                                   DHS had deployed 447 radiation portal monitors at U.S. seaports, through which
                                                   approximately 99 percent of all containers arriving by sea pass.




                                             Page 28                                                     GAO-13-9 Supply Chain Security
                                           Appendix I: Federal Strategy for Ensuring the
                                           Security of Maritime Cargo Container
                                           Shipments




Program and year introduced                      Description
Partnerships with foreign governments and the trade industry
Container Security Initiative, 2002             CBP places staff at participating foreign ports to work with host country customs
                                                officials to target and examine high-risk container cargo for weapons of mass
                                                destruction before they are shipped to the United States. CBP officials identify the
                                                containers that may pose a risk for terrorism and request that their foreign
                                                counterparts examine the contents of the containers.
Secure Freight Initiative, 2006                 CBP and the Department of Energy initiated this program at selected ports to scan
                                                100 percent of U.S.-bound container cargo for nuclear and radiological materials
                                                overseas using integrated examination systems that couple NII and radiation
                                                detection equipment. Since its inception, all but one of the selected ports has
                                                reverted to Container Security Initiative operations.
Customs-Trade Partnership Against Terrorism, CBP develops voluntary partnerships with members of the international trade
2001                                         community composed of importers; manufacturers; customs brokers; forwarders; air,
                                             sea, and land carriers; and contract logistics providers. Private companies that
                                             implement specific security measures and best practices receive facilitated
                                             processing, such as a reduced likelihood of security-based examinations of their
                                             cargo.
Mutual recognition arrangements, various        Through mutual recognition arrangements with other countries, the security-related
      b
years                                           practices and programs taken by the customs administration of one country are
                                                recognized and accepted by the administration of another. According to CBP, the
                                                essential concept is that the Customs-Trade Partnership Against Terrorism and the
                                                foreign programs are compatible in both theory and practice so that one program
                                                may recognize the findings and validation results provided by another.
                                           Source: GAO summary of information provided by DHS.
                                           a
                                            Container Status Messages report terminal container movements, such as loading and discharging
                                           the vessel, and report the change in the status of containers, such as if they are empty or full. The
                                           stow plan contains the position of each cargo container on a vessel.
                                           b
                                            These mutual recognition arrangements were individually negotiated and signed in different years.
                                           CBP has signed arrangements with New Zealand (2007), Canada (2008), Jordan (2008), Japan
                                           (2009), Korea (2010), and the European Union (2012).




                                           Page 29                                                            GAO-13-9 Supply Chain Security
Appendix II: Past Audit Findings and
                          Appendix II: Past Audit Findings and
                          Recommendations to Improve the Targeting
                          Process


Recommendations to Improve the Targeting
Process
                          Since 2004, we have conducted audits of CBP’s targeting process and
                          ATS. In particular, we published reports related to these topics in
                          February 2004, August 2006, and September 2010. 1 In addition, the
                          Coast Guard and Maritime Transportation Act of 2004 requires the
                          Department of Homeland Security’s (DHS) Office of Inspector General
                          (OIG) to report annually on its evaluation of the current targeting system
                          for international intermodal cargo containers. 2 Our and DHS OIG’s audits
                          have addressed, among other things, incorporating key elements of a risk
                          management framework and recognized modeling practices,
                          documenting the targeting rule development process, improving data
                          collection and ATS’s use of data, and providing additional information
                          outside of ATS to targeters. Collectively, these audits have made
                          recommendations to DHS and CBP for improving the targeting process,
                          and CBP has taken actions to implement them, although in some cases
                          CBP’s implementation efforts have been slow, leaving CBP without the
                          benefits of these improvements for several years. This appendix provides
                          an overview of key findings and recommendations from these audits, as
                          well as the status of actions taken to implement the recommendations. 3


Incorporating Key         In February 2004, we reported that while CBP had taken steps to address
Elements of a Risk        the terrorism risks posed by maritime cargo containers, its targeting
Management Framework      strategy did not incorporate all key elements of a risk management
                          framework, and ATS was not consistent with certain recognized modeling
and Recognized Modeling   practices. We recommended, among other things, that CBP improve the
Practices                 targeting strategy by incorporating key elements of a risk management
                          framework and recognized modeling practices. These recommendations
                          included specific steps CBP should take to help ensure it could achieve
                          the objectives of its overall targeting strategy and better ensure that the
                          tools it uses to protect against terrorism are working effectively at the


                          1
                           GAO, Supply Chain Security: CBP Has Made Progress in Assisting the Trade Industry in
                          Implementing the New Importer Security Filing Requirements, but Some Challenges
                          Remain, GAO-10-841 (Washington, D.C.: Sept. 10, 2010). Because the reports we issued
                          in February 2004 and August 2006 regarding CBP’s targeting practices contain sensitive
                          information, they are not publicly available.
                          2
                           Pub. L. No. 108-293, § 809(g), 118 Stat. 1028, 1087.
                          3
                           Some recommendations from these products addressed other elements of CBP’s cargo
                          container security strategy, such as policies and procedures for examining cargo
                          containers. The scope of our review focused on ATS and CBP’s targeting efforts, and
                          therefore the recommendations listed in this appendix are those that fall within that scope.




                          Page 30                                                     GAO-13-9 Supply Chain Security
                                               Appendix II: Past Audit Findings and
                                               Recommendations to Improve the Targeting
                                               Process




                                               nation’s ports. We later reported, in August 2006, that CBP had made
                                               progress addressing the recommendations but had not yet fully
                                               implemented the recommendations in our February 2004 report. 4 Since
                                               that time, CBP has fully implemented the February 2004
                                               recommendations aimed at improving the targeting strategy by
                                               incorporating key elements of a risk management framework and
                                               recognized modeling practices. Table 2 provides more detail on our
                                               February 2004 recommendations and CBP’s efforts to implement them.

Table 2: Status of GAO Recommendations from February 2004

Recommended steps for incorporating
key elements of a risk management
framework and recognized modeling
practices                                      Status of the recommendation
Conduct and use a comprehensive set of         Implemented
threat, criticality, vulnerability, and risk   In August 2006, we reported that CBP was conducting risk assessments based on threat
assessments related to maritime cargo          and intelligence information it was receiving. Although CBP was preparing threat
containers to determine the need for risk      assessments, it did not have a documented methodology in place to guide its personnel
mitigation actions                             in identifying sources of threat information, what approach to take in conducting the
                                               assessment, and key elements to include that could help ensure consistency in the
                                               preparation of such assessments. As a result, we made an additional recommendation in
                                               August 2006 related to establishing and documenting a methodology for conducting
                                               threat assessments, which CBP subsequently implemented in October 2006.
Initiate an external peer review of ATS to     Implemented
evaluate, among other things, the types        In response to this recommendation, CBP contracted with a consulting firm in 2005 and
and sources of data being used and the         2006 to conduct a peer review and to develop performance measures, obtain additional
appropriateness and weighting of targeting     insights into ATS’s performance, and determine whether ATS is more effective at
rules                                          targeting shipments than a random sampling approach.




                                               4
                                                In November 2006, the DHS OIG also reported CBP was in the process of developing
                                               national ATS performance measures in response to our February 2004 recommendations.
                                               The DHS OIG did not issue any related recommendations in that report. That DHS OIG
                                               report contains sensitive information and is therefore not publicly available.




                                               Page 31                                                 GAO-13-9 Supply Chain Security
                                            Appendix II: Past Audit Findings and
                                            Recommendations to Improve the Targeting
                                            Process




Recommended steps for incorporating
key elements of a risk management
framework and recognized modeling
practices                                       Status of the recommendation
Implement a mandatory random sampling           Implemented
program that cannot be waived, which            CBP’s Compliance Measurement Program randomly selects shipments based on
would allow CBP to better compare the           customs entry information submitted by the trade industry. We reviewed CBP’s sampling
inspection results from its random sample       methodology and determined that its sampling techniques support a statistically valid
program with those of ATS, and which may        random probability sample. In August 2006 we reported that CBP had made progress in
result in improvements to the targeting         conducting random sampling, but that at the time, CBP was unable to compare the
rules.a                                         examination results from its Compliance Measurement Program with ATS inspection
                                                results because CBP did not yet have an automated system in place to compare multiple
                                                sets of data—like results of random examinations with results of routine ATS inspections.
                                                Subsequently, in March 2008, CBP implemented the Cargo Enforcement Reporting and
                                                Tracking System (CERTS) within ATS to capture examination findings.b As a result of its
                                                actions, CBP is better positioned to systematically analyze inspection results for
                                                adjusting ATS.
Conduct simulated events to operationally       Implemented
test and validate the targeting strategy        We reported in August 2006 that CBP had not yet conducted simulated events such as
                                                covert tests and computer-generated simulations to test and validate the effectiveness of
                                                ATS in targeting oceangoing cargo shipped in containers that pose the highest risk of
                                                having links to terrorism. CBP has since taken actions to implement this
                                                recommendation by, among other things, developing and implementing a testing and
                                                simulation environment to conduct computer-generated tests of ATS. In June 2007, CBP
                                                reported to us that as of May 2007, it had tested approximately 26,000 mock cargo
                                                shipments and modified or updated information contained in ATS.
Monitor and evaluate the effectiveness of       Implemented
risk mitigation actions taken so that the       CBP contracted a consulting firm to develop performance measures, and the consulting
targeting strategy may be amended as            firm produced a report in April 2006 that outlined performance measures and a
needed and responds to changes in risk          methodology for assessing the effectiveness of ATS. CBP has continued to use these
                                                performance measures, but as we have stated in this report, CBP’s assessments of
                                                ATS’s effectiveness have not occurred on a regular basis.
                                            Source: GAO.
                                            a
                                             In addition to this recommendation, the DHS OIG made a recommendation in July 2005 that CBP
                                            use examination results to refine ATS targeting rules. See DHS OIG, Audit of Targeting Oceangoing
                                            Cargo Containers (Unclassified Summary), OIG-05-26 (Washington, D.C.: July 2005).
                                            b
                                             The DHS OIG recommended in August 2007 that CBP develop systematic procedures to extract
                                            oceangoing container examination results information and begin using it to refine existing targeting
                                            rules and developing new rules. DHS OIG stated in its report that the intention of its recommendation
                                            was satisfied as CBP had plans for and had begun implementing CERTS. This August 2007 DHS
                                            OIG report contains sensitive information and is therefore not publicly available. The DHS OIG
                                            subsequently issued a report in June 2008 with recommendations related to the management and
                                            oversight of the development and implementation of CERTS. See DHS OIG, Targeting of Cargo
                                            Containers 2008: Review of CBP’s Cargo Enforcement Reporting and Tracking System, OIG-08-65
                                            (Washington, D.C.: Jun. 11, 2008). As noted above in the table, CBP fully implemented CERTS in
                                            May 2008.


                                            In August 2006, among other things, we reported on the status of
                                            recommendations issued in our February 2004 report and reiterated the
                                            importance of the recommendations. Additionally, with respect to the
                                            recommendation regarding risk assessments, at the time, CBP had



                                            Page 32                                                           GAO-13-9 Supply Chain Security
Appendix II: Past Audit Findings and
Recommendations to Improve the Targeting
Process




begun taking action to implement the recommendation by conducting and
using risk assessments that incorporated discussions of potential threats
and estimates of the relative importance of assets and vulnerabilities
associated with the supply chain. We noted, though, that CBP did not
have a methodology in place to guide its staff in identifying sources of
threat information, such as agencies to contact, what approach to take in
conducting the assessment, and key elements to include that would help
ensure consistency in the preparation of threat assessments associated
with the movement of cargo shipped in containers. As a result, we further
recommended that CBP establish and document a methodology for
conducting threat assessments associated with cargo shipped in
containers to help ensure that CBP staff responsible for conducting threat
assessments consult relevant information sources, prepare threat
assessments consistently, and include key elements to effectively
communicate risk to program managers.

In response to that recommendation, in October 2006, CBP issued a
protocol to assist its intelligence research specialists in preparing port
threat assessments in support of the Container Security Initiative. Under
this initiative, CBP places staff at foreign seaports to work with foreign
counterparts to inspect high-risk containers before they are shipped to the
United States. The October 2006 protocol included a discussion of
information sources for CBP staff to consult and the overall methodology
to follow in making the port threat assessment. CBP also distributed a
template that discussed the key elements its specialists should include
when preparing port threat assessments. In addition, CBP developed a
checklist for its specialists to use to help ensure that appropriate
information sources are consulted in making port threat assessments. To
assist in addressing threats related to global supply chain logistics and
the movement of maritime containers carrying cargo arriving in the United
States, CBP established an Office of Intelligence and Operations
Coordination in October 2007, which has since been renamed the Office
of Intelligence and Investigative Liaison. This office includes the Analysis
and Targeting Division (A&T), which is composed of program managers
with operational experience and intelligence analysts and is responsible
for conducting risk assessments of countries (known as country risk
profiles) that consider threats, vulnerabilities, and the associated criticality
of related assets from which or through which cargo is shipped to the
United States. In 2008, the A&T Division assessed risks in two primary
mission areas: (1) terrorism and weapons of mass destruction and (2)
narcotics. On the basis of these assessments, the A&T Division
developed a risk assessment methodology to rank countries according to
the level of risk associated with each mission area. The A&T Division


Page 33                                             GAO-13-9 Supply Chain Security
                            Appendix II: Past Audit Findings and
                            Recommendations to Improve the Targeting
                            Process




                            teams translated these rankings into scores, and program managers
                            integrated them into ATS. Specifically, CBP integrated the rankings for
                            the terrorism mission into ATS in 2008 and for the narcotics mission in
                            2009. Thus, CBP has fully addressed this recommendation and is better
                            positioned to ensure consistency in the preparation of risk assessments
                            associated with the movement of maritime cargo container shipments.


Documenting the Targeting   In January 2010, the DHS OIG reported on several aspects of CBP’s
Rule Development Process    process for developing and updating targeting rules in ATS. 5 In this
                            report, the DHS OIG stated that CBP could improve its process for
                            changing or deleting targeting rules by, among other things, documenting
                            (1) rule change decisions and (2) the testing and evaluation of rule
                            changes. Specifically, one component of the rule update process involves
                            the review of the proposed rule changes by subject matter experts, and
                            the DHS OIG reported that CBP could improve the process by ensuring
                            the rationale for changes implemented or not implemented are
                            documented and recorded for future use. Furthermore, the DHS OIG
                            reported that CBP tested new rules using actual data to determine how
                            well the new rules are working, but the DHS OIG noted that this process
                            for testing and evaluating the rules, and subsequent modifications of the
                            new rules, was not documented. The DHS OIG recommended that CBP
                            ensure it documents each stage of the process for analyzing and
                            developing ATS rules, including the rationale for making changes and the
                            details on tools used to improve application consistency and rule change
                            standardization. According to the DHS OIG report, in response to this
                            recommendation, CBP (1) developed a documentation process to capture
                            and record information that includes the rationale for rule changes and
                            the utilization of tools and (2) introduced more formality into the rules
                            process by implementing a structure to guide national conferences, rule
                            evaluation, targeting development, and process management, among
                            other things. The DHS OIG stated in its report that it considered the
                            actions taken by CBP to be responsive to the recommendation.

Improving Data Collection   In February 2004, in addition to making the recommendations discussed
and ATS’s Use of Data       earlier, we also reported that CBP was relying on the manifest as its
                            principal data input, and CBP did not mandate the transmission of
                            additional information before a cargo’s risk level was assigned. We



                            5
                            DHS OIG, Cargo Targeting and Examinations, OIG-10-34.




                            Page 34                                             GAO-13-9 Supply Chain Security
Appendix II: Past Audit Findings and
Recommendations to Improve the Targeting
Process




reported that terrorism experts, members of the international trade
community, and CBP inspectors at the ports we visited as part of that
review characterized the ship’s manifest as one of the least reliable or
useful types of information for targeting purposes. We reported that
terrorism experts, trade community representatives, and some CBP
inspectors at ports we visited told us that CBP should explore requiring
more timely electronic transmittal of additional data elements for cargo
container targeting purposes, such as stowage plans (a map of where
each container aboard a ship is stored), container movement tracking
data, and entry data. Although we did not analyze the feasibility or the
costs and benefits of these suggestions, we reported that it could be
useful for CBP to explore requiring appropriate parties in the supply
chain, such as the importer, to provide additional data elements for use in
ATS to perform more complex linkage analyses and identify potential
anomalies in the shipping documents filed. Although we did not make a
recommendation directly related to data collection because we did not
analyze the feasibility or the costs and benefits of collecting additional
data, the recommendations in our report focused generally on
incorporating key elements of recognized modeling practices, and one of
the recognized modeling practices applicable to ATS is enhancing the
sources and types of information input into ATS. Furthermore, in July
2005, the DHS OIG issued an unclassified summary of an audit regarding
CBP’s targeting for maritime cargo containers, which concluded that
improvements were needed in the data to which ATS targeting rules are
applied. 6

In January 2009, CBP initiated an effort to collect additional data through
the Importer Security Filing and Additional Carrier Requirements,
collectively known as the 10+2 rule. The rule requires importers and
carriers to provide 10 data elements and 2 data elements, respectively, to
CBP for improving CBP’s ability to identify high-risk cargo container
shipments. Specifically, the 10+2 rule requires importers to submit
information about the commodities being transported in a shipment and
about entities involved in the supply chain. These additional data
elements include information that we reported could be helpful in
improving targeting efforts. In particular, the rule requires stowage plans
and some elements of entry data. We reported in September 2010 that



6
 DHS OIG, Audit of Targeting Oceangoing Cargo Containers (Unclassified Summary),
OIG-05-26.




Page 35                                               GAO-13-9 Supply Chain Security
                         Appendix II: Past Audit Findings and
                         Recommendations to Improve the Targeting
                         Process




                         the 10+2 rule data elements were available for identifying high-risk cargo
                         at that time, but that CBP had not yet updated ATS to fully incorporate the
                         data into its targeting criteria. 7 We recommended that CBP establish
                         milestones and time frames for updating the targeting criteria. In
                         December 2010, CBP provided us with a project plan for integrating the
                         data into its targeting criteria, and in early 2011, CBP implemented the
                         updated targeting criteria to address risk factors present in the Importer
                         Security Filing data.


Providing Additional     In November 2006, the DHS OIG reported on issues related to
Information Outside of   information that is available to targeters to conduct targeting activities. 8
ATS to Targeters         The DHS OIG found that CBP targeters did not always have access to a
                         particular system that directly accesses a database containing billions of
                         records on individuals and businesses. The DHS OIG recommended that
                         CBP provide targeters with access to that system or a similar system that
                         would allow targeters to access business records. In its report, the DHS
                         OIG stated that it considered the recommendation to be implemented
                         because CBP had acquired funding to grant personnel access to that
                         system and had issued a memo stating that all personnel that were to be
                         scheduled for targeting training should also have access to the system. 9
                         The DHS OIG also found that it was unclear which personnel involved in
                         targeting should have security clearances, and as a result, important
                         information affecting container targeting and inspection decisions may not
                         be available to the staff for making these decisions in a timely manner.
                         The DHS OIG recommended that CBP increase the number of targeters
                         with security clearances. In response to this recommendation, according
                         to the DHS OIG, CBP presented a corrective action plan with an
                         established completion date of June 30, 2007, for this recommendation.
                         The DHS OIG further noted that in July 2006, CBP issued a memo for
                         Security Clearance for Counter Terrorism Response Officers, directing
                         CBP field offices to forward applications of port personnel that need to



                         7
                          GAO-10-841.
                         8
                           The DHS OIG report issued in November 2006 contains sensitive information and is
                         therefore not publicly available.
                         9
                          The system discussed in the November 2006 DHS OIG report is the predecessor to the
                         third-party database of public and proprietary records that is mentioned in this report as a
                         tool that targeters use outside of ATS to research supply chain parties involved in a
                         shipment and to inform their decision whether to target a shipment for examination.




                         Page 36                                                     GAO-13-9 Supply Chain Security
Appendix II: Past Audit Findings and
Recommendations to Improve the Targeting
Process




have security clearances. On the basis of these efforts, the DHS OIG
reported that CBP had satisfied the intent of this recommendation.




Page 37                                        GAO-13-9 Supply Chain Security
Appendix III: Comments from the
             Appendix III: Comments from the Department
             of Homeland Security



Department of Homeland Security




             Page 38                                      GAO-13-9 Supply Chain Security
Appendix III: Comments from the Department
of Homeland Security




Page 39                                      GAO-13-9 Supply Chain Security
Appendix IV: GAO Contact and Staff
                  Appendix IV: GAO Contact and Staff
                  Acknowledgments



Acknowledgments

                  Stephen L. Caldwell, Director (202) 512-9610 or caldwells@gao.gov
GAO Contact
                  In addition to the contact named above, Christopher Conrad (Assistant
Acknowledgments   Director), Alana Finley, Richard Hung, Katie Mauldin, and Janay Sam
                  made key contributions to this report. Also contributing to this report were
                  Richard Brown, Frances Cook, Stanley Kostyla, and Lara Miklozek.




                  Page 40                                          GAO-13-9 Supply Chain Security
Related GAO Products
             Related GAO Products




             Supply Chain Security: Container Security Programs Have Matured, but
             Uncertainty Persists over the Future of 100 Percent Scanning.
             GAO-12-422T. Washington, D.C.: February 7, 2012.

             Maritime Security: Responses to Questions for the Record.
             GAO-11-140R. Washington, D.C.: October 22, 2010.

             Supply Chain Security: CBP Has Made Progress in Assisting the Trade
             Industry in Implementing the New Importer Security Filing Requirements,
             but Some Challenges Remain. GAO-10-841. Washington, D.C.:
             September 10, 2010.

             Supply Chain Security: Feasibility and Cost-Benefit Analysis Would Assist
             DHS and Congress in Assessing and Implementing the Requirement to
             Scan 100 Percent of U.S.-Bound Containers. GAO-10-12. Washington,
             D.C.: October 30, 2009.

             Intellectual Property: Better Data Analysis and Integration Could Help
             U.S. Customs and Border Protection Improve Border Enforcement
             Efforts. GAO-07-735. Washington, D.C.: April 26, 2007.

             Cargo Container Inspections: Preliminary Observations on the Status of
             Efforts to Improve the Automated Targeting System. GAO-06-591T.
             Washington, D.C.: March 30, 2006.

             Homeland Security: Summary of Challenges Faced in Targeting
             Oceangoing Cargo Containers for Inspection. GAO-04-557T.
             Washington, D.C.: March 31, 2004.




(441011)
             Page 41                                         GAO-13-9 Supply Chain Security
                      The Government Accountability Office, the audit, evaluation, and
GAO’s Mission         investigative arm of Congress, exists to support Congress in meeting its
                      constitutional responsibilities and to help improve the performance and
                      accountability of the federal government for the American people. GAO
                      examines the use of public funds; evaluates federal programs and
                      policies; and provides analyses, recommendations, and other assistance
                      to help Congress make informed oversight, policy, and funding decisions.
                      GAO’s commitment to good government is reflected in its core values of
                      accountability, integrity, and reliability.

                      The fastest and easiest way to obtain copies of GAO documents at no
Obtaining Copies of   cost is through GAO’s website (http://www.gao.gov). Each weekday
GAO Reports and       afternoon, GAO posts on its website newly released reports, testimony,
                      and correspondence. To have GAO e-mail you a list of newly posted
Testimony             products, go to http://www.gao.gov and select “E-mail Updates.”

Order by Phone        The price of each GAO publication reflects GAO’s actual cost of
                      production and distribution and depends on the number of pages in the
                      publication and whether the publication is printed in color or black and
                      white. Pricing and ordering information is posted on GAO’s website,
                      http://www.gao.gov/ordering.htm.
                      Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
                      TDD (202) 512-2537.
                      Orders may be paid for using American Express, Discover Card,
                      MasterCard, Visa, check, or money order. Call for additional information.
                      Connect with GAO on Facebook, Flickr, Twitter, and YouTube.
Connect with GAO      Subscribe to our RSS Feeds or E-mail Updates. Listen to our Podcasts.
                      Visit GAO on the web at www.gao.gov.
                      Contact:
To Report Fraud,
Waste, and Abuse in   Website: http://www.gao.gov/fraudnet/fraudnet.htm
                      E-mail: fraudnet@gao.gov
Federal Programs      Automated answering system: (800) 424-5454 or (202) 512-7470

                      Katherine Siggerud, Managing Director, siggerudk@gao.gov, (202) 512-
Congressional         4400, U.S. Government Accountability Office, 441 G Street NW, Room
Relations             7125, Washington, DC 20548

                      Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800
Public Affairs        U.S. Government Accountability Office, 441 G Street NW, Room 7149
                      Washington, DC 20548




                        Please Print on Recycled Paper.