oversight

Customs Service Modernization: Actions Initiated to Correct ACE Management and Technical Weaknesses

Published by the Government Accountability Office on 1999-05-13.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                          United States General Accounting Office

GAO                       Testimony
                          Before the Committee on Finance, U.S. Senate




For Release on Delivery
Expected at
10 a.m.
                          CUSTOMS SERVICE
Thursday,
May 13, 1999              MODERNIZATION

                          Actions Initiated to Correct
                          ACE Management and
                          Technical Weaknesses
                          Statement of Randolph C. Hite
                          Associate Director, Governmentwide and Defense
                          Information Systems
                          Accounting and Information Management Division




GAO/T-AIMD-99-186
Mr. Chairman and Members of the Committee:

Thank you for inviting me to participate in today’s Customs Service
oversight hearing. My statement will focus on Customs’ Automated
Commercial Environment, better known as ACE. Through ACE, Customs
intends to implement much needed improvements in the way it currently
enforces import trade laws and regulations, and assesses and collects
import duties, taxes, and fees, which total $22 billion annually.

The need to leverage information technology (IT) to improve the way that
Customs does business in the import arena is undeniable. Customs’
existing import processes and supporting systems are simply not
responsive to the business needs of either Customs or the trade
community, whose members collectively import about $1 trillion in goods
annually. These existing processes and systems are paper-intensive, error-
prone, and transaction-based, and they are out of step with the just-in-time
inventory practices used by the trade. Recognizing this, Congress enacted
the Customs Modernization and Informed Compliance Act, or “Mod” Act, to
define legislative requirements for improving import processing through an
automated system.1

Customs fully recognizes the severity of the problems with its approach to
managing import trade and is modernizing its import processes and
undertaking ACE as its import system solution. Begun in 1994, Customs’
estimate of the system’s 15-year life cycle cost is about $1.05 billion,
although this estimate is being revised upwards. In light of ACE’s
enormous mission importance and price tag, Customs’ approach to
investing in and engineering ACE demands disciplined and rigorous
management practices. Such practices are embodied in the Clinger-Cohen
Act of 19962 and other legislative and regulatory requirements, as well as



1Customs refers to Title VI of the North American Free Trade Agreement Implementation Act (Public
Law 103-182, 19 U.S.C. 1411 et seq) as the Customs Modernization and Informed Compliance Act or
“Mod” Act.

2
  Although the Clinger-Cohen Act (Public Law 104-106) was passed after Customs began developing
ACE, its principles are based on practices that are widely considered to be integral to successful IT
investments. For an analysis of the management practices of several leading private and public sector
organizations on which the Clinger-Cohen Act is based, see Executive Guide: Improving Mission
Performance Through Strategic Information Management and Technology (GAO/AIMD-94-115, May
1994). For an overview of the IT management process envisioned by Clinger-Cohen, see Assessing Risk
and Returns: A Guide for Evaluating Federal Agencies’ IT Investment Decision-making (GAO/AIMD-
10.1.13, February 1997).




Page 1                                                                         GAO/T-AIMD-99-186
                       accepted industry system/software engineering models, such as those
                       published by the Software Engineering Institute (SEI).3

                       Unfortunately, Customs has not employed such practices on ACE over the
                       last 5 years. Our February 1999 report on ACE,4 upon which my testimony
                       today is based, describes serious ACE management and technical
                       weaknesses. The weaknesses that we reported are (1) building ACE
                       without a complete and enforced enterprise systems architecture,
                       (2) investing in ACE without a firm basis for knowing that it is a cost-
                       effective system solution, and (3) building ACE without employing
                       engineering rigor and discipline. My testimony will address each of these
                       points as well as our recommendations for correcting them. To Customs’
                       credit, its leadership has agreed with our findings, has initiated actions to
                       implement our recommendations, and is committed to seeing that these
                       actions are completed before investing huge sums of money in the system.



ACE: A Brief History   Customs began ACE in 1994, and its early estimate of the cost and time to
                       develop the system was $150 million over 10 years. At this time, Customs
                       also decided to first develop a prototype of ACE, referred to as NCAP
                       (National Customs Automation Program prototype), and then to complete
                       the system. In May 1997,5 we testified that Customs’ original schedule for
                       completing the prototype was January 1997, and that Customs did not have
                       a schedule for completing ACE. At that time, Customs agreed to develop a
                       comprehensive project plan for ACE.

                       In November 1997, Customs estimated that the system would cost
                       $1.05 billion to develop, operate, and maintain throughout its life cycle.
                       Customs plans to develop and deploy the system in 21 increments from
                       1998 through 2005, the first four of which would constitute NCAP.

                       Currently, Customs is well over 2 years behind its original NCAP schedule.
                       Because Customs experienced problems in developing NCAP software in-


                       3Software  Development Capability Maturity ModelSM (SW-CMM®) and Software Acquisition Capability
                       Maturity ModelSM (SA-CMM®). Capability Maturity ModelSM is a service mark of Carnegie Mellon
                       University, and CMM® is registered in the U.S. Patent and Trademark Office.

                       4CustomsService Modernization: Serious Management and Technical Weaknesses Must Be Corrected
                       (GAO/AIMD-99-41, February 26, 1999).

                       5
                           Customs Service Modernization: ACE Poses Risks and Challenges (GAO/T-AIMD-97-96, May 15, 1997).




                       Page 2                                                                       GAO/T-AIMD-99-186
                     house, the first NCAP release was not deployed until May 1998--16 months
                     late. In view of the problems it experienced with the first release, Customs
                     contracted out for the second NCAP release, and deployed this release in
                     October 1998--21 months later than originally planned. Customs’ most
                     recent dates for deploying the final two NCAP releases (0.3 and 0.4) are
                     March 1999 and September 1999, which are 26 and 32 months later than the
                     original deployment estimates, respectively. According to Customs, these
                     dates will slip farther because of funding delays.

                     Additionally, Customs officials told us that a new ACE life cycle cost
                     estimate is being developed, but that it was not ready to be shared with us.
                     At the time of our review, Customs’ $1.05 billion estimate developed in 1997
                     was the official ACE life cycle cost estimate. However, a January 1999 ACE
                     business plan specifies a $1.48 billion life cycle cost estimate.



Customs Has Been     At the time of our review, Customs was not building ACE within the
                     context of an enterprise systems architecture, or “blueprint” of its
Developing ACE       agencywide future systems environment. Such an architecture is a
Without a Complete   fundamental component of any rationale and logical strategic plan for
                     modernizing an organization’s systems environment. As such, the Clinger-
Enterprise Systems   Cohen Act requires agency Chief Information Officers (CIO) to develop,
Architecture         maintain, and implement an information technology architecture. Also, the
                     Office of Management and Budget (OMB) issued guidance in 1996 that
                     requires agency IT investments to be architecturally compliant. These
                     requirements are consistent with, and in fact based on, information
                     technology management practices of leading private and public sector
                     organizations.

                     Simply stated, an enterprise systems architecture specifies the system (e.g.,
                     software, hardware, communications, security, and data) characteristics
                     that the organization’s target systems environment is to possess. Its
                     purpose is to define, through careful analysis of the organization’s strategic
                     business needs and operations, the future systems configuration that
                     supports not only the strategic business vision and concept of operations,
                     but also defines the optimal set of technical standards that should be met to
                     produce homogeneous systems that can interoperate effectively and be
                     maintained efficiently. Our work has shown that in the absence of an
                     enterprise systems architecture, incompatible systems are produced that




                     Page 3                                                       GAO/T-AIMD-99-186
require additional time and resources to interconnect and to maintain and
that suboptimize the organization’s ability to perform its mission.6

We first reported on Customs’ need for a systems architecture in May 1996
and testified on this subject in May 1997.7 In response, Customs developed
and published an architecture in July and August 1997. We reviewed this
architecture and reported in May 1998 that it was not effective because it
was neither complete nor enforced.8 For example, the architecture did not

1. fully describe Customs’ business functions and their relationships,

2. define the information needs and flows among these functions, and

3. establish the technical standards, products, and services that would be
characteristic of its target systems environment on the basis of these
business specifications.

Accordingly, we recommended that Customs complete its enterprise
information systems architecture and establish compliance with the
architecture as a requirement of Customs’ information technology
investment management process. In response, Customs agreed to develop
a complete architecture and establish a process to ensure compliance.
Customs reports that its architecture will be completed in May 1999. Also,
in January 1999, Customs changed its internal procedures to provide for
effective enforcement of its architecture, once it is completed. Until the
architecture is completed and enforced, Customs risks spending millions of
dollars to develop, acquire, and maintain information systems, including
ACE, that do not effectively and efficiently support the agency’s mission
needs.




6
 Air Traffic Control: Complete and Enforced Architecture Needed for FAA Systems Modernization
(GAO/AIMD-97-30, February 3, 1997).

7Customs Service Modernization: Strategic Information Management Must Be Improved for National
Automation Program To Succeed (GAO/AIMD-96-57, May 9, 1996) and Customs Service Modernization:
ACE Poses Risks and Challenges (GAO/T-AIMD-97-96, May 15, 1997).

8
 Customs Service Modernization: Architecture Must Be Complete and Enforced to Effectively Build
and Maintain Systems (GAO/AIMD-98-70, May 5, 1998).




Page 4                                                                      GAO/T-AIMD-99-186
Customs Has Not Been   Effective IT investment management is predicated on answering one basic
                       question: Is the organization doing the “right thing” by investing specified
Managing Its           time and resources in a given project or system? The Clinger-Cohen Act
Investment in ACE      and OMB and GAO guidance together provide an effective IT investment
                       management framework for answering this question. Among other things,
Effectively            they describe the need for

                       1. identifying and analyzing alternative system solutions,

                       2. developing reliable estimates of the alternatives’ respective costs and
                       benefits and investing in the most cost-beneficial alternative, and

                       3. to the maximum extent practical, structuring major projects into a series
                       of increments to ensure that each increment constitutes a wise investment.

                       Customs did not satisfy any of these requirements for ACE. First, Customs
                       did not identify and evaluate a full range of alternatives to its defined ACE
                       solution before commencing development activities. For example,
                       Customs did not consider how ACE would relate to another Treasury-
                       proposed system for processing import trade data, known as the
                       International Trade Data System (ITDS), including considering the extent
                       to which ITDS should be used to satisfy needed import processing
                       functionality. Initiated in 1995 as a project to develop a coordinated,
                       governmentwide system for the collection, use, and dissemination of trade
                       data, the ITDS project is headed by the Treasury Deputy Assistant
                       Secretary for Regulatory, Tariff and Trade Enforcement. The system is
                       expected to reduce the burden federal agencies place on organizations by
                       requiring that they respond to duplicative data requests. Treasury intends
                       for the system to serve as the single point for collecting, editing, and
                       validating trade data as well as collecting and accounting for trade revenue.
                       At the time of our review of ACE, these functions were also planned for
                       ACE.

                       Similarly, Customs did not evaluate different ACE architectural designs,
                       such as the use of a mainframe-based versus client/server-based hardware
                       architecture. Also, Customs did not evaluate alternative development
                       approaches, such as acquisition versus in-house development. In short,
                       Customs committed to and began building ACE without knowing whether
                       it had chosen the most cost-effective alternative and approach.




                       Page 5                                                       GAO/T-AIMD-99-186
Second, Customs did not develop a reliable life cycle cost estimate for the
approach it selected. SEI has developed a method for project managers to
use to determine the reliability of project cost estimates. Using SEI’s
method, we found that Customs’ $1.05 billion ACE life cycle cost estimate
was not reliable, and that it did not provide a sound basis for Customs’
decision to invest in ACE. For example, in developing the cost estimate,
Customs did not (1) use a cost model, (2) account for changes in its
approach to building different ACE increments, (3) account for changes to
ACE software and hardware architecture, or (4) have historical project
cost data upon which to compare its ACE estimate.

Moreover, the $1.05 billion cost estimate used to economically justify ACE
omitted relevant costs. For instance, the costs of technology refreshment
and system requirements definition were not included (see table 1).
Exacerbating this problem, Customs represented its ACE cost estimate as a
precise point estimate rather than explicitly disclosing to investment
decisionmakers in Treasury, OMB, and Congress the estimate’s inherent
uncertainty.



Table 1: Estimated Costs Omitted From Customs’ ACE Cost-Benefit Analysis

                                                                   Excluded cost
Excluded cost description                                          estimate
Hardware and software upgrades at each port office (e.g., desktop $73 to $172 million
workstations and operating systems, application and data servers,
and database management systems).
Security analysis, project planning and management, and            $23 million
independent verification and validation.
Requirements definition, component integration, regression testing, No estimate
and training.                                                       available

Customs’ projections of ACE benefits were also unreliable because they
were either overstated or unsupported. For example, the analysis includes
$203.5 million in savings attributable to 10 years of avoided maintenance
and support costs on the Automated Commercial System (ACS)--the
system ACE is to replace. However, Customs would not have avoided
maintenance and support costs for 10 years. At the time of Customs’
analysis, it planned to run both systems in parallel for 4 years, and thus
planned to spend about $53 million on ACS maintenance and support
during this period. As another example, $650 million in savings was not
supported by verifiable data or analysis, and $644 million was based on




Page 6                                                               GAO/T-AIMD-99-186
                          assumptions that were analytically sensitive to slight changes, making this
                          $644 million a “best case” scenario.

                          Third, Customs is not making its investment decisions incrementally as
                          required by the Clinger-Cohen Act and OMB. Although Customs has
                          decided to implement ACE as a series of 21 increments, it is not justifying
                          investing in each increment on the basis of defined costs and benefits and a
                          positive return on investment for each increment. Further, once it has
                          deployed an increment at a pilot site for evaluation, it is not validating the
                          benefits that the increment actually provides, and it is not accounting for
                          costs on each increment so that it can demonstrate that a positive return on
                          investment was actually achieved. Instead, Customs estimated the costs
                          and benefits for the entire system--all 21 increments, and used this as
                          economic justification for ACE.

                          Mr. Chairman, our work has shown that such estimates of many system
                          increments to be delivered over many years are impossible to make
                          accurately because later increments are not well understood or defined.
                          Also, these estimates are subject to change in light of experiences on
                          nearer term increments and changing business needs. By using an
                          inaccurate, aggregated estimate that is not refined as increments are
                          developed, Customs is committing enormous resources with no assurance
                          that it will achieve a reasonable return on its investment. This “grand
                          design” approach to managing large system modernization projects has
                          repeatedly proven to be ineffective across the federal government,
                          resulting in huge sums invested in systems that do not provide expected
                          benefits. Failure of the grand design approach was a major impetus for the
                          IT management reforms contained in the Clinger-Cohen Act.



Customs Has Not Been      Software process maturity is one important and recognized measure of
                          determining whether an organization is managing a system or project the
Managing ACE              “right way,” and thus whether or not the system will be completed on time
Software                  and within budget and will deliver promised capabilities. The Clinger-
                          Cohen Act requires agencies to implement effective IT management
Development/              processes, such as processes for managing software development and
Acquisition Effectively   acquisition. SEI has developed criteria for determining an organization’s
                          software development and acquisition effectiveness or maturity.

                          Customs lacks the capability to effectively develop or acquire ACE
                          software. Using SEI criteria for process maturity at the “repeatable” level,
                          which is the second level on SEI’s five-level scale and means that an



                          Page 7                                                       GAO/T-AIMD-99-186
organization has the software development/acquisition rigor and discipline
to repeat project successes, we evaluated ACE software processes. In
February 1999,9 we reported that the software development processes that
Customs was employing on NCAP 0.1, the first release of ACE, were not
effective. For example, we reported that Customs lacked effective
software configuration management, which is important for establishing
and maintaining the integrity of the software products during development.
Also, we reported that Customs lacked a software quality assurance
program, which greatly increased the risk of ACE software not meeting
process and product standards. Further, we reported that Customs lacked
a software process improvement program to effectively address these and
other software process weaknesses. Our findings concerning ACE
software development maturity are summarized in table 2.



Table 2: Summary of ACE Software Development Maturity

Key process areas                                                    Satisfied       Not satisfied
Requirements management                                                                            X
Software project planning                                                                          X
Software project tracking and oversight                                                            X
Software quality assurance                                                                         X
Software configuration management                                                                  X
Note: These represent five of six level 2 key process areas in SEI’s Software Development Capability
Maturity Model. We did not evaluate ACE in the sixth level 2 key process area--software subcontract
management--because Customs did not use subcontractors on ACE.


As discussed in our brief history of ACE, after Customs developed NCAP
0.1 in-house, it decided to contract out for the development of NCAP 0.2,
thus changing its role on ACE from being a software developer to being a
software acquirer. According to SEI, the capabilities needed to effectively
acquire software are different than the capabilities needed to effectively
develop software. Regardless, we reported later in February 199910 that
the software acquisition processes that Customs was employing on NCAP
0.2 were not effective. For example, Customs did not have an effective
software acquisition planning process and, as such, could not effectively


9Customs Service Modernization: Ineffective Software Development Processes Increase Customs
System Development Risks (GAO/AIMD-99-35, February 11, 1999).

10
     GAO/AIMD-99-41, February 26, 1999.




Page 8                                                                        GAO/T-AIMD-99-186
                        establish reasonable plans for performing software engineering and for
                        managing the software project. Also, Customs did not have an effective
                        evaluation process, meaning that it lacked the capability for ensuring that
                        contractor-developed software satisfied defined requirements. Our
                        findings concerning ACE software acquisition maturity are summarized in
                        table 3.



                        Table 3: Summary of ACE Software Acquisition Maturity

                        Key process areas                                                     Satisfied      Not satisfied
                        Software acquisition planning                                                                    X
                        Solicitation                                                                                     X
                        Requirements development and management                                                          X
                        Project office management                                                                        X
                        Contract tracking and oversight                                                                  X
                        Evaluation                                                                                       X
                        Transition and support                                                                           X
                        Acquisition risk management                                                                      X
                        Note: These represent seven level 2 key process areas in SEI’s Software Acquisition Capability
                        Maturity Model. We also evaluated one key process area associated with the “defined” level of
                        process maturity (level 3)--acquisition risk management.




Customs Has Initiated   To address ACE management weaknesses, we recommended that Customs

Actions to Implement    • analyze alternative approaches to satisfying its import automation
Our Recommendations       needs, including addressing the ITDS/ACE relationship;
                        • invest in its defined ACE solution incrementally, meaning for each
for Strengthening ACE     system increment (1) rigorously estimate and analyze costs and
Management                benefits, (2) require a favorable return-on-investment and compliance
                          with Customs’ enterprise systems architecture, and (3) validate actual
                          costs and benefits once an increment is piloted, compare actuals to
                          estimates, use the results in deciding on future increments, and report
                          the results to congressional authorizers and appropriators;
                        • establish an effective software process improvement program and
                          correct the software process weaknesses identified in our report,
                          thereby bringing ACE software process maturity to a least an SEI level 2;
                          and
                        • require at least SEI level 2 processes of all ACE software contractors.



                        Page 9                                                                        GAO/T-AIMD-99-186
              In commenting on our February 1999 report, the Commissioner of Customs
              agreed with our findings and committed to implementing our
              recommendations. In April 13, 1999, testimony, the Commissioner outlined
              several actions Customs has underway to improve ACE project
              management and address our recommendations.11 In brief, Customs

              • plans to acquire the services of a prime contractor that is at least SEI
                level 3 certified to help Customs implement mature software processes
                and plan, implement, and manage its modernization efforts, including
                ACE;
              • plans to hire a Federally Funded Research and Development Center
                (FFRDC) to support solicitation, selection, contract award, contract
                management, and ongoing oversight of the prime contractor;
              • has hired a contractor to update and improve the ACE life cycle cost
                estimate;
              • has retained an audit firm to provide independent reviews of Customs’
                methodology for estimating ACE costs and revised cost/benefit analysis;
              • has engaged a contractor to update and improve the ACE cost/benefit
                analysis by addressing our concerns, including use of ITDS as the
                interface for ACE;
              • plans to perform additional cost/benefit analyses of ACE increments
                and analyze alternative approaches to building ACE; and
              • plans to ensure that each ACE increment is compliant with Customs’
                enterprise systems architecture.



Conclusions   Successful systems modernization is absolutely critical to Customs’ ability
              to perform its trade import mission efficiently and effectively in the 21 st
              century. Systems modernization success, however, depends on doing the
              “right thing, the right way.” To be “right,” organizations must (1) invest in
              and build systems within the context of a complete and enforced enterprise
              systems architecture, (2) make informed, data-driven decisions about
              investment options based on expected and actual return-on-investment for
              system increments, and (3) build system increments using mature software
              engineering practices. Our reviews of agency system modernization efforts
              over the last 5 years point to weaknesses in these three areas as the root



              11
               Statement of Commissioner Raymond W. Kelly, Commissioner of the Customs Service, Authorization
              Hearing with the Customs Service Before the House Committee on Ways and Means Trade
              Subcommittee, April 13, 1999.




              Page 10                                                                   GAO/T-AIMD-99-186
                   causes of their not delivering promised system capabilities on time and
                   within budget.12

                   Until Customs corrects its ACE management and technical weaknesses,
                   the federal government’s troubled experience on other modernization
                   efforts is a good indicator for ACE. In fact, although Customs does not
                   collect data to know whether the first two ACE releases are already falling
                   short of cost and performance expectations, the data it does collect on
                   meeting milestones show that the first two releases have taken about 2
                   years longer than originally planned. This is precisely the type of
                   unaffordable outcome that can be avoided by making the management and
                   technical improvements we recommended.

                   To Customs’ credit, it fully recognizes the seriousness of the situation, has
                   quickly initiated actions to begin correcting its ACE management and
                   technical weaknesses, and is committed to each of these actions. We are
                   equally committed to working with Customs as it strives to do so and with
                   Congress as it oversees this important initiative.

                   This concludes my statement. I would be glad to respond to any questions
                   that you or other Members of the Committee may have at this time.




                   12
                     Tax System Modernization: Management and Technical Weaknesses Must Be Corrected If
                   Modernization Is to Succeed (GAO/AIMD-95-156, July 26, 1995); Tax Systems Modernization: Actions
                   Underway but IRS Has Not Yet Corrected Management and Technical Weaknesses (GAO/AIMD-96-106,
                   June 7, 1996); Tax Systems Modernization: Blueprint Is a Good Start but Not Yet Sufficiently Complete
                   to Build or Acquire Systems (GAO/AIMD/GGD-98-54, February 24, 1998); Air Traffic Control: Immature
                   Software Acquisition Processes Increase FAA System Acquisition Risks (GAO/AIMD-97-47, March 21,
                   1997); Air Traffic Control: Complete and Enforced Architecture Needed for FAA Systems
                   Modernization (GAO/AIMD-97-30, February 3, 1997); and Air Traffic Control: Improved Cost
                   Information Needed to Make Billion Dollar Modernization Investment Decisions (GAO/AIMD-97-20,
                   January 22, 1997).




(511151)   Leter   Page 11                                                                        GAO/T-AIMD-99-186
Ordering Information

The first copy of each GAO report and testimony is free.
Additional copies are $2 each. Orders should be sent to the
following address, accompanied by a check or money order made
out to the Superintendent of Documents, when necessary, VISA and
MasterCard credit cards are accepted, also.

Orders for 100 or more copies to be mailed to a single address are
discounted 25 percent.

Orders by mail:

U.S. General Accounting Office
P.O. Box 37050
Washington, DC 20013

or visit:

Room 1100
700 4th St. NW (corner of 4th and G Sts. NW)
U.S. General Accounting Office
Washington, DC

Orders may also be placed by calling (202) 512-6000
or by using fax number (202) 512-6061, or TDD (202) 512-2537.

Each day, GAO issues a list of newly available reports and
testimony. To receive facsimile copies of the daily list or any list
from the past 30 days, please call (202) 512-6000 using a touchtone
phone. A recorded menu will provide information on how to obtain
these lists.

For information on how to access GAO reports on the INTERNET,
send an e-mail message with “info” in the body to:

info@www.gao.gov

or visit GAO’s World Wide Web Home Page at:

http://www.gao.gov
United States                       Bulk Rate
General Accounting Office      Postage & Fees Paid
Washington, D.C. 20548-0001           GAO
                                 Permit No. GI00
Official Business
Penalty for Private Use $300

Address Correction Requested