oversight

Customs Service Modernization: Serious Management and Technical Weaknesses Must Be Corrected

Published by the Government Accountability Office on 1999-02-26.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                 United States General Accounting Office

GAO              Report to Congressional Requesters




February 1999
                 CUSTOMS SERVICE
                 MODERNIZATION
                 Serious Management
                 and Technical
                 Weaknesses Must Be
                 Corrected




GAO/AIMD-99-41
      United States
GAO   General Accounting Office
      Washington, D.C. 20548

      Accounting and Information
      Management Division

      B-280416

      February 26, 1999

      The Honorable Ben Nighthorse Campbell
      Chairman, Subcommittee on Treasury,
        General Government, and Civil Service
      Committee on Appropriations
      United States Senate

      The Honorable Jim Kolbe
      Chairman, Subcommittee on Treasury,
        Postal Service, and General Government
      Committee on Appropriations
      House of Representatives

      This report responds to your requests that we review the U. S. Customs Service’s management
      of the Automated Commercial Environment (ACE), including whether Customs has adequately
      justified ACE cost-effectiveness. Customs plans to spend over $1 billion on ACE, which is planned
      to support modernized import processing. We found that Customs’ is not managing ACE
      effectively and it does not have a firm basis for concluding that ACE is cost-effective.
      Accordingly, we are making recommendations to the Commissioner of Customs for
      strengthening the management and technical weaknesses we identified.

      We are sending copies of this report to the Secretary of the Treasury; Commissioner of
      Customs; Director of the Office of Management and Budget; and Ranking Minority Members of
      the Subcommittee on Treasury, General Government, and Civil Service of the Senate
      Committee on Appropriations and the Subcommittee on Treasury, Postal Service, and General
      Government of the House Committee on Appropriations. We are also sending copies of this
      report to the Chairmen and Ranking Minority Members of the Senate Committee on Finance
      and the House Committee on Ways and Means and to other congressional committees. We will
      also make copies available to other interested parties upon request. If you have questions or
      wish to discuss the issues in this report, please contact me at (202) 512-6240. Major contributors
      to this report are listed in appendix V.




      Randolph C. Hite
      Associate Director, Governmentwide
        and Defense Information Systems
Executive Summary


             The U.S. Customs Service plans to spend well over $1 billion to modernize
Purpose      its systems environment for certain core missions: facilitating
             international trade, enforcing laws governing the flow of goods across U.S.
             borders, and assessing and collecting about $22 billion annually on
             imported merchandise. The Clinger-Cohen Act of 1996, and related system
             and software engineering best practices, provide federal agencies with a
             framework for effectively managing such modernization efforts.

             The Customs modernization effort, known as the Automated Commercial
             Environment or ACE, is of longstanding concern to the House
             Appropriations Committee, Subcommittee on Treasury, Postal Service,
             and General Government, and the Senate Appropriations Committee,
             Subcommittee on Treasury, General Government, and Civil Service.
             Accordingly, the Subcommittee Chairmen asked GAO to determine whether
             Customs is effectively managing ACE, including whether Customs has
             adequately justified ACE cost-effectiveness.


             Title VI of the North American Free Trade Agreement Implementation Act,
Background   Public Law 103-182, enabled Customs to speed the processing of imports
             and improve compliance with trade laws. Customs refers to this legislation
             as the “Customs Modernization and Informed Compliance Act” or “Mod”
             Act. The primary purpose of the act is to streamline and automate
             Customs’ commercial operations. According to Customs, modernized
             commercial operations will permit it to more efficiently handle its
             burgeoning import workloads and expedite the movement of merchandise
             at more than 300 ports of entry. For 1995 through 2001, Customs estimates
             that the annual dollar volume of import trade will increase from
             $761 billion to $1.1 trillion, with the number of commercial entries
             processed annually increasing from 13.1 million to 20.6 million.

             ACE  is Customs’ system solution to a modernized commercial environment.
             In November 1997, Customs estimated that it would cost $1.05 billion to
             develop, operate, and maintain ACE over the 15 year period between fiscal
             year 1994 and 2008. Customs plans to develop and deploy ACE in multiple
             increments. The first four increments are known collectively as the
             National Customs Automation Program (NCAP). The first increment, NCAP
             0.1, was deployed for field operation and evaluation in May 1998. As of the
             end of fiscal year 1998, Customs reported that it had spent $62.1 million on
             ACE.




             Page 2                             GAO/AIMD-99-41 Customs Service Modernization
                   Executive Summary




                   Customs is not managing ACE effectively, and it does not have a firm basis
Results in Brief   for concluding that ACE is a cost-effective solution to modernizing its
                   commercial environment. GAO found serious weaknesses relating to
                   architectural definition, investment management, and software
                   development and acquisition that must be corrected before further
                   investment in ACE is justified.

                   First, Customs is not building ACE within the context of a complete and
                   enforced systems architecture or “construction plan” that precludes
                   inconsistent system design and development decisions. In May 1998, GAO
                   reported that Customs’ architecture was incomplete because it was not
                   based on a complete understanding of its enterprisewide functional and
                   information needs. GAO also reported that Customs had not yet instituted
                   effective procedures for ensuring compliance with the architecture once it
                   is completed. Customs is attempting to complete its architecture, but it
                   has not yet done so. Until its architecture is completed and effectively
                   enforced, Customs will not have adequate assurance that information
                   systems like ACE will optimally support its needs across all business areas.

                   Further, Customs lacks a reliable estimate of what ACE will cost to build,
                   deploy, and maintain; and Customs has neither adequately justified, nor is
                   it effectively monitoring, ACE’s cost-effectiveness. Specifically, Customs
                   did not use rigorous cost estimating techniques in preparing its cost
                   estimate, and did not disclose the inherent imprecision of the estimate.
                   Additionally, Customs omitted costs and inflated benefits in preparing its
                   cost-benefit analysis. Moreover, Customs is not using effective incremental
                   investment management practices. While Customs plans to
                   develop/acquire ACE in 21 increments, these increments are not
                   individually cost-benefit justified, and Customs is not determining what
                   benefits each increment, once operational, actually provides. As a result,
                   Customs will not know if ACE’s expected return-on-investment is actually
                   being realized until it has already spent hundreds of millions of dollars
                   developing/acquiring the entire system. This combination of unreliable
                   investment information and analysis, and a “grand design” approach to
                   justifying and managing system investments,1 has failed consistently on
                   other agency modernization efforts over the past two decades, has been
                   abandoned by successful organizations, and was a major reason for the
                   information technology investment management reforms in the
                   Clinger-Cohen Act of 1996.


                   1
                    The “grand design” approach involves investing in a large, long-term, expensive project based on cost
                   and benefit estimates prepared at the outset and attempting to deliver the entire project years later as
                   a single increment.



                   Page 3                                           GAO/AIMD-99-41 Customs Service Modernization
                         Executive Summary




                         These investment risks and uncertainties are compounded by the fact that
                         Customs is not employing effective software engineering practices on ACE.
                         Specifically, Customs developed the first ACE increment in-house using its
                         own software developers, but because of cost and schedule delays, it
                         decided to acquire the second ACE increment from a software development
                         contractor. GAO found that Customs has neither the capability to
                         effectively develop nor acquire ACE and that its processes for doing both,
                         according to widely accepted and proven software capability maturity
                         models, are ad hoc, immature, and ineffective.



Principal Findings

Customs Is Developing    The Clinger-Cohen Act recognizes the importance of architectures by
ACE Without a Complete   requiring agency Chief Information Officers (CIO) to develop, maintain, and
Enterprise Systems       facilitate an integrated systems architecture. Customs does not have a
                         complete target systems architecture. In May 1998, GAO reported that
Architecture             Customs’ target systems architecture was not effective because it was
                         neither complete nor enforced, and GAO made several recommendations
                         for needed improvements.2 For example, GAO reported that the
                         architecture did not (1) fully describe Customs’ business functions,
                         (2) define the information needs and flows among these functions, and
                         (3) establish the technical standards, products, and services that will be
                         used to build systems that support these defined business functions and
                         needs. GAO also reported that Customs did not require that its systems be
                         architecturally compliant or that architectural deviations be justified and
                         documented.

                         Customs officials acknowledged the limitations in the agency’s
                         architecture and its enforcement. They agreed to define functions,
                         information needs, and flows across Customs’ six business areas and, in
                         light of this definition, reevaluate the technical characteristics the
                         architecture specified for its system environment. Customs originally
                         planned to have completed its target architecture by September 1998, but
                         that date has slipped to May 1999. Since receiving a draft of this report,
                         Customs has changed its investment management procedures with the
                         intent of ensuring that its architecture, once completed, can be enforced
                         effectively. Until its architecture is complete, however, Customs risks
                         spending millions of dollars to develop, acquire, and maintain information

                         2
                          Customs Service Modernization: Architecture Must Be Complete and Enforced to Effectively Build
                         and Maintain Systems (GAO/AIMD-98-70, May 1998).



                         Page 4                                        GAO/AIMD-99-41 Customs Service Modernization
                    Executive Summary




                    systems, including ACE, that do not effectively and efficiently support the
                    agency’s mission needs.


ACE Investment      The Clinger-Cohen Act and Office of Management and Budget (OMB)
Management Is Not   guidance provide an effective framework for information technology (IT)
Effective           investment management. Together, they establish requirements for
                    (1) identifying all promising alternative system solutions, (2) developing
                    reliable estimates of project life-cycle costs and benefits, and investing in
                    the most cost-beneficial alternative, and (3) to the maximum extent
                    practical, structuring major projects into a series of increments to ensure
                    that each increment constitutes a wise investment.

                    Customs did not consider potential alternatives, such as using the
                    International Trade Data System (ITDS), to perform certain critical
                    functions, before deciding to invest in ACE.3 ITDS was initiated in 1995 as a
                    project to implement the National Performance Review recommendation
                    to develop a coordinated, governmentwide system for the collection, use,
                    and dissemination of trade data. At that time, a multiagency board of
                    directors was established, headed by the Department of the Treasury with
                    Customs and other major trade-related agencies represented. The
                    Department of the Treasury is responsible for designing and developing
                    ITDS, which is expected to reduce the burden that federal agencies place on
                    international organizations by requiring that they respond to duplicative
                    data requests. Treasury intends for the system to serve as the single point
                    for collecting, editing, and validating trade data as well as collecting and
                    accounting for trade revenue—functions that are also planned for ACE.

                    Further, for the alternative it selected, i.e., ACE, Customs did not develop a
                    reliable life-cycle cost estimate. Carnegie Mellon University’s Software
                    Engineering Institute (SEI) has developed criteria by which the reliability
                    of project cost estimates can be assessed. Using SEI’s criteria, GAO found
                    that the processes used by Customs to develop its $1.05 billion ACE
                    life-cycle cost estimate were neither thorough nor disciplined, and as a
                    result, Customs’ ACE cost estimate is not reliable and does not provide a
                    sound basis for investment decision-making. For example, Customs did
                    not use a cost model, did not account for changes in its approach to
                    building different ACE increments, did not account for changes to ACE
                    software and hardware architecture, and did not have the requisite

                    3
                     The U.S. Department of the Treasury has developed an ITDS project plan, system specification,
                    cost-benefit analysis, and related documentation. It plans to begin developing ITDS in 1999 and to fully
                    implement it by 2003. It estimates that ITDS will cost $256 million to develop, deploy, and maintain
                    through 2005.



                    Page 5                                          GAO/AIMD-99-41 Customs Service Modernization
Executive Summary




historical project cost data upon which to compare its ACE estimate.
Additionally, the $1.05 billion cost estimate omits various relevant cost
elements, such as requirements definition, data warehouse development,
system documentation development, system integration, training, and
hardware/software technology refreshment. Customs then exacerbated
these problems by representing its ACE cost estimate as a precise, point
estimate, rather than explicitly describing the estimate’s inherent
uncertainty to ensure that it would be used appropriately.

Moreover, Customs’ projections of ACE benefits are overstated by at least
$52.8 million. The analysis includes $203.5 million in savings attributable
to 10 years of avoided maintenance and support costs on the Automated
Commercial System (ACS)—the system that ACE is to replace. However,
Customs will not avoid maintenance and support costs for 10 years.
Because Customs plans to run both systems in parallel for 4 years, it will
expend $52.8 million on continued maintenance and support of ACS during
this period.

Lastly, although Customs has decided to implement ACE as a series of 21
increments, it is not making its investment decisions incrementally as
required by the Clinger-Cohen Act and OMB. Specifically, Customs is not
justifying investing in each increment on the basis of measurable benefits,
and once it has deployed an increment at a pilot site for evaluation, it is
not validating the benefits that the increment actually provides. Instead,
Customs has estimated costs and benefits for the entire system (i.e., all 21
increments). Such estimates of many system increments to be delivered
over many years are impossible to make accurately because later
increments are not well understood or defined, and are subject to change
in light of experiences on nearer term increments and changing business
needs. By using an inaccurate, aggregated estimate that is not refined as
increments are developed, Customs is committing enormous resources
with no assurance that it will achieve a reasonable return on its
investment. This “grand design” or “big bang” approach to managing large
system modernization projects has repeatedly proven to be ineffective,
resulting in huge sums invested in systems that do not provide expected
benefits.




Page 6                              GAO/AIMD-99-41 Customs Service Modernization
                           Executive Summary




Customs Lacks the          The Clinger-Cohen Act requires the establishment of effective IT
Capability to Develop or   management processes, including processes for managing software
Acquire ACE Software       development and acquisition. SEI has developed criteria for determining
                           organizations’ software development and acquisition effectiveness or
Effectively                maturity.4

                           Using SEI criteria for process maturity at the “repeatable” level, which is
                           the second level on SEI’s five-level scale and means that an organization
                           has the software development/acquisition rigor and discipline to repeat
                           project successes, GAO evaluated ACE software processes. In
                           February 1999, GAO reported that Customs lacked the capability to develop
                           software effectively on several projects, including ACE.5 For example, GAO
                           reported that NCAP 0.1 lacked an effective software configuration
                           management process, which is important for establishing and maintaining
                           the integrity of the software products during development, and NCAP 0.1
                           did not have any type of software quality assurance program, which
                           greatly increases the risk of ACE software not meeting process and product
                           standards. GAO also reported that Customs lacked a software process
                           improvement program to effectively address these and other software
                           process weaknesses. Accordingly, GAO made several recommendations for
                           needed improvements. Customs agreed with GAO’s findings and stated that
                           it initiated steps to implement GAO’s recommendations, including assigning
                           responsibility for software process improvement.

                           Additionally, GAO found that Customs lacks the capability to acquire ACE
                           software effectively. For example, Customs did not have an effective
                           software acquisition planning process, and therefore could not effectively
                           establish reasonable plans for performing software engineering and for
                           managing the software project. Also, Customs did not have an effective
                           evaluation process, which means that it lacked the means for assuring that
                           contractor-developed software satisfied defined requirements.

                           Because of these and other serious software process weaknesses,
                           Customs’ ability to either develop or acquire ACE software is immature, and
                           therefore project success is unlikely.




                           4
                             Software Development Capability Maturity ModelSM (SW-CMM) and Software Acquisition Capability
                           Maturity ModelSM (SA-CMM). Capability Maturity ModelSM is a service mark of Carnegie Mellon
                           University, and CMM is registered in the U.S. Patent and Trademark Office.
                           5
                            Customs Service Modernization: Immature Software Development Processes Increase Customs
                           System Development Risks (GAO/AIMD-99-35, February 11, 1999).



                           Page 7                                       GAO/AIMD-99-41 Customs Service Modernization
                  Executive Summary




                  In addition to previous recommendations to improve Customs’
Recommendations   management of information technology,L6 GAO recommends that Customs
                  correct the management and technical weaknesses discussed in this report
                  before building ACE. To accomplish this, GAO recommends that the
                  Commissioner of Customs, with the support of Customs’ CIO, ensure that
                  Customs

                  (1) rigorously analyze alternative approaches to building ACE, including
                  ITDS as an alternative to developing ACE entirely within Customs;


                  (2) make investment decisions incrementally, i.e., for each increment:

                  •   use disciplined processes to prepare a rigorous life-cycle cost estimate,
                      including an explicit discussion of its inherent uncertainty;
                  •   prepare realistic and supportable benefit expectations;
                  •   require a favorable return-on-investment and compliance with Customs’
                      architecture before making any investment; and
                  •   validate actual costs and benefits once an increment is piloted, compare
                      these with estimates, use the results in making further decisions on
                      subsequent increments, and report the results to Customs’ House and
                      Senate appropriations and authorizing committees; and

                  (3) strengthen ACE software acquisition management by:

                  •   establishing an effective process improvement program and correcting
                      the weaknesses in ACE software acquisition processes identified in this
                      report, thereby bringing ACE processes to at least SEI level 2 and
                  •   requiring at least SEI level 2 processes of all ACE software contractors.


                  In its written comments on a draft of this report, Customs agreed with
Agency Comments   GAO’s conclusions and recommendations and stated that it is committed to
                  addressing the problems discussed in the report. To this end, Customs
                  cited a number of actions that it has underway and planned over the next
                  few years to improve management of information technology in general,
                  and ACE in specific. To fully correct the management and technical
                  weaknesses discussed in this report, Customs must follow through and
                  effectively implement actions to address all of GAO’s recommendations.
                  Customs’ comments are discussed in greater detail in chapter 5 and the
                  full text of its comments are reproduced in appendix I of this report.


                  6
                   GAO/AIMD-99-35, February 11, 1999 and GAO/AIMD-98-70, May 5, 1998.



                  Page 8                                      GAO/AIMD-99-41 Customs Service Modernization
Page 9   GAO/AIMD-99-41 Customs Service Modernization
Contents



Executive Summary                                                                                   2


Chapter 1                                                                                          14
                        Customs’ Core Business Operations: A Brief Description                     14
Introduction            Customs Is Legislatively Mandated to Modernize Operations and              16
                          Systems
                        ACE Purpose, Plans, and Status                                             16
                        Objectives, Scope, and Methodology                                         18


Chapter 2                                                                                          23
                        A Framework for Effective Systems Architectures                            23
ACE Is Being            Customs Is Developing, but Has Not Yet Completed, Its Target               24
Developed Without a       Systems Architecture
                        Customs Has Recently Announced Process for Ensuring                        25
Complete Enterprise       Architectural Compliance
Systems Architecture    Conclusions                                                                26


Chapter 3                                                                                          27
                        ACE Alternatives Were Not Considered                                       27
Customs’ ACE            Customs’ $1.05 Billion ACE Cost Estimate Is Not Reliable                   28
Investment              Customs’ Analysis of ACE Costs and Benefits Is Also Unreliable             31
                        Customs Is Not Investing Incrementally in ACE                              32
Management              Conclusions                                                                33
Practices Are
Ineffective
Chapter 4                                                                                          34
                        SEI’s Software Capability Models and Method: A Brief                       34
ACE Software              Description
Development and         Customs Lacks the Capability to Develop ACE Software                       37
                          Effectively
Acquisition Processes   Customs Lacks the Capability to Acquire ACE Software                       42
Are Not Effective         Effectively
                        Conclusions                                                                47




                        Page 10                           GAO/AIMD-99-41 Customs Service Modernization
                       Contents




Chapter 5                                                                                          48
                       Recommendations                                                             48
Overall Conclusions,   Agency Comments                                                             49
Recommendations,
and Agency
Comments
Appendixes             Appendix I: Comments From the U.S. Customs Service                          52
                       Appendix II: Detailed Comparison of SEI’s Checklist for Reliable            56
                         Cost Estimates and Customs’ Practices for Estimating ACE Costs
                       Appendix III: Results of Software Development Capability                    64
                         Maturity Model (SW-CMM) Evaluation for NCAP 0.1
                       Appendix IV: Results of Software Acquisition Capability Maturity            72
                         Model (SA-CMM) Evaluation for NCAP 0.2
                       Appendix V: Major Contributors to This Report                               81


Tables                 Table 3.1: Summary of ACE Project Cost Estimate’s Satisfaction              30
                         of SEI’s Checklist Criteria
                       Table 4.1: Description of SW-CMM                                            35
                         Level 2 KPAs
                       Table 4.2: Description of SA-CMM Level 2 KPA’s and the Risk                 36
                         Management Level 3 KPA
                       Table II.1: Are the Objectives of the Estimate Clear and Correct?           56
                       Table II.2: Has the Task Been Appropriately Sized?                          57
                       Table II.3: Is the Estimated Cost Consistent With Demonstrated              58
                         Accomplishments on Other Projects?
                       Table II.4: Have the Factors That Affect the Estimate Been                  59
                         Identified and Explained?
                       Table II.5: Have Steps Been Taken to Ensure the Integrity of the            60
                         Estimating Process?
                       Table II.6: Is the Organization’s Historical Evidence Capable of            61
                         Supporting a Reliable Estimate?
                       Table II.7: Has the Situation Remained Unchanged Since the                  63
                         Estimate Was Prepared?
                       Table III.1: Key Process Area: Requirements Management                      64
                       Table III.2: Key Process Area: Software Project Planning                    65
                       Table III.3: Key Process Area: Software Project Tracking and                67
                         Oversight
                       Table III.4: Key Process Area: Software Quality Assurance                   69




                       Page 11                            GAO/AIMD-99-41 Customs Service Modernization
         Contents




         Table III.5: Key Process Area: Software Configuration                      70
           Management
         Table IV.1: Key Process Area: Software Acquisition Planning                72
         Table IV.2: Key Process Area: Solicitation                                 73
         Table IV.3: Key Process Area: Requirements Development and                 74
           Management
         Table IV.4: Key Process Area: Project Management                           75
         Table IV.5: Key Process Area: Contract Tracking and Oversight              76
           Findings
         Table IV.6: Key Process Area: Evaluation                                   77
         Table IV.7: Key Process Area: Transition to Support                        79
         Table IV.8: Key Process Area: Acquisition Risk Management                  80


Figure   Figure 1.1: Planned Versus Actual NCAP Deployment Schedules                18




         Abbreviations

         ACE          Automated Commercial Environment
         ACS          Automated Commercial System
         CIO          chief information officer
         CMM         Capability Maturity ModelSM
         IRB          Investment Review Board
         IT           information technology
         ITDS         International Trade Data System
         KPA          key process area
         NAFTA        North American Free Trade Agreement
         NCAP         National Customs Automation Program
         OMB          Office of Management and Budget
         SA-CMM      Software Acquisition Capability Maturity ModelSM
         SCE          software capability evaluation
         SEI          Software Engineering Institute
         SW-CMM      Software Development Capability Maturity ModelSM


         Page 12                           GAO/AIMD-99-41 Customs Service Modernization
Page 13   GAO/AIMD-99-41 Customs Service Modernization
Chapter 1

Introduction


                           The mission of the Customs Service is to ensure that all goods and persons
                           entering and exiting the United States do so in compliance with all U.S.
                           laws and regulations. It does this by (1) enforcing the laws governing the
                           flow of goods and persons across U.S. borders and (2) assessing and
                           collecting duties, taxes, and fees on imported merchandise. During fiscal
                           year 1997, Customs collected $22.1 billion in revenue1 at more than 300
                           ports of entry, and it processed nearly 450 million passengers entering the
                           United States.


                           To accomplish its mission, Customs is organized into the following six
Customs’ Core              business areas: trade compliance, outbound, passenger, finance, human
Business Operations:       resources, and investigations. Each business area is described below.
A Brief Description
                       •   Trade compliance includes enforcement of laws and regulations
                           associated with the importation of goods into the United States. To do so,
                           Customs (1) works with the trade community to promote understanding of
                           applicable laws and regulations, (2) selectively examines cargo to ensure
                           that only eligible goods enter the country, (3) reviews documentation
                           associated with cargo entries to ensure that they are properly valued and
                           classified, (4) collects billions of dollars annually in duties, taxes, and fees
                           associated with imported cargo, (5) assesses fines and penalties for
                           noncompliance with trade laws and regulation, (6) seizes and accounts for
                           illegal cargo, and (7) manages the collection of these moneys to ensure
                           that all trade-related debts due to Customs are paid and properly
                           accounted for.
                       •   Outbound includes Customs enforcement of laws and regulations
                           associated with the movement of merchandise and conveyances from the
                           United States. To do so, Customs (1) selectively inspects cargo at U.S.
                           ports to guard against the exportation of illegal goods, such as protected
                           technologies, stolen vehicles, and illegal currency, (2) collects,
                           disseminates, and uses intelligence to identify high-risk cargo and
                           passengers, (3) assesses and collects fines and penalties associated with
                           the exportation of illegal cargo, and (4) physically examines baggage and
                           cargo at airport facilities for explosive and nuclear materials. In addition,
                           the outbound business includes collecting and disseminating trade data
                           within the federal government. Accurate trade data are crucial to
                           establishing accurate trade statistics on which to base trade policy
                           decisions and negotiate trade agreements with other countries. By the year
                           2000, Customs estimates that exports will be valued at $1.2 trillion, as
                           compared to a reported $696 billion in 1994.

                           1
                            Includes tariff duty, user fees, Internal Revenue Service excise taxes, and other assessments.



                           Page 14                                          GAO/AIMD-99-41 Customs Service Modernization
    Chapter 1
    Introduction




•   Passenger includes processing all passengers and crew of arriving and
    departing air, sea, and land conveyances and pedestrians. In fiscal year
    1997, Customs reported that it processed nearly 450 million travelers, and
    by the year 2000, expects almost 500 million passengers to arrive in the
    United States annually. Many of Customs’ passenger activities are
    coordinated with other federal agencies, such as the Immigration and
    Naturalization Service and the Department of Agriculture’s Animal and
    Plant Health Inspection Service. Activities include targeting high-risk
    passengers, which requires timely and accurate information, and
    physically inspecting selected passengers, baggage, and vehicles to
    determine compliance with laws and regulations.
•   Finance includes asset and revenue management activities. Asset
    management consists of activities to (1) formulate Customs’ budget,
    (2) properly allocate and distribute funds, and (3) acquire, manage, and
    account for personnel, goods, and services. Revenue management
    encompasses all Customs activities to identify and establish amounts
    owed Customs, collect these amounts, and accurately report the status of
    revenue from all sources. Sources of revenue include duties, fees, taxes,
    other user fees, and forfeited currency and property. The revenue
    management activities interrelate closely with the revenue collection
    activities in the trade compliance, outbound, and passenger business
    areas.
•   Human resources is responsible for filling positions, providing employee
    benefits and services, training employees, facilitating workforce
    effectiveness, and processing personnel actions for Customs’ 18,000
    employees and managers.
•   Investigations includes activities to detect and eliminate narcotics and
    money laundering operations. Customs works with other agencies and
    foreign governments to reduce drug-related activity by interdicting
    (seizing and destroying) narcotics, investigating organizations involved in
    drug smuggling, and deterring smuggling efforts through various other
    methods. Customs also develops and provides information to the trade
    and carrier communities to assist them in their efforts to prevent
    smuggling organizations from using cargo containers and commercial
    conveyances to introduce narcotics into the United States.

    To carry out its responsibilities, Customs relies on information systems
    and processes to assist its staff in (1) documenting, inspecting, and
    accounting for the movement and disposition of imported goods and
    (2) collecting and accounting for the related revenues. Customs expects its
    reliance on information systems to increase as a result of its burgeoning
    workload. For 1995 through 2001, Customs estimates that the annual



    Page 15                            GAO/AIMD-99-41 Customs Service Modernization
                       Chapter 1
                       Introduction




                       volume of import trade between the United States and other countries will
                       increase from $761 billion to $1.1 trillion. This will result in Customs
                       processing an estimated increase of 7.5 million import entries—from
                       13.1 million to 20.6 million annually—during the same period. Recent trade
                       agreements, such as the North American Free Trade Agreement (NAFTA),
                       have also increased the number and complexity of trade provisions that
                       Customs must enforce.


                       Both Customs and the Congress recognize that the ability to process the
Customs Is             growing volume of imports while improving compliance with trade laws
Legislatively          depends heavily on successfully modernizing Customs’ trade compliance
Mandated to            process and its supporting automated systems. To speed the processing of
                       imports and improve compliance with trade laws, the Congress enacted
Modernize Operations   Title VI of the North American Free Trade Agreement Implementation Act
and Systems            in December 1993.2 Customs refers to this legislation as the “Customs
                       Modernization and Informed Compliance Act” or “Mod” Act.

                       The primary purpose of the act is to streamline and automate Customs’
                       commercial operations by eliminating certain legislatively mandated paper
                       requirements and requiring Customs to establish the National Customs
                       Automation Program (NCAP). The legislation also specified certain
                       functions that NCAP must provide, including giving members of the trade
                       community the capability to electronically file import entries at remote
                       locations as well as enabling Customs to electronically process
                       “drawback” claims, which are refunds of duties and taxes paid on
                       imported goods that are subsequently exported or destroyed. According to
                       Customs, the act provides the legal framework to automate commercial
                       operations and thereby streamline the processing, and expedite the
                       movement of merchandise at more than 300 ports of entry.


                       ACE is intended to support the trade compliance business process,
ACE Purpose, Plans,    implement Mod Act requirements, and replace the existing import system,
and Status             the Automated Commercial System (ACS). ACS is nearly 15 years old and
                       Customs reports that it is becoming increasingly difficult and expensive to
                       operate, maintain, and enhance due to the system’s antiquated hardware
                       and software and limited processing capacity.

                       Currently, ACE’s architecture is to include both (1) mainframe-based,
                       centralized processing to support high-volume, repetitive transactions and

                       2
                        Public Law 103-182, 19 U.S.C. 1411 et seq.



                       Page 16                                       GAO/AIMD-99-41 Customs Service Modernization
Chapter 1
Introduction




(2) distributed, client-server processing and desktop PC interfaces to
support port office analysis and decision support needs. The
transaction-intensive applications are to run on IBM mainframes at
Customs’ national data center and are to be written in COBOL and C++.
The distributed processing is to occur on UNIX servers and the
applications are to be written in C++ and Java. To support this processing
environment, Customs plans to design and implement a single, integrated
agencywide database. It also plans to connect port offices to the national
data center through existing communications networks comprising the
Treasury Communications System, and to connect the trade community to
the data center through the Internet and a combination of dial-up lines and
dedicated lines.

According to Customs, ACE will facilitate increased compliance of
individuals, businesses, and governments with the trade laws and
regulations of the United States and increased communication with the
trade community. It will also provide an integrated, account-based,
automated information system for collecting, disseminating, and analyzing
trade-related data and ensuring the proper collection and allocation of
revenues.

Customs began developing ACE in 1994 and plans to develop and deploy
ACE in 21 increments from 1998 through 2005. Each increment consists of
the software and hardware necessary to perform a discrete portion of the
total ACE functionality. The first four increments, known collectively as the
NCAP prototype, will process pre-identified merchandise shipped into the
U.S. via truck by three selected importers.3 The first two increments, NCAP
0.1 and NCAP 0.2, were deployed for operational evaluation at three pilot
border port locations (Detroit, Michigan; Laredo, Texas; and Port Huron,
Michigan) during May 1998 and October 1998, respectively.

The succeeding 17 increments are intended to build upon the foundational
capabilities contained within the four NCAP prototype increments,
providing additional functionality including the capability to (1) process
merchandise imported via rail, air, sea, and couriers, (2) process
collections and refunds associated with imported cargo, (3) process
warehouse entry cargo, and (4) process cargo subject to drawbacks. Until
all of these deployments are completed, Customs intends to operate and
maintain ACS.



3
The importers participating in the NCAP prototype are General Motors Corporation, Ford Motor
Company, and Chrysler Corporation.



Page 17                                      GAO/AIMD-99-41 Customs Service Modernization
                                       Chapter 1
                                       Introduction




                                       Customs’ estimate of ACE costs has escalated since August 1997, when it
                                       initially estimated that ACE’s 10-year life-cycle cost would be $895 million
                                       for the period 1998 through 2007. In November 1997, Customs revised this
                                       10-year cost estimate to $987.6 million and also estimated that, over 15
                                       years, from 1994 through 2008, it would spend $1.05 billion on ACE. It then
                                       used the $1.05 billion estimate along with estimated expected savings of
                                       $1.9 billion to justify ACE. Customs is currently reevaluating this estimate.

                                       Figure 1.1 compares planned and actual deployments of the NCAP
                                       increments. As the figure illustrates, Customs is well over 2 years behind
                                       its original schedule for NCAP.



Figure 1.1: Planned Versus Actual NCAP Deployment Schedules




                                       Note: NCAP 0.3 and NCAP 0.4 current dates are unknown pending budget approval.




                                       The Chairman, Subcommittee on Treasury, Postal Service, and General
Objectives, Scope,                     Government, House Committee on Appropriations, and the Chairman,
and Methodology

                                       Page 18                                    GAO/AIMD-99-41 Customs Service Modernization
Chapter 1
Introduction




Subcommittee on Treasury, General Government, and Civil Service,
Senate Appropriations Committee, requested that we review ACE. Our
objective was to determine whether Customs is effectively managing ACE,
including whether Customs has adequately justified ACE cost-effectiveness.
In making these determinations, we relied primarily on the following
criteria: (1) the Clinger-Cohen Act of 19964 and other legislative reforms
that require federal agencies to develop and maintain integrated systems
architectures and improve their information technology investment
management processes, (2) Office of Management and Budget guidance
related to the acquisition and management of information resources,
(3) GAO and Treasury systems architecture guidance, and (4) related
Software Engineering Institute system engineering standards concerning
cost estimating and software development and acquisition maturity.

To address our objective, we reviewed pertinent documentation (e.g., ACE
project plan, system/software development process practices/standards,
ACE technical architecture descriptions, and ACE project status reports) and
interviewed responsible Treasury officials and Customs ACE project
officials in order to identify (1) Customs’ management structure for
developing and deploying ACE, (2) the ACE system/software development
methodology, (3) the planned ACE hardware/software and communications
architecture and configuration, and (4) the current ACE system
development status.

Additionally, we met with Customs ACE officials to determine the status of
the agency’s implementation of recommendations we made in May 1998 to
complete its systems architecture. These recommendations included
(1) ensuring that the architecture fully described Customs’ business
functions, (2) defining the information needs and flows among these
functions, (3) establishing the technical standards, products, and services
that will be used to build systems that support these business functions
and needs, and (4) enforcing compliance with the architecture.

Further, we obtained and reviewed supporting documentation and
interviewed Treasury officials, Customs ACE officials, and Customs
Investment Review Board (IRB) officials to (1) identify the current ACE


4
 Although the Clinger-Cohen Act (Public Law 104-106) was passed after Customs began developing
ACE, its principles are based on practices that are widely considered to be integral to successful IT
investments. For an analysis of the management practices of several leading private and public sector
organizations on which the Clinger-Cohen Act is based see Executive Guide: Improving Mission
Performance Through Strategic Information Management and Technology, (GAO/AIMD-94-115, May
1994). For an overview of the IT management process envisioned by Clinger-Cohen see Assessing Risk
and Returns: A Guide for Evaluating Federal Agencies’ IT Investment Decision-making
(GAO/AIMD-10.1.13, February 1997).



Page 19                                        GAO/AIMD-99-41 Customs Service Modernization
Chapter 1
Introduction




project cost and life-cycle cost estimate baselines and determine the
current actual expenditures to date, (2) identify the current ACE project
schedule baseline and determine the actual progress to date against
scheduled milestones, (3) define what ACE is intended to do and how it is
expected to benefit Customs’ mission, (4) determine the extent to which
ACE mission-related goals/benefits have been achieved to date and how
Customs is measuring the accrued benefits, and (5) determine Customs’
strategic approach to managing the development/acquisition, integration,
and deployment of ACE. The documentation we analyzed included the ACE
project plan and strategic plan, functional/performance requirements, and
NCAP technical assessment documents, budget/financial data, cost-benefit
policy and guidance, cost-benefit analyses, various life-cycle and project
cost estimates, project status reports, testing problem reports, and
configuration management documentation.

Also, we reviewed project documentation to determine how the life-cycle
cost baseline was estimated and how this estimating approach compared
to criteria established by SEI in A Manager’s Checklist for Validating
Software Cost and Schedule Estimates.5 The SEI criteria defines seven
primary questions, each supported by more detailed secondary questions,
that can be used to determine whether defined and disciplined processes
were used to derive a given cost estimate. We also interviewed ACE project
officials.

In addition, we assessed whether Customs thoroughly analyzed technical
alternatives to ACE including the possibility of (1) enhancing and
continuing to use the legacy trade system, ACS, (2) using different
architectural designs for ACE, (3) following different
development/acquisition strategies, and/or (4) using the Department of the
Treasury’s planned multiagency International Trade Data System (ITDS),
instead of ACE, for some functions, such as collecting, editing, and
validating trade data and collecting and accounting for trade revenue.

Last, we used the Software Engineering Institute’s (SEI’s) Software
Development Capability Maturity ModelSM (SW-CMM),6 Software
Acquisition Capability Maturity ModelSM (SA-CMM), and its Software
Capability Evaluation Method, to evaluate Customs’ ability to manage its
NCAP 0.1 software development project and NCAP 0.2 software acquisition
effort, respectively. These models and methods provide a logical and

5
 CMU/SEI-95-SR-004, January 1995.
6
  Capability Maturity ModelSM is a service mark of Carnegie Mellon University, and CMM is registered
in the U.S. Patent and Trademark Office.



Page 20                                         GAO/AIMD-99-41 Customs Service Modernization
Chapter 1
Introduction




widely accepted framework for baselining an organization’s current
process capabilities (i.e., strengths and weaknesses) and assessing
whether an organization has the necessary process discipline in place to
repeat earlier successes on similar projects. Organizations that do not
satisfy the requirements for the “repeatable” level are by default judged to
be at the “initial” level of maturity, meaning that their processes are ad
hoc, sometimes even chaotic, with few of the processes defined and
success dependent mainly on the heroic efforts of individuals.

In following the SEI method, GAO staff trained at SEI evaluated Customs’ ACE
project software development/maintenance maturity in five of the six key
process areas (KPA) that are necessary to attain a “repeatable” level of
process maturity.7 GAO did not evaluate Customs in the sixth repeatable
level KPA, software subcontract management, because Customs did not use
subcontractors on the ACE project. Additionally, GAO staff trained at SEI
evaluated Customs’ ACE project software acquisition maturity in the seven
KPAs that are necessary to attain a repeatable level of process capability,
and one KPA associated with the “defined” level of process
maturity—acquisition risk management.8 The purpose of acquisition risk
management is to formally identify risks as early as possible and adjust the
acquisition to mitigate those risks. Many software experts consider
acquisition risk management to be an integral part of the solicitation,


7
 The five KPAs are requirements management, software project planning, software project tracking
and oversight, software quality assurance, and software configuration management. According to the
SW-CMM, (1) requirements management is the process of establishing a common understanding between
the customer and the software developer of the customer’s requirements, (2) software project
planning is the process of establishing reasonable plans for engineering the software and managing the
software project, (3) software project tracking and oversight is the process of providing adequate
visibility into the software project’s progress to permit effective action when deviations from plans
occur, (4) software quality assurance is the process of verifying for management that software process
and product procedures and standards are being followed, and (5) software configuration management
is the process of establishing and maintaining the integrity of the software products throughout their
life-cycle.
8
 The seven KPA’s relating to the repeatable level are software acquisition planning, solicitation,
requirements development and management, project office management, contract tracking and
oversight, evaluation, and transition and support. The KPA relating to the defined level is acquisition
risk management. According to the SA-CMM, (1) software acquisition planning is the process of ensuring
that reasonable planning for all elements of the software acquisition occur, (2) solicitation is the
process of ensuring that award is made to the contractor most capable of satisfying the specified
requirements, (3) requirements development and management is the process of establishing an
unambiguous and agreed upon set of software requirements, (4) project office management is the
process of effective and efficient management of project office activities, (5) contract tracking and
oversight is the process of ensuring that contractor activities, products, and services satisfy contract
requirements, (6) evaluation is the process of determining that acquired software products and
services satisfy contract requirements prior to acceptance, (7) transition and support is the process of
transferring acquired software products to the eventual support organization, and (8) acquisition risk
management is the process of identifying software risks early and adjusting the acquisition strategy to
mitigate those risks.



Page 21                                         GAO/AIMD-99-41 Customs Service Modernization
Chapter 1
Introduction




project performance management, and contract performance management
processes.

Customs provided written comments on a draft of this report. These
comments are presented in chapter 5, and are reprinted in appendix I. We
performed our work at Customs and Department of the Treasury
headquarters offices in Washington, D.C., and at the Customs Data Center
facility in Newington, Virginia between February 1998 and November 1998,
in accordance with generally accepted government auditing standards.




Page 22                          GAO/AIMD-99-41 Customs Service Modernization
Chapter 2

ACE Is Being Developed Without a
Complete Enterprise Systems Architecture

                    Architectures are critical for designing and developing large and complex
                    information systems. These comprehensive “construction plans”
                    systematically and completely describe an organization’s target business
                    environment, both in logical (e.g., missions, business functions,
                    information flows) terms and technical (e.g., software, hardware,
                    communications) terms. Without a target architecture to guide and
                    constrain IT investment, there is no systematic way to preclude either
                    inconsistent system design and development decisions or the resulting
                    suboptimal performance and added cost associated with incompatible
                    systems.

                    The Clinger-Cohen Act of 1996 requires agency CIOs to develop and
                    maintain an integrated system architecture. In addition, OMB issued
                    guidance in 1996 that, among other things, requires agency IT investments
                    to be consistent with federal, agency, and bureau architectures.1

                    Customs does not currently have a complete target architecture but has
                    recently established a process for enforcing an architecture once one is
                    completed. Customs has plans for completing its target architecture by
                    May 1999 and ensuring its enforcement. Thus far, however, it has only
                    defined its current architectural environment.


                    Reflecting the general consensus in the industry that large, complex
A Framework for     systems development and acquisition efforts should be guided by explicit
Effective Systems   architectures, we issued a report in 1992 defining a comprehensive
Architectures       framework for designing and developing systems architectures.2 This
                    framework, which is consistent with guidance that Treasury has provided
                    to its bureaus,3 divides systems architectures into (1) a logical or business
                    component, which is developed first, and (2) a technical or systems
                    component, which is based on the first component.

                    The logical component ensures that the systems meet the business needs
                    of the organization. It provides a high-level description of the
                    organization’s mission and target concept of operations; the business
                    functions being performed and the relationships among functions; the

                    1
                     OMB Memorandum M-97-02, Funding Information Systems Investments, October 25, 1996.
                    2
                     Strategic Information Planning: Framework for Designing and Developing System Architectures
                    (GAO/IMTEC-92-51, June 1992).
                    3
                      This guidance—Treasury Information Systems Architecture Framework (TISAF) version 1.0,
                    January 3, 1997—is also included in OMB’s guidance on developing system architecture. See, OMB
                    Memorandum M-97-16, Information Technology Architectures, June 18, 1997.



                    Page 23                                       GAO/AIMD-99-41 Customs Service Modernization
                      Chapter 2
                      ACE Is Being Developed Without a
                      Complete Enterprise Systems Architecture




                      information needed to perform the functions; the users and locations of
                      the functions and information; and the information systems needed to
                      support the agency’s business needs.

                      The technical component ensures that systems are interoperable, function
                      together efficiently, and are cost-effective over their life-cycles (including
                      maintenance costs). The technical component details specific standards
                      and approaches that will be used to build systems, including hardware,
                      software, communications, data management, security, and performance
                      characteristics.


                      Customs does not currently have a complete enterprise systems
Customs Is            architecture. In May 1998, we reported that for five of its six business
Developing, but Has   areas (outbound, passenger, finance, human resources, and
Not Yet Completed,    investigations), Customs’ architecture does not (1) describe all of the
                      agency’s business functions, (2) completely identify the users and
Its Target Systems    locations of the functions, or (3) define the information needed to perform
Architecture          the functions.4 Further, while the architecture and related documentation
                      describe business functions and users and work locations for the sixth
                      business area (trade compliance), they do not identify all of the
                      information needs and flows for all of the trade functions. Nonetheless,
                      Customs had defined many characteristics of its systems’ hardware,
                      software, communications, data management, and security components.
                      Because these characteristics were not based on a complete
                      understanding of its enterprisewide functional and information needs, as
                      specified in both best practice and Treasury guidance, we concluded that
                      Customs did not have adequate assurance that its information systems will
                      optimally support its needs across all business areas.

                      We recommended that the Commissioner of Customs direct the Customs
                      CIO, in consultation with the Treasury CIO, to complete the architecture.
                      Specifically, we recommended that, at a minimum, the architecture should
                      (1) describe Customs’ target business operations, (2) fully define Customs’
                      interrelated business functions to support these target operations,
                      (3) clearly describe information needs and flows among these functions,
                      (4) identify the systems that will provide these functions and support these
                      information needs and flows, and (5) use this information to specify the
                      technical standards and related characteristics that these systems should
                      possess to ensure that they interoperate, function together efficiently, and
                      are cost-effective to maintain.

                      4
                       GAO/AIMD-98-70, May 5, 1998.



                      Page 24                                    GAO/AIMD-99-41 Customs Service Modernization
                       Chapter 2
                       ACE Is Being Developed Without a
                       Complete Enterprise Systems Architecture




                       Customs and Treasury officials have acknowledged the limitations in
                       Customs’ architecture, and agreed to define the agency’s logical
                       architecture (e.g., functions, information needs and flows and users)
                       across its six business areas and, in light of this definition, reevaluate the
                       technical characteristics it has specified for its technical architecture (i.e.,
                       systems environment). Thus far, Customs has defined its current (i.e., its
                       “as-is” or “in-state”) architectural environment, including its current
                       business operations, its current supporting business functions and their
                       information needs and flows, the systems currently supporting these
                       functions, and the technical characteristics (e.g., hardware, software, and
                       communications) of these supporting systems. However, Customs has not
                       defined an architecture for its target (i.e., future) business and systems
                       environment. Customs originally planned to have completed its target
                       architecture by September 1998, but that date is now May 1999.


                       In May 1998, we reported that Customs’ investment management process5
Customs Has Recently   did not ensure that all new systems would conform to the architecture. In
Announced Process      particular, we reported that Customs’ investment review board (IRB) used
for Ensuring           four criteria in scoring competing investment options and allocating
                       funding among them. The four criteria were
Architectural
Compliance             (1) risk (e.g., technical, schedule, and cost);

                       (2) strategic alignment (e.g., cross-functional benefits, linkage to Customs’
                       business plan, and compliance with legislative mandates);

                       (3) mission effectiveness (e.g., contributions to service delivery); and

                       (4) cost-benefit ratio (e.g., tangible and intangible benefits, and costs).

                       Because compliance with the architecture was considered under the risk
                       criterion but was not required, the process did not preclude funding
                       projects that were inconsistent with the enterprise architecture. Moreover,
                       the process did not require that such deviations from the architecture be
                       rigorously justified.

                       To ensure that Customs effectively enforced its information systems
                       architecture, we recommended that Customs require that all new projects
                       comply with the architecture unless an exception could be justified by


                       5
                        IT Investment Management Process, U.S. Customs Service, August 28, 1997.



                       Page 25                                       GAO/AIMD-99-41 Customs Service Modernization
              Chapter 2
              ACE Is Being Developed Without a
              Complete Enterprise Systems Architecture




              careful, thorough, and documented analysis.6 Customs agreed and, in
              January 1999, changed its investment management process to explicitly
              require that proposed IT investments comply with its architecture, unless
              an exception is justified and a waiver is granted by the technical review
              committee.


              Until Customs completes and enforces its enterprise systems architecture,
Conclusions   it will not have adequate assurance that ACE and other systems it plans to
              build and operationally deploy (1) will effectively meet the agency’s
              business needs and (2) are compatible, efficient, and cost-effective to
              develop, integrate, and maintain.




              6
               GAO/AIMD-98-70, May 5, 1998.



              Page 26                                    GAO/AIMD-99-41 Customs Service Modernization
Chapter 3

Customs’ ACE Investment Management
Practices Are Ineffective

                      The Clinger-Cohen Act and OMB guidance provide an effective framework
                      for IT investment management. Together, they set requirements for
                      (1) identifying all potential alternative system solutions, (2) developing
                      reliable estimates of project life-cycle costs and benefits, and investing in
                      the most cost-beneficial alternative, and (3) to the maximum extent
                      practical, structuring major projects into a series of increments to ensure
                      that each increment constitutes a wise investment.

                      Customs has not effectively implemented any of these investment
                      management practices on ACE. Specifically, (1) Customs’ investment
                      analysis did not address alternatives to its chosen ACE system solution,
                      (2) Customs’ did not use rigorous cost estimating techniques in preparing
                      its ACE cost estimate and its cost-benefit analysis for ACE omitted
                      substantial costs and inflated benefits, and (3) Customs is not justifying
                      and validating the costs and benefits for each ACE increment. As a result,
                      Customs lacks a sound basis for making ACE investment decisions.


                      Before embarking on a major, costly systems initiative such as ACE,
ACE Alternatives      agencies should thoroughly assess the full range of technical options. In
Were Not Considered   the case of ACE, Customs had several alternatives to satisfying its mission
                      needs as specified in the Mod Act. For example, it could enhance ACS, use
                      different architectural designs for ACE, follow different development/
                      acquisition strategies for ACE, and/or use Treasury’s planned
                      governmentwide trade system, International Trade Data System (ITDS), to
                      supplement some ACE functions, such as collecting and disseminating
                      trade data. ITDS was initiated in 1995 as a project to implement the National
                      Performance Review recommendation to develop a coordinated,
                      governmentwide system for the collection, use, and dissemination of trade
                      data. At that time, a multiagency board of directors was established,
                      headed by the Department of the Treasury with Customs and other major
                      trade-related agencies represented. The Department of the Treasury is
                      responsible for designing and developing ITDS, which is expected to reduce
                      the burden that federal agencies place on the international trading
                      organizations by requiring that they respond to duplicative data requests.
                      Treasury intends for the system to serve as the single point for collecting,
                      editing, and validating trade data as well as collecting and accounting for
                      trade revenue.1



                      1
                       Treasury has developed an ITDS project plan, system specification, cost-benefit analysis, and related
                      documentation. It plans to begin developing ITDS in 1999 and to fully implement it by 2003. It
                      estimates that ITDS will cost $256 million to develop, deploy, and maintain through 2005.



                      Page 27                                         GAO/AIMD-99-41 Customs Service Modernization
                         Chapter 3
                         Customs’ ACE Investment Management
                         Practices Are Ineffective




                         By thoroughly considering these and other choices, Customs would have
                         ensured that the most cost-effective and beneficial alternative was chosen
                         before deciding to invest $1.05 billion in ACE. In fact, OMB requires that
                         agencies consider alternative system solutions to meet mission needs,
                         including different system architectures, upgrading existing systems, or
                         contracting for development and integration of major systems.

                         Customs did not identify and evaluate a full range of alternatives to ACE. In
                         fact, Customs considered only (1) enhancing and operating ACS to provide
                         the same functionality of ACE, (2) operating ACS without any enhancement,
                         (3) operating ACS with limited enhancements, and (4) developing and
                         deploying part of ACE’s planned functionality. Customs discarded the last
                         three because none provided for meeting all of the Mod Act requirements.
                         With respect to the first, Customs compared ACS to ACE and decided that
                         ACE was the more cost-effective alternative.


                         Customs did not evaluate other alternatives to its ACE solution, including
                         (1) adopting a different ACE architectural design (e.g., using a combination
                         mainframe and client-server configuration instead of a mainframe only
                         system), (2) contracting out for ACE development rather than developing
                         ACE in-house, or (3) using ITDS to satisfy part of its functional needs. As a
                         result, Customs committed to and began investing in ACE development
                         without knowing whether it had chosen the most cost-effective alternative.

                         Since making its decision to develop ACE, Customs has changed ACE’s
                         architectural design, and it has decided to acquire the second ACE
                         increment rather than develop it in-house. However, these decisions are
                         not supported by any verifiable alternatives analysis, meaning that
                         Customs still lacks adequate assurance that it is following the most
                         cost-effective approach to satisfying its mission needs.


                         Reliable estimates of IT projects’ expected costs are essential to
Customs’ $1.05 Billion   determining a project’s cost-effectiveness. Without such information, the
ACE Cost Estimate Is     likelihood of poor investment decisions is increased, not only when a
Not Reliable             project is initiated but also throughout its life-cycle.

                         To assist management in assessing the credibility of a given project’s cost
                         estimate, SEI developed the following seven questions for decisionmakers




                         Page 28                              GAO/AIMD-99-41 Customs Service Modernization
Chapter 3
Customs’ ACE Investment Management
Practices Are Ineffective




to use to assess the reliability of a project’s cost estimate and detailed
criteria to assist in evaluating how well a project satisfies each question.2

(1) Are the objectives of the estimate clear and correct?

(2) Has the task been appropriately sized?

(3) Is the estimated cost consistent with demonstrated accomplishments
on other projects?

(4) Have the factors that affect the estimate been identified and explained?

(5) Have steps been taken to ensure the integrity of the estimating
process?

(6) Is the organization’s historical evidence capable of supporting a
reliable estimate?

(7) Has the situation remained unchanged since the estimate was
prepared?

We analyzed the approach that Customs followed in deriving its
$1.05 billion ACE life-cycle cost estimate using SEI’s criteria. Among these
criteria were several very significant and closely intertwined requirements
that are at the core of effective cost estimating. Specifically, embedded in
several of the aforementioned seven questions are requirements for using
(1) formal cost models, (2) structured and documented processes for
determining the software size and reuse inputs to the models, and
(3) relevant, measured, and normalized historical cost data (estimated and
actual) to calibrate the models.3

We found that Customs did not satisfy any of these requirements. In
particular, Customs did not use a cost model. Instead, it used an
unsophisticated spreadsheet to extrapolate the cost of each ACE increment.
Further, Customs’ approach to determining software size and reuse was
not documented and was not well supported or convincing. Customs
estimated the size of each ACE software increment (most of which were
still undefined) by extrapolating from the estimated size of the first


2
 A Manager’s Checklist for Validating Software Cost and Schedule Estimates (CMU/SEI-95-SR-004,
January 1995).
3
 Examples of such data are the productivity value of a “staff-month,” that is, the time and cost to
produce a specified unit of code, such as a line of code.



Page 29                                          GAO/AIMD-99-41 Customs Service Modernization
                                        Chapter 3
                                        Customs’ ACE Investment Management
                                        Practices Are Ineffective




                                        increment based on individuals’ undocumented best judgments about
                                        increment functionality and complexity. Last, Customs did not have any
                                        historical project cost data at the time it developed the $1.05 billion
                                        estimate, and it did not account for relevant, measured, and normalized
                                        differences among the increments. For example, it did not account for
                                        (1) the change in ACE’s architecture from a mainframe system written in
                                        COBOL and C++ to a combined mainframe and Internet-based system
                                        written in C++ and Java and (2) the change in ACE from an in-house
                                        software development project (NCAP 0.1) to a software acquisition (NCAP
                                        0.2). Clearly, such fundamental changes can have a dramatic effect on
                                        system costs, and should have been addressed explicitly in Customs’ cost
                                        estimates.

                                        Table 3.1 summarizes the complete results of our assessment of Customs’
                                        cost estimating process. For each of SEI’s questions and supporting
                                        criteria, Customs was rated as demonstrating a strength, i.e., effectively
                                        implementing the criterion; a weakness, i.e., ineffectively implementing
                                        the criterion; or, where evidence was inconclusive and could not support
                                        characterization as either a strength or a weakness, an observation was
                                        noted. (See appendix II for further detail on SEI’s criteria and our findings).

Table 3.1: Summary of ACE Project
Cost Estimate’s Satisfaction of SEI’s                                                  Number of     Number of       Number of
Checklist Criteria                      SEI checklist questions                         strengths   weaknesses     observations
                                        Are the objectives of the estimate clear                4              0              0
                                        and correct?
                                        Has the task been appropriately sized?                  0              7              1
                                        Is the estimated cost consistent with                   1             10              0
                                        demonstrated accomplishments on
                                        other projects?
                                        Have the factors that affect the estimate               3              2              0
                                        been identified and explained?
                                        Have steps been taken to ensure the                     3              1              3
                                        integrity of the estimating process?
                                        Is the organization’s historical evidence               2              8              0
                                        capable of supporting a reliable
                                        estimate?
                                        Has the situation remained unchanged                    0              3              0
                                        since the estimate was prepared?
                                        Totals                                                 13             31              4




                                        Page 30                                     GAO/AIMD-99-41 Customs Service Modernization
                             Chapter 3
                             Customs’ ACE Investment Management
                             Practices Are Ineffective




Customs’ Cost Estimate       Software and systems development experts agree that early project
Implies an Unjustified       estimates are by definition imprecise, and that this inherent imprecision
Level of Precision           decreases during the project’s life-cycle as more information becomes
                             known about the system. These experts emphasize that, to be useful, each
                             cost estimate should include an indication of its degree of uncertainty,
                             possibly as an estimated range or qualified by some factor of confidence.
                             For example, a cost estimate of $1 million could be presented as a range
                             from $750,000 to $1.25 million or as $1 million with a confidence level of
                             90 percent, indicating that there is a 10 percent chance that costs will
                             exceed this estimate.

                             Customs did not reveal the degree of uncertainty of its cost estimate for
                             ACE to managers involved in investment decisions. For example, Customs
                             did not disclose that it made the estimate before fully defining ACE
                             functionality. Instead, Customs presented its $1.05 billion ACE life-cycle
                             cost estimate as an unqualified point estimate. This suggests an element of
                             precision that cannot exist for such an undefined system and obscures the
                             investment risk remaining in this project.


                             ACE  cost estimates are understated, benefit estimates are overstated, and
Customs’ Analysis of         both are unreliable. Customs’ August 1997 cost-benefit analysis estimated
ACE Costs and                that ACE would produce cumulative savings of $1.9 billion over a 10-year
Benefits Is Also             life-cycle beginning in fiscal year 1998 and ending in fiscal year 2007.
                             However, this analysis was unreliable because certain benefits and costs
Unreliable                   were unsupported, other benefits were overstated, and relevant costs were
                             omitted. For example:

                         •   The analysis identified $650 million (35 percent of the total ACE estimated
                             savings) in savings to the trade community from more accurate and timely
                             data, improved customer compliance, reduced support staff, reduced
                             transportation costs, elimination of duplicate recordkeeping systems, and
                             lower processing costs associated with periodic payments. However,
                             Customs could not produce any verifiable data or analysis to support this
                             claim. According to Customs officials, this benefit amount was projected
                             on the basis of undocumented information supplied by one company. The
                             officials stated that other companies considered such data confidential
                             and would not provide them to Customs.
                         •   The analysis identified an additional $644 million (33 percent of the total
                             savings) resulting from increased productivity. Because this estimate is
                             driven by Customs’ assumption that every minute “saved” by processing
                             transactions or analyzing data faster using ACE, rather than ACS, would be



                             Page 31                              GAO/AIMD-99-41 Customs Service Modernization
                           Chapter 3
                           Customs’ ACE Investment Management
                           Practices Are Ineffective




                           productively used by all workers, it can be viewed as a “best case,” upper
                           limit on estimated productivity improvements. Given the magnitude of the
                           potential savings, even a small change in the assumption translates into a
                           large reduction in benefits. For example, conservatively assuming that
                           three-fourths of each minute saved would be productively utilized by
                           three-fourths of all workers, the expected benefits would be reduced by
                           about $282 million.
                       •   The analysis excluded costs for hardware and system software upgrades
                           (e.g., desktop workstations and operating systems, application and data
                           servers, data base management systems) at each port office. Using
                           Customs’ estimate for acquiring the initial suite of port office hardware
                           and system software, and assuming a technology refreshment cycle of
                           every 3 to 5 years, we estimated this cost to be between $72.9 million and
                           $171.8 million.
                       •   The analysis excluded $52.8 million of costs needed to support the data
                           center and maintain ACS through fiscal year 2002. Since Customs intends to
                           operate and maintain ACS in parallel with ACE through 2002, these costs
                           should be included.
                       •   The analysis excluded costs to conduct security analysis, project planning
                           and management, and independent verification and validation. Customs
                           estimates that these costs collectively total $23 million.
                       •   The analysis excluded other relevant cost items, such as the cost of
                           defining ACE requirements, integrating ACE components, and conducting
                           ACE regression testing and training. Customs did not have an estimated
                           value for these costs.
                       •   The analysis included annual telecommunications costs of $60.3 million
                           once ACE is deployed to all sites. However, Customs could not provide us
                           with any supporting data or analysis for this estimate.


                           Incremental project management involves three fundamental components:
Customs Is Not             (1) developing/acquiring a large system in a series of smaller projects or
Investing                  system increments, (2) individually justifying investment in each separate
Incrementally in ACE       increment on the basis of return-on-investment, and (3) monitoring actual
                           benefits achieved and costs incurred on completed increments and
                           modifying subsequent increments/investments to reflect lessons learned.
                           By doing so, agencies avoid discovering too late that their system is not
                           cost-beneficial, and they can reduce the enormous risks associated with
                           large, expensive projects. The Clinger-Cohen Act requires that agencies
                           acquire information technology incrementally, to the maximum extent
                           practicable, and have milestones for senior managers to obtain
                           information on the cost, timeliness, quality, and capability of information



                           Page 32                              GAO/AIMD-99-41 Customs Service Modernization
              Chapter 3
              Customs’ ACE Investment Management
              Practices Are Ineffective




              system investments to meet requirements. Additionally, OMB policy
              requires that investments in major information systems be implemented in
              increments with each increment delivering measureable benefits.4

              Customs is developing and acquiring ACE in a series of 21 increments. It
              has, to date, defined the functionality of only the first two increments and
              will define later increments in the future. Nonetheless Customs has
              estimated costs and benefits for, and has committed to, investing in the
              entire system (i.e., all 21 increments). It has not estimated the costs and
              benefits for each increment and does not know whether each increment
              will produce a reasonable return on investment. Furthermore, once it has
              deployed an increment at a pilot site for evaluation, Customs is not
              validating that estimated benefits were actually achieved. As a result,
              Customs will not even know whether the first ACE increment, now being
              piloted at three sites, is producing expected benefits or is cost-effective.
              Instead, Customs will only determine whether the first increment is
              performing at a level “equal to or better than” ACS.

              Customs “grand design” approach to ACE does not constitute wise
              investment management. Customs will not know whether earlier
              increments were cost beneficial before it invests in later increments; it
              does not have reliable cost or benefit data upon which to base investment
              decisions; and it is committing substantial resources with no assurance
              that it will achieve a reasonable return-on-investment.


              Because Customs does not have reliable information on ACE costs and
Conclusions   benefits and has not analyzed other viable alternatives to the system
              solution it selected, it does not have adequate assurance that ACE is the
              optimal approach. In fact, it has no assurance at all that ACE is cost
              effective. Furthermore, because it is not justifying the return on its
              investment in each ACE increment, Customs will not be able to
              demonstrate whether ACE is cost-effective until it has already spent
              hundreds of millions of dollars to develop/acquire the entire system.




              4
               Memorandum For Heads Of Executive Department And Agencies, October 25, 1996 (M-97-02).



              Page 33                                     GAO/AIMD-99-41 Customs Service Modernization
Chapter 4

ACE Software Development and Acquisition
Processes Are Not Effective

                        The Clinger-Cohen Act requires agency CIOs to establish effective IT
                        management processes, such as processes for developing or acquiring
                        software. Customs has developed the first ACE software increment (NCAP
                        0.1) in-house and is acquiring the second increment (NCAP 0.2) from a
                        contractor. Customs has not yet decided whether it will develop or acquire
                        future ACE software increments. If it is to do either effectively, Customs
                        must use mature software processes.

                        SEI, recognized for its expertise in software processes, has developed
                        models and methods that define and determine, respectively, software
                        development and software acquisition process maturity. We evaluated
                        both ACE software development and ACE software acquisition processes
                        using SEI’s software development capability maturity model (SW-CMM) and
                        software acquisition capability maturity model (SA-CMM), respectively, and
                        SEI’s software capability evaluation (SCE) method. Our evaluations focused
                        on the key process areas (KPA) necessary to obtain a “repeatable” level of
                        maturity, the second level of SEI’s five-level process maturity models.
                        Organizations that do not satisfy these second-level KPA requirements are
                        by default at the “initial” or first level, meaning that their processes are ad
                        hoc, sometimes even chaotic.

                        Our evaluations found that Customs lacks the capability to either develop
                        or acquire ACE software effectively, and that it lacks a software process
                        improvement program.1 Because of the number and severity of the KPA
                        weaknesses found, Customs did not achieve the “repeatable” level of
                        process maturity as either a software developer or acquirer, meaning that
                        any attempts to do so are likely to produce software that does not perform
                        as intended, costs more than expected, and is delivered late. Further,
                        without a software process improvement program, it is unlikely that
                        Customs can effectively address its software process weaknesses.


                        The SW-CMM and the SA-CMM rank organizational maturity according to five
SEI’s Software          levels. Maturity levels 2 through 5 require verifiable existence and use of
Capability Models and   certain key process areas, known as KPAs. The SW-CMM includes six level 2
Method: A Brief         KPAs. We evaluated Customs’ against five of the six. The sixth level 2 KPA,
                        software subcontract management, was not evaluated because Customs
Description             did not use subcontractors on NCAP 0.1 (see table 4.1 for a description of
                        each KPA).

                        1
                         The results of our evaluation of ACE software development maturity and Customs’ software process
                        improvement efforts were first published in our report on Customs-wide software development
                        capability (Customs Service Modernization: Immature Software Development Processes Increase
                        Customs Systems Development Risks (GAO/AIMD-99-35, February 11, 1999)).



                        Page 34                                       GAO/AIMD-99-41 Customs Service Modernization
                                   Chapter 4
                                   ACE Software Development and Acquisition
                                   Processes Are Not Effective




Table 4.1: Description of SW-CMM
Level 2 KPAs                       SW-CMM Level 2 KPAs          Description
                                   Requirements management      Defining, validating, and prioritizing requirements, such
                                                                as functions, performance, and delivery dates.
                                   Software project planning    Developing estimates for the work to be performed,
                                                                establishing the necessary commitments, and defining
                                                                the plan to perform the work.
                                   Software project tracking and Tracking and reviewing software accomplishments and
                                   oversight                     results against documented estimates, commitments, and
                                                                 plans and adjusting these based on the actual
                                                                 accomplishments and results.
                                   Software subcontract         Selecting qualified contractors and managing them
                                   management                   effectively.
                                   Software quality assurance   Reviewing and auditing the software products and
                                                                activities to ensure that they comply with the applicable
                                                                processes, standards, and procedures and providing the
                                                                staff and managers with the results of these reviews and
                                                                audits.
                                   Software configuration       Selecting project baseline items, such as specifications,
                                   management                   systematically controlling these items and changes to
                                                                them, and recording and reporting status and change
                                                                activity for these items.

                                   The SA-CMM includes seven level 2 KPAs. We evaluated Customs against all
                                   seven level 2 KPAs. We also evaluated Customs against one level 3
                                   KPA—acquisition risk management—because it is considered by software
                                   experts to be an integral part of the solicitation, project performance
                                   management, and contract performance management processes (see table
                                   4.2 for a description of each KPA).




                                   Page 35                                GAO/AIMD-99-41 Customs Service Modernization
                                       Chapter 4
                                       ACE Software Development and Acquisition
                                       Processes Are Not Effective




Table 4.2: Description of SA-CMM
Level 2 KPA’s and the Risk             SA-CMM Level 2 KPAs            Description
Management Level 3 KPA                 Software acquisition planning Ensuring that reasonable planning for the software
                                                                     acquisition is conducted and that all elements of the
                                                                     project are included.
                                       Solicitation                   Ensuring that award is made to the contractor most
                                                                      capable of satisfying the specified requirements.
                                       Requirements development       Establishing a common and unambiguous definition of
                                       and management                 software acquisition requirements understood by the
                                                                      acquisition team, system user, and the contractor.
                                       Project office management      Managing the activities of the project office and
                                                                      supporting contractor(s) to ensure a timely, efficient, and
                                                                      effective software acquisition.
                                       Contract tracking and          Ensuring that the software activities under contract are
                                       oversight                      being performed in accordance with contract
                                                                      requirements, and that products and services will satisfy
                                                                      contract requirements.
                                       Evaluation                     Determining that the acquired software products and
                                                                      services satisfy contract requirements prior to
                                                                      acceptance and transition to support.
                                       Transition and support         Providing for the transition of the software products being
                                                                      acquired to their eventual support organization.
                                       SA-CMM Level 3 KPA             Description
                                       Acquisition risk management    Identifying risks as early as possible, adjusting acquisition
                                                                      strategy to mitigate those risks, and developing and
                                                                      implementing a risk management process as an integral
                                                                      part of the acquisition process.

                                       For both models, each KPA contains five common attributes that indicate
                                       whether the implementation and institutionalization of a KPA can be
                                       effective, repeatable, and lasting. The five common features are:

                                   •   Commitment to perform: The actions that the organization must take to
                                       establish the process and ensure that it can endure. Commitment to
                                       perform typically involves establishing organizational policies and
                                       sponsorship.
                                   •   Ability to perform: The preconditions that must exist in the project or
                                       organization to implement the process competently. Ability to perform
                                       typically involves allocating resources, establishing effective
                                       organizational structures, and ensuring that personnel have the needed
                                       training.
                                   •   Activities performed: The roles and procedures necessary to implement
                                       the process. Activities performed typically involve establishing plans and
                                       procedures, performing the work, tracking it, and taking appropriate
                                       management actions.




                                       Page 36                                   GAO/AIMD-99-41 Customs Service Modernization
                            Chapter 4
                            ACE Software Development and Acquisition
                            Processes Are Not Effective




                        •   Measurement and analysis: The activities performed to measure the
                            process and analyze the measurements. Measurement and analysis
                            typically include defining the measurements to be taken and the analyses
                            to be conducted to determine the status and effectiveness of the activities
                            performed.
                        •   Verifying implementation: The steps to ensure that the activities are
                            performed in compliance with the process that has been established.
                            Verification typically includes reviews by management.

                            In accordance with SEI’s SCE method, for each KPA in level 2 and the one KPA
                            in level 3 (risk management), we evaluated institutional policies and
                            practices and compared project-specific guidance and practices against
                            the five common attributes. This project-specific comparison can result in
                            one of four possible outcomes: (1) project strength—effective
                            implementation of the key practice, (2) project weakness—ineffective
                            implementation of a key practice or failure to implement a key practice,
                            (3) project observation—key practice evaluated but evidence is
                            inconclusive and cannot support characterization as either a strength or a
                            weakness, and (4) not rated—key practice not currently relevant to
                            project, therefore not evaluated.


                            In February 1999, we reported that NCAP 0.1 did not fully satisfy any of the
Customs Lacks the           KPAs that SEI’s SW-CMM requires to be CMM level 2 or repeatable.2 While
Capability to Develop       NCAP 0.1 exhibited many strengths in three KPA’s—requirements

ACE Software                management, software project planning, and software project tracking and
                            oversight—it had numerous and significant weaknesses in the remaining
Effectively                 two KPA’s—software quality assurance and software configuration
                            management. As a result, Customs’ ACE software development capability is
                            immature.

                            In our February 1999 report, we also stated that Customs lacked a
                            software process improvement program. Without one, it is unlikely that
                            Customs can effectively address its software process weaknesses.
                            Accordingly, we recommended that, after ensuring that its mission-critical
                            systems are Year 2000 compliant, but before investing in major software
                            development efforts like ACE, Customs (1) assign responsibility and
                            authority for software process improvement, (2) develop and implement a
                            formal plan for software process improvement that, among other things,
                            was based on our software development process maturity findings,
                            (3) ensure that every new software development effort in Customs adopts

                            2
                             GAO/AIMD-99-35, February 11, 1999.



                            Page 37                                GAO/AIMD-99-41 Customs Service Modernization
                          Chapter 4
                          ACE Software Development and Acquisition
                          Processes Are Not Effective




                          processes that satisfy at least SW-CMM level 2 requirements, and (4) ensure
                          that process improvement activities be initiated for all ongoing essential
                          software maintenance projects. Customs agreed with our findings and
                          stated its commitment to software process improvement and maturity,
                          including stating its plans for establishing a software process improvement
                          program and addressing the weaknesses that we identified.

                          Each of the five SW-CMM KPAs, along with examples of how the ACE software
                          development organization compares to each KPA practices, is summarized
                          below. (See appendix III for more detailed information on the KPAs and our
                          findings.)


Requirements Management   The purpose of requirements management is to establish a common
                          understanding of the requirements between the customer and the software
                          developers. According to the SW-CMM, a repeatable requirements
                          management process, among other things, includes (1) documenting the
                          system requirements allocated to software, (2) providing adequate
                          resources and funding for managing the allocated requirements,
                          (3) training members of the software engineering group to perform their
                          requirements management activities, (4) using the allocated requirements
                          as the basis for software plans, work products, and activities, (5) following
                          a written organizational policy for requirements management, and
                          (6) having a quality assurance group that reviews the activities and work
                          products for managing allocated requirements and reports the results.

                          The NCAP 0.1 project exhibited strengths in almost all key practices within
                          the requirements management KPA. For example, (1) the system
                          requirements allocated to software were documented, (2) adequate
                          resources and funding were provided for managing the allocated
                          requirements, (3) members of the software engineering group were trained
                          to perform their requirements management activities, and (4) the software
                          developers used the allocated requirements as the basis for software
                          plans, work products, and activities.

                          Nevertheless, two very important requirements management key practices
                          were not being performed on the NCAP 0.1 project. Specifically, there was
                          no written organizational policy for requirements management and there
                          was no quality assurance group to review the activities. In the absence of
                          these practices, management is missing important means for ensuring that
                          software requirements are managed in a prescribed manner.




                          Page 38                                GAO/AIMD-99-41 Customs Service Modernization
                            Chapter 4
                            ACE Software Development and Acquisition
                            Processes Are Not Effective




Software Project Planning   The purpose of software project planning is to establish reasonable plans
                            for performing software engineering and for managing the software
                            project. According to the SW-CMM, a repeatable software project planning
                            process, among other things, includes (1) documenting the software
                            project plan, and preparing plans for software engineering facilities and
                            support tools, (2) identifying the work products needed to establish and
                            maintain control of the software project, (3) making and using
                            measurements to determine the status of planning activities, (4) following
                            a written organizational policy for planning a software project, (5) training
                            personnel in software project planning and estimating, (6) estimating the
                            software project’s efforts, costs, critical computer resources, and schedule
                            according to documented procedures, and (7) having a quality assurance
                            group that reviews the activities and work products for software project
                            planning and reports the results.

                            The NCAP 0.1 project exhibited strengths in many of the key practices
                            within the software project planning KPA. For example, (1) the software
                            project plan and plans for the software engineering facilities and support
                            tools were documented, (2) software work products that are needed to
                            establish and maintain control of the project were identified in the project
                            plan, and (3) measurements were made and used to determine the status
                            of software planning activities.

                            However, we also found several significant key practice weaknesses.
                            Specifically, (1) NCAP 0.1 did not follow a written policy for planning a
                            software project, (2) project personnel were not trained in project
                            planning and estimating procedures, (3) estimates for the software
                            project’s effort and costs, critical computer resources, and project
                            schedule were not derived according to documented procedures, and
                            (4) there was no software quality assurance group to review or audit the
                            activities and work products for software project planning. Such project
                            planning weaknesses mean that management has no assurance that it will
                            get the consistent, complete, and reliable information about the project’s
                            expected costs and schedules needed to make expeditious and informed
                            investment decisions.


Software Project Tracking   The purpose of software project tracking and oversight is to provide
and Oversight               adequate visibility into the progress of the software development so that
                            management can take effective actions when the software project’s
                            performance deviates significantly from the software plans. According to
                            the SW-CMM, effective software project tracking and oversight, among other



                            Page 39                                GAO/AIMD-99-41 Customs Service Modernization
                   Chapter 4
                   ACE Software Development and Acquisition
                   Processes Are Not Effective




                   things, includes (1) designating a project software manager to be
                   responsible for the project’s software activities and results, (2) having a
                   documented software development plan for tracking software activities
                   and communicating status, (3) conducting periodic internal reviews to
                   track technical progress, plans, performance, and issues against the
                   software development plan, (4) periodically reviewing the activities for
                   software project tracking and oversight with senior management,
                   (5) following a written organizational policy for managing the project,
                   (6) training software project managers in the technical and personnel
                   aspects of the software project, (7) reviewing software project
                   commitments and changes to commitments with senior management, and
                   (8) independently reviewing or auditing the activities and work products
                   for software project tracking and oversight.

                   NCAP  0.1 exhibited strengths in most of the key practices within the
                   software project tracking and oversight KPA. For example, (1) a project
                   manager had been designated to be responsible for the project’s software
                   activities and results, (2) the project had a documented software
                   development plan, (3) the software engineering group conducted periodic
                   internal reviews to track technical progress, plans, performance, and
                   issues against the software development plan (e.g., the sizes of software
                   work products and the risks associated with software costs, resources,
                   and schedule), and (4) software project tracking and oversight activities
                   were reviewed with senior management on a weekly basis.

                   However, significant weaknesses also existed. For example, (1) there was
                   no written organizational policy for managing the software project, (2) the
                   software managers were not trained in managing the technical and
                   personnel aspects of the software project, (3) software project
                   commitments and changes to commitments were not reviewed with senior
                   management, and (4) because no software quality assurance group exists,
                   there were no independent reviews or audits of the activities and work
                   products for software project tracking and oversight.


Software Quality   The purpose of software quality assurance is to independently review and
Assurance          audit software products and activities to verify that they comply with
                   applicable procedures and standards and to provide the software project
                   and higher level managers with the results of these independent reviews
                   and audits. According to the SW-CMM, a repeatable software quality
                   assurance process, among other things, includes (1) preparing a software
                   quality assurance plan according to a documented procedure, (2) having a



                   Page 40                                GAO/AIMD-99-41 Customs Service Modernization
                         Chapter 4
                         ACE Software Development and Acquisition
                         Processes Are Not Effective




                         written organizational policy for implementing software quality assurance,
                         (3) conducting audits of designated work processes and products to verify
                         compliance, (4) documenting deviations identified in the software
                         activities and software work products and handling them according to a
                         documented procedure, and (5) having experts independent of the
                         software quality assurance group periodically review the activities and
                         work products of the project’s software quality assurance group.

                         NCAP  0.1 did not satisfy any of the key practices within the software quality
                         assurance KPA. Because its software quality assurance is ineffective, ACE is
                         at risk that (1) process and product standards will not be met and
                         (2) software will not perform as intended, and will cost more and take
                         longer to develop than expected.


Software Configuration   The purpose of software configuration management is to establish and
Management               maintain the integrity of the products of the software project throughout
                         the project’s software life-cycle. According to the SW-CMM, a repeatable
                         software configuration management process, among other things, includes
                         (1) preparing a software configuration management plan according to a
                         documented procedure, (2) establishing a configuration management
                         library system as a repository for the software baselines, (3) identifying
                         software work products to be placed under configuration management,
                         (4) controlling the release of products from the software baseline library
                         according to a documented procedure, (5) following a written
                         organizational policy for implementing software configuration
                         management, (6) recording the status of configuration items/units
                         according to a documented procedure, (7) making and using
                         measurements to determine the status of software configuration
                         management activities, and (8) periodically reviewing software
                         configuration management activities with senior management.

                         NCAP  0.1 had strengths in several of the key practices within the software
                         configuration management KPA. For example, (1) a software configuration
                         management plan was developed according to a documented procedure,
                         (2) a configuration management library system was established as a
                         repository for the software baselines, (3) software work products to be
                         placed under configuration management were identified, and (4) the
                         release of products from the software baseline library was controlled
                         according to a documented procedure.




                         Page 41                                GAO/AIMD-99-41 Customs Service Modernization
                        Chapter 4
                        ACE Software Development and Acquisition
                        Processes Are Not Effective




                        However, NCAP 0.1 exhibited weaknesses in most of the software
                        configuration management key practices, which collectively jeopardize
                        Customs’ ability to maintain the integrity of the project’s software
                        products. Specifically, (1) there was no organizational policy for
                        implementing software configuration management, (2) the status of
                        configuration management items was not recorded according to a
                        documented procedure, (3) no measurements were taken to determine the
                        status of software configuration management activities, and (4) software
                        configuration management activities were not periodically reviewed with
                        senior management.


                        NCAP 0.2 did not fully satisfy any of the KPA’s that SEI’s SA-CMM requires to be
Customs Lacks the       a CMM level 2 or repeatable. We found extensive weaknesses in all KPA’s
Capability to Acquire   except one—requirements development and management. Each of the
ACE Software            eight KPA’s, along with examples of how Customs’ software acquisition
                        processes compare to the KPA goals, is summarized below. (See appendix
Effectively             IV for more detailed information on KPAs and our findings.)


Software Acquisition    The purpose of software acquisition planning is to ensure that reasonable
Planning                planning for the software acquisition is conducted and that all aspects of
                        the total software acquisition effort are included in these plans. According
                        to the SA-CMM, a repeatable software acquisition planning process, among
                        other things, includes (1) having a written software acquisition policy,
                        (2) developing and documenting the software acquisition strategy and
                        plan, (3) having management review software acquisition planning
                        activities, and (4) making and using measurements to determine the status
                        of software acquisition planning activities.

                        NCAP 0.2 had no strengths in this KPA. For example, (1) the project followed
                        no written policy for planning the software acquisition, (2) there was no
                        documented software acquisition strategy or plan, (3) software acquisition
                        planning activities were not being reviewed by management, and
                        (4) software acquisition planning activities were not being measured. As a
                        result, management lacks an effective means to identify problems in
                        project performance and take corrective action expeditiously.


Solicitation            The purpose of solicitation is to prepare a request for proposal that
                        delineates a project’s software-related requirements, and, consistent with
                        relevant solicitation laws and regulations, select a contractor that can



                        Page 42                                GAO/AIMD-99-41 Customs Service Modernization
                  Chapter 4
                  ACE Software Development and Acquisition
                  Processes Are Not Effective




                  most cost-effectively satisfy these requirements. According to the SA-CMM,
                  specific requirements for a repeatable solicitation process include, among
                  other things, (1) designating responsibility for the software portion of the
                  solicitation, (2) designating a selection official to be responsible for the
                  selection process and decision, (3) preparing cost and schedule estimates
                  for the software products and services being acquired, (4) having a written
                  policy for the conduct of the software portion of the solicitation,
                  (5) having and following a solicitation plan, (6) independently reviewing
                  cost and schedule estimates for the software products and services being
                  acquired, and (7) making and using measurements to determine the status
                  of the solicitation activities and resultant products.

                  NCAP  0.2 had some strengths in this area, including (1) designating
                  responsibility for the software solicitation, (2) designating a selection
                  official for the selection process and decision, and (3) preparing cost and
                  schedule estimates for the software products and services being acquired.

                  However, we found many weaknesses, including (1) no written policy for
                  the conduct of the software solicitation and (2) no solicitation plans to
                  guide the performance of solicitation activities. These weaknesses
                  increase the risk of Customs not adequately or uniformly evaluating the
                  offerors’ proposals, and making a suboptimal selection. Other weaknesses
                  included (1) no independent review of software cost and schedule
                  estimates and (2) no measurements to determine the status of solicitation
                  activities and resultant products. As a result, management cannot identify
                  solicitation problems early and resolve them expeditiously.


Requirements      The purpose of requirements development and management is to establish
Development and   and maintain a common and unambiguous definition of software
Management        requirements among the acquisition team, the system users, and the
                  software development contractor. This KPA involves two subprocesses:
                  (1) developing a baseline set of software-related contractual requirements
                  and (2) managing these requirements and changes to these requirements
                  for the duration of the acquisition.

                  SA-CMM requirements development and management practices necessary to
                  achieve a repeatable maturity include (1) having a group that is
                  responsible for performing requirements development and management
                  activities, (2) ensuring that individuals performing requirements
                  development and management activities have experience or receive
                  training, (3) measuring and reporting on the status of requirements



                  Page 43                                GAO/AIMD-99-41 Customs Service Modernization
                            Chapter 4
                            ACE Software Development and Acquisition
                            Processes Are Not Effective




                            development and management activities and resultant products to
                            management, (4) periodically reviewing requirements development and
                            management activities by acquisition organization management, (5) having
                            a written organizational policy for establishing and managing the
                            software-related contractual requirements, (6) performing requirements
                            development and management activities in accordance with documented
                            plans, (7) appraising the impact of system requirements change requests
                            on the software being acquired, and (8) maintaining traceability between
                            the software-related contractual requirements and the contractor’s
                            software work products.

                            NCAP  0.2 had many strengths in requirements development and
                            management. For example, (1) there was a group that is responsible for
                            performing requirements development and management, (2) group
                            members had experience or have received training in the activities,
                            (3) measurements of requirements development and management
                            activities were made, and (4) management reviewed the activities
                            periodically.

                            However, we found weaknesses in important key practices that jeopardize
                            effective control of the requirements baseline and can result in software
                            products that do not meet cost, schedule, or performance objectives.
                            Specifically, (1) there was no written policy for establishing and managing
                            the software-related contractual requirements, (2) there was no
                            documented requirements development and management plan, (3) system
                            requirements change requests were not appraised for their impact on the
                            software being acquired, and (4) traceability between the software-related
                            contractual requirements and the contractor’s software work products
                            was not maintained.


Project Office Management   The purpose of project office management is to manage the activities of
                            the project office and supporting contractors to ensure a timely, efficient,
                            and effective software acquisition. According to the SA-CMM, effective
                            project office management requires, among other things, (1) designating
                            responsibility for project management, (2) providing project teams with
                            authority to alter either the project’s performance, cost, or schedule
                            baseline, (3) having a written policy for the management of the software
                            project, (4) using a software acquisition management plan, and (5) using a
                            corrective action system for identifying, recording, tracking, and
                            correcting problems.




                            Page 44                                GAO/AIMD-99-41 Customs Service Modernization
                        Chapter 4
                        ACE Software Development and Acquisition
                        Processes Are Not Effective




                        NCAP  0.2 had several practice strengths in the KPA including (1) designating
                        a project manager as responsible for the project and (2) providing project
                        teams with authority to change either the performance, cost, or schedule
                        software acquisition baseline when necessary. However, we found
                        numerous weaknesses, including (1) no written policy for management of
                        the software project, (2) no software acquisition management plan, and
                        (3) no corrective action system. These weaknesses jeopardize the project’s
                        ability to ensure that important project office and contractor activities are
                        defined, understood, and completed.


Contract Tracking and   The purpose of contract tracking and oversight is to ensure that the
Oversight               software development contractor performs according to the terms of the
                        contract; needed contract changes are identified, negotiated, and
                        incorporated into the contract; and contractor performance issues are
                        identified early, when they are easier to address. According to the SA-CMM,
                        a repeatable contract tracking and oversight process, among other things,
                        includes (1) designating responsibility for contract tracking and oversight,
                        (2) providing support for the project team by contracting specialists,
                        (3) ensuring that individuals performing contract tracking and oversight
                        are suitably experienced or trained, (4) having a written organizational
                        policy for contract tracking and oversight, (5) having a documented plan
                        for contract tracking and oversight, (6) reviewing required contractor
                        software planning documents to oversee the contractor’s software
                        engineering effort, and (7) tracking problems or issues found by the
                        project team during contract tracking and oversight in a corrective action
                        system.

                        NCAP 0.2 had several strengths in this KPA, including (1) designating
                        responsibility for contract tracking and oversight activities, (2) providing
                        contracting specialist support to the project team, and (3) ensuring that
                        personnel responsible for contract tracking and oversight are suitably
                        experienced or trained. However, we found significant weaknesses,
                        including (1) no written organizational policy for contract tracking and
                        oversight, (2) no documented contract tracking and oversight plan, (3) no
                        required contractor software planning documents that could be used to
                        oversee the contractor’s software engineering effort, and (4) no corrective
                        action tracking system for recording problems or issues found by the
                        project team. Because of these weaknesses, Customs’ contractor tracking
                        and oversight activities are undisciplined and unstructured, thereby
                        increasing the chances of its software acquisitions being late, costing more
                        than expected, and not performing as intended.



                        Page 45                                GAO/AIMD-99-41 Customs Service Modernization
                         Chapter 4
                         ACE Software Development and Acquisition
                         Processes Are Not Effective




Evaluation               The purpose of evaluation is to determine that the acquired software
                         products and services satisfy contract requirements prior to acceptance.
                         According to the SA-CMM, a repeatable evaluation process includes
                         (1) designating responsibility for planning, managing, and performing
                         evaluation activities, (2) ensuring that individuals performing evaluation
                         activities have experience or receive training, (3) managing the evaluation
                         of the acquired software according to a written policy, (4) documenting
                         evaluation plans and conducting evaluation activities in accordance with
                         the plan, (5) developing and managing evaluation requirements in
                         conjunction with developing software technical requirements, (6) tracking
                         contractor performance of evaluation activities for compliance with the
                         contract, and (7) measuring and reporting on the status of evaluation
                         activities to management.

                         NCAP 0.2 had some evaluation strengths, including (1) designating
                         responsibility for planning, managing, and performing evaluation activities
                         and (2) ensuring that individuals performing evaluation activities have
                         experience or receive training. However, we found many significant
                         weaknesses, including, (1) there was no written policy for managing the
                         evaluation of the acquired software, (2) there was no documented
                         evaluation plan to use as the basis for conducting evaluation activities,
                         (3) evaluation requirements were not developed in conjunction with
                         development of the system or software technical requirements,
                         (4) contractor evaluation activity performance was not tracked for
                         compliance with the contract, and (5) no measurements were made to
                         determine the status of evaluation activities. Because of these pervasive
                         evaluation weaknesses, Customs has no assurance that
                         contractor-developed ACE software will satisfy defined requirements.


Transition and Support   The purpose of transition and support is to provide for the effective and
                         efficient “hand-off” of the acquired software products to the support
                         organization responsible for software maintenance. According to the
                         SA-CMM, repeatable transition and support processes, among other things,
                         include (1) having a written policy for transitioning software products to
                         the support organization, (2) designating a group that is responsible for
                         coordinating transition and support activities, (3) performing project
                         activities in accordance with a documented transition and support plan,
                         and (4) measuring the status of the transition and support activities.

                         NCAP 0.2 had no practice strengths in the transition and support KPA. In
                         particular, (1) there was no written policy for the transitioning of software



                         Page 46                                GAO/AIMD-99-41 Customs Service Modernization
                   Chapter 4
                   ACE Software Development and Acquisition
                   Processes Are Not Effective




                   products to the software support organization, (2) there was no designated
                   group responsible for coordinating transition and support activities,
                   (3) there was no transition and support plan, and (4) the status of
                   transition and support activities were not measured. As a result, Customs
                   is not effectively positioned to assume support responsibility for the ACE
                   software that it acquires.


Acquisition Risk   The purpose of acquisition risk management is to formally identify risks as
Management         early as possible and adjust the acquisition to mitigate those risks.
                   According to the SA-CMM, an effective risk management process, among
                   other things, includes (1) having a written policy on acquisition risk
                   management, (2) developing a software acquisition risk management plan,
                   (3) conducting software risk management activities in accordance with the
                   plan (e.g., identifying risks, taking mitigation actions, and tracking risk
                   mitigation actions to completion), and (4) measuring and reporting on the
                   status of acquisition risk management activities to management.

                   NCAP  0.2 was weak in all key practices of acquisition risk management. In
                   particular, (1) there was no written policy on acquisition risk management,
                   (2) there was no software acquisition risk management plan, (3) no risk
                   management activities were being performed, and (4) no measurements
                   were taken to determine the status of risk management activities. Because
                   of these weaknesses, Customs has no assurance that it will identify risks
                   early and effectively mitigate them.


                   Customs’ software acquisition and development processes are
Conclusions        insufficiently mature to support the effective development or acquisition
                   of ACE. Until these processes are strengthened, there is no basis for
                   confidence that ACE will be delivered on time, within budget, or perform as
                   specified.




                   Page 47                                GAO/AIMD-99-41 Customs Service Modernization
Chapter 5

Overall Conclusions, Recommendations,
and Agency Comments

                  Successful systems modernization is critical to Customs’ ability to
                  implement the provisions of its legislative mandates and to achieve its
                  goals of increased compliance with trade laws and speedier processing of
                  imports. However, Customs currently lacks the full management and
                  technical capability needed to successfully modernize its systems. As a
                  result, satisfaction of its legislative mandate and realization of the
                  attendant benefits by both Customs and the trade community are in
                  jeopardy.

                  With ACE, Customs has not adequately demonstrated that it is “doing the
                  right thing,” i.e., that the system that it is building is the most cost effective
                  system for meeting the needs of Customs and the trade community.
                  Specifically, by not having a complete information systems architecture to
                  guide the development and evolution of its systems, including ACE,
                  Customs runs the risk that its efforts will not provide optimum
                  performance and will be incompatible and expensive to maintain. Also, by
                  using unreliable and incomplete ACE cost and benefit data, Customs does
                  not know if ACE is a cost effective system solution to its needs, or if its
                  investments in ACE are wise.

                  Customs also has not clearly demonstrated that it is developing ACE “the
                  right way.” In particular, without incrementally managing such a
                  mammoth system as ACE, Customs will not know whether its cost and
                  benefit expectations are being met until it has already spent hundreds of
                  millions of dollars. Such a “grand design” approach has proven repeatedly
                  to be ineffective and has been abandoned by successful IT organizations.
                  Additionally, because its ability to develop or acquire software is
                  immature, Customs can do neither effectively. Therefore, its software
                  acquisition and development processes must be strengthened before it can
                  hope to acquire or develop ACE software on time and within budget, as
                  well as to achieve expected benefits.


                  In addition to previous recommendations to improve Customs’
Recommendations   management of information technology,1 we recommend that Customs
                  correct the management and technical weaknesses discussed in this report
                  before building ACE. To accomplish this, GAO recommends that the
                  Commissioner of Customs, with the support of Customs’ CIO, ensure that
                  Customs



                  1
                   GAO/AIMD-99-35, February 11, 1999 and GAO/AIMD-98-70, May 5, 1998.



                  Page 48                                     GAO/AIMD-99-41 Customs Service Modernization
                      Chapter 5
                      Overall Conclusions, Recommendations,
                      and Agency Comments




                      (1) rigorously analyze alternative approaches to building ACE, including
                      ITDS as an alternative to developing ACE entirely within Customs and;


                      (2) make investment decisions incrementally, i.e., for each increment:

                  •   use disciplined processes to prepare a rigorous life-cycle cost estimate,
                      including an explicit discussion of its inherent uncertainty;
                  •   prepare realistic and supportable benefit expectations;
                  •   require a favorable return-on-investment and compliance with Customs’
                      architecture before making any investment; and
                  •   validate actual costs and benefits once an increment is piloted, compare
                      these with estimates, use the results in making further decisions on
                      subsequent increments, and report the results to Customs’ House and
                      Senate appropriations and authorizing committees; and

                      (3) strengthen ACE software acquisition management by:

                  •   establishing an effective process improvement program and correcting the
                      weaknesses in ACE software acquisition processes identified in this report,
                      thereby bringing ACE processes to at least SEI level 2 and
                  •   requiring at least SEI level 2 processes of all ACE software contractors.


                      In its written comments on a draft of this report, Customs agreed with our
Agency Comments       conclusions and recommendations and stated that it is committed to
                      addressing the problems discussed in the report. To this end, Customs
                      cited a number of actions that it has underway and planned to improve
                      management of information technology in general, and ACE specifically.
                      For example, Customs stated that it has recruited a CIO with extensive
                      experience in enterprise architecture and major systems acquisition,
                      reorganized its Office of Information and Technology, and revised its
                      System Development Life Cycle process. Also, Customs stated that it has
                      engaged a contractor to update and improve the ACE cost-benefit analysis
                      and plans for another contractor to independently review the new ACE
                      cost-benefit analysis. Additionally, Customs reported that it revised its
                      investment management procedures to provide for effective systems
                      architecture enforcement, as we recommended in our May 1998 report.2
                      The report has been modified to reflect this information. To fully correct
                      the management and technical weaknesses discussed in this report,
                      Customs must follow through and effectively implement its plans to


                      2
                       GAO/AIMD-98-70, May 5, 1998.



                      Page 49                                 GAO/AIMD-99-41 Customs Service Modernization
Chapter 5
Overall Conclusions, Recommendations,
and Agency Comments




address all of our recommendations. Appendix I provides the full text of
Customs’ comments.




Page 50                                 GAO/AIMD-99-41 Customs Service Modernization
Page 51   GAO/AIMD-99-41 Customs Service Modernization
Appendix I

Comments From the U.S. Customs Service




              Page 52     GAO/AIMD-99-41 Customs Service Modernization
Appendix I
Comments From the U.S. Customs Service




Page 53                                  GAO/AIMD-99-41 Customs Service Modernization
Appendix I
Comments From the U.S. Customs Service




Page 54                                  GAO/AIMD-99-41 Customs Service Modernization
Appendix I
Comments From the U.S. Customs Service




Page 55                                  GAO/AIMD-99-41 Customs Service Modernization
Appendix II

Detailed Comparison of SEI’s Checklist for
Reliable Cost Estimates and Customs’
Practices for Estimating ACE Costs
                                                The following seven tables rate Customs’ practices for estimating life-cycle
                                                costs of $1.05 billion against SEI’s criteria for reliable cost estimating. A
                                                strength indicates that Customs satisfied the SEI criterion; a weakness
                                                indicates that it did not. Where evidence was inconclusive and did not
                                                clearly support a determination of either strength or weakness, a rating of
                                                “observation” was given.


Table II.1: Are the Objectives of the Estimate Clear and Correct?
SEI checklist item                                       Finding                                                    Rating
The objectives of the estimate are stated in writing.      The objectives of the estimates are stated in writing.   Strength
The life-cycle to which the estimate applies is clearly    The 10-year life-cycle is clearly defined for the ACE    Strength
defined.                                                   cost estimate, and consists of 6 years of development
                                                           and deployment and 4 years of operations and
                                                           maintenance.
The tasks and activities included in (and excluded from)   The tasks and activities that Customs included in (and   Strength
the estimate are clearly identified.                       excluded from) the estimate are clearly identified.
The tasks and activities included in the estimate are      The tasks and activities included in the estimate are    Strength
consistent with the objectives of the estimate.            consistent with the stated objectives of the estimate.




                                                Page 56                                  GAO/AIMD-99-41 Customs Service Modernization
                                               Appendix II
                                               Detailed Comparison of SEI’s Checklist for
                                               Reliable Cost Estimates and Customs’
                                               Practices for Estimating ACE Costs




Table II.2: Has the Task Been Appropriately Sized?
SEI checklist item                                            Finding                                                      Rating
A structured process has been used to estimate and            A structured process has not been used to estimate and Weakness
describe the size of the software project.                    describe the size of the entire software project. Customs
                                                              followed a structured process for estimating the size of
                                                              NCAP 0.1, the first ACE software release; however, the
                                                              methodology used to estimate the size of NCAP 0.2 and
                                                              all subsequent ACE software releases was not structured
                                                              and lacked critical elements, such as defined functional
                                                              requirements for NCAP 0.2 and all subsequent releases.
A structured process has been used to estimate and            A structured process has not been used to estimate and       Weakness
describe the extent of software reuse.                        describe the extent of software reuse. According to
                                                              Customs officials, software reuse was discussed during
                                                              an ACE planning conference; however, the results were
                                                              not documented.
The processes for estimating size and reuse are               The processes for estimating size and reuse are not          Weakness
documented.                                                   documented. The documentation available on the
                                                              estimate, including the ACE/NCAP workload estimate
                                                              spreadsheets, does not document the size and reuse
                                                              estimating processes followed by Customs.
The descriptions of size and reuse identify what is           The description of size identifies what is included in the   Weakness
included in (and excluded from) the size and reuse            size measures used (e.g., screens, business objects,
measures used.                                                and operations); however, no description of reuse
                                                              measures exists.
The measures of reuse distinguish between code that will      The measures of reuse are not documented; however,           Observation
be modified and code that will be integrated as is into the   the ACE workload estimate spreadsheets do include an
system.                                                       estimate of existing code that will be modified for use in
                                                              subsequent NCAP software releases.
The definitions, measures, and rules used to describe size Customs did not use a model to estimate the cost of             Weakness
and reuse are consistent with the requirements (and        ACE. Furthermore, the reuse definitions, measures, and
calibrations) of the models used to estimate cost.         rules are not documented.
The size estimate was checked by relating it to measured      The size estimate for the entire ACE project was not      Weakness
sizes of other software products or components.               checked by relating it to measured sizes of other
                                                              software products or components. Customs did not have
                                                              a comparable internal software product to benchmark
                                                              against at the time of the estimate. According to Customs
                                                              officials, the NCAP 0.1 size estimate was compared (via
                                                              informal and undocumented telephone inquiries) to
                                                              similar products and components previously developed
                                                              outside of Customs. The size estimate for NCAP 0.2 and
                                                              subsequent releases was based upon the NCAP 0.1 size
                                                              estimate.
The size estimating process was checked by testing its        The size estimating process was not checked by testing       Weakness
predictive capabilities against measured sizes of             its predictive capabilities against measured sizes of
completed products.                                           completed projects.




                                               Page 57                                      GAO/AIMD-99-41 Customs Service Modernization
                                                Appendix II
                                                Detailed Comparison of SEI’s Checklist for
                                                Reliable Cost Estimates and Customs’
                                                Practices for Estimating ACE Costs




Table II.3: Is the Estimated Cost Consistent With Demonstrated Accomplishments on Other Projects?
SEI checklist item                                      Finding                                                              Rating
The organization has a structured process for relating          Customs does not have a structured, documented               Weakness
estimates to actual costs of completed work                     process for relating estimates to actual costs of
—the process is documented                                      completed work.
—the process was followed.
The cost models that were used have been calibrated to          Customs did not use a cost model and did not have any        Weakness
relevant historical data. (Models of some sort are needed       relevant historical data to calibrate.
to provide consistent rules for extrapolating from previous
experience.)
The cost models quantify demonstrated organizational            Customs did not use a cost model. Further, Customs           Weakness
performance in ways that normalize for differences among        changed its software development approach from an
software products and projects. (So that a simple,              in-house development (NCAP 0.1) to an acquisition
unnormalized, lines-of-code per staff-month extrapolation       (NCAP 0.2). Nonetheless, Customs used the NCAP 0.1
is NOT the basis for the estimate.)                             estimate as the basis for extrapolating the estimate for
                                                                the remaining ACE releases without normalizing the
                                                                NCAP 0.1 data to account for the differences in the
                                                                approaches.
The consistency achieved when fitting the cost models to        Customs did not use a cost model. At the time that the     Weakness
historical data has been measured and reported.                 ACE cost estimate was developed, Customs had no
                                                                historical data with which to evaluate consistency. Cost
                                                                data are currently being collected, but the collected data
                                                                has not been used to measure and report the
                                                                consistency achieved when using the cost models.
The values used for cost models’ parameters appear valid Customs did not use a cost model; therefore,                        Weakness
when compared to values that fit the models well to past comparison of cost models’ parameter values to those
projects.                                                used on past projects was not done.
The calibration of cost models was done with the same           Customs did not use cost models; therefore, calibration      Weakness
versions of the models that were used to prepare the            was not done.
estimate.
The methods used to account for reuse recognize that            The methods used to account for reuse recognize that         Strength
reuse is not free. (The estimate accounts for activities such   reuse is not free. The Customs estimating spreadsheets
as interface design, modification, integration, testing, and    account for activities associated with NCAP software
documentation that are associated with effective reuse.)        reuse, such as modification of existing code, integration,
                                                                and testing.
Extrapolations from past projects account for differences       Extrapolations from NCAP 0.1 to NCAP 0.2 and                 Weakness
in application technology. (For example, data from              subsequent releases did not account for differences in
projects that implemented traditional mainframe                 application technology (e.g., the change in architecture
applications require adjustments if used as a basis for         from the legacy mainframe system to a combined
estimating client-server implementation. Some cost              mainframe and Internet-based client-server system).
models provide capabilities for this, others do not.)
Extrapolations from past projects account for observed,         Extrapolations from NCAP 0.1 to NCAP 0.2 and                 Weakness
long-term trends in software technology improvement.            subsequent releases did not account for differences in
(Although some cost models attempt this internally, the         tools, languages, and personnel due to the change in
best methods are usually based on extrapolating                 architecture from the legacy mainframe system written in
measured trends in calibrated organizational                    COBOL and C++ to a combined mainframe and
performance.)                                                   Internet-based system written in C++ and Java.
                                                                                                                                 (continued)




                                                Page 58                                      GAO/AIMD-99-41 Customs Service Modernization
                                                Appendix II
                                                Detailed Comparison of SEI’s Checklist for
                                                Reliable Cost Estimates and Customs’
                                                Practices for Estimating ACE Costs




SEI checklist item                                              Finding                                                    Rating
Extrapolations from past projects account for the effects       Extrapolations from NCAP 0.1 to NCAP 0.2 and               Weakness
of introducing new software technology or processes.            subsequent releases did not account for the change in
(Introducing a new technology or process can initially          process from an in-house software development project
reduce an organization’s productivity.)                         (NCAP 0.1) to a software acquisition (NCAP 0.2).
Work-flow schematics have been used to evaluate how             Work-flow schematics have not been used to evaluate        Weakness
this project is similar to (and how it differs from) projects   how this project is similar to (and how it differs from)
used to characterize the organization’s past performance.       projects used to characterize the organization’s past
                                                                performance.


Table II.4: Have the Factors That Affect the Estimate Been Identified and Explained?
SEI checklist item                                       Finding                                                           Rating
A written summary of parameter values and their                 A written summary of the parameters affecting the cost    Weakness
rationales accompanies the estimate.                            estimate exists; however, a written summary of the values
                                                                used and their rationales does not exist.
Assumptions have been identified and explained.                 The assumptions have been identified and explained.        Strength
A structured process such as a template or format has           Customs has used spreadsheets designed specifically        Strength
been used to ensure that key factors have not been              for the ACE NCAP project in an effort to ensure that key
overlooked.                                                     factors have not been overlooked.
Uncertainties in parameter values have been identified          No uncertainties in parameter values were identified or    Weakness
and quantified.                                                 quantified.
A risk analysis has been performed, and risks that affect       A risk analysis has been performed, and risks that affect Strength
cost have been identified and documented. (Elements             cost have been identified and documented (e.g., lack of
addressed include issues such as probability of                 requirement definition for later releases, technical
occurrence, effects on parameter values, cost impacts,          complications, stability of development and configuration
schedule impacts, and interactions with other                   management environments and process).
organizations.)




                                                Page 59                                      GAO/AIMD-99-41 Customs Service Modernization
                                               Appendix II
                                               Detailed Comparison of SEI’s Checklist for
                                               Reliable Cost Estimates and Customs’
                                               Practices for Estimating ACE Costs




Table II.5: Have Steps Been Taken to Ensure the Integrity of the Estimating Process?
SEI checklist item                                       Finding                                                        Rating
Management reviewed and agreed to the values for all         According to Customs project officials, management         Observation
descriptive parameters before costs were estimated.          reviewed and agreed to the values for all descriptive
                                                             parameters before costs were estimated; however,
                                                             management approval has not been documented.
Adjustments to parameter values to meet a desired cost       Not applicable. According to Customs project officials,    N/A
or schedule have been documented.                            no adjustments to parameter values to meet a desired
                                                             cost or schedule were made.
If a dictated schedule has been imposed, the estimate is     Not applicable. According to Customs project officials,    N/A
accompanied by an estimate of (1) the normal schedule        no dictated schedule has been imposed.
and (2) the additional expenditures required to meet the
dictated schedule.
Adjustments to parameter values to meet a desired cost       Not applicable. No adjustments to parameter values         N/A
or schedule are accompanied by management action that        were made, and thus no accompanying management
makes the values realistic.                                  action occurred.
More than one cost model or estimating approach has          Two estimating approaches (spreadsheets) were used         Strength
been used, and the differences in results have been          for NCAP, and the resulting differences in the estimates
analyzed and explained.                                      (within 15 percent) have been analyzed and explained.
People from related but different projects or disciplines    People from the requirements, design, systems              Strength
were involved in preparing the estimate.                     engineering, integration, testing, and project
                                                             management teams were involved in preparing the
                                                             estimate.
At least one member of the estimating team is an             Two members of the estimating team have practical          Observation
experienced estimator, trained in the cost models that       experience, but none have been trained in cost
were used.                                                   estimating and cost models.
Estimators independent of the performing organization        According to Customs project officials, estimators         Observation
concur with the reasonableness of the parameter values       independent of the performing organization concurred
and estimating methodology.                                  with the reasonableness of the parameter values and
                                                             estimating methodology, but their analysis is not
                                                             documented.
The groups that will be doing the work accept the estimate Customs (NCAP 0.1 developer) and the contractor              Strength
as an achievable target.                                   (NCAP 0.2 developer) accepted the estimates as
                                                           achievable targets.
Memorandums of agreement have been completed and             No memorandums of agreement have been completed          Weakness
signed with the other organizations whose contributions      and signed with the requirements, design, systems
affect cost.                                                 engineering, integration, testing, project management
                                                             team, and other organizations whose contributions affect
                                                             cost.




                                               Page 60                                      GAO/AIMD-99-41 Customs Service Modernization
                                                 Appendix II
                                                 Detailed Comparison of SEI’s Checklist for
                                                 Reliable Cost Estimates and Customs’
                                                 Practices for Estimating ACE Costs




Table II.6: Is the Organization’s Historical Evidence Capable of Supporting a Reliable Estimate?
SEI checklist item                                        Finding                                                           Rating
The estimating organization has a method for organizing         Customs did not have a method for organizing and            Weakness
and retaining information on completed projects (a              retaining information on completed projects (a historical
historical database).                                           database) at the time that this estimate was developed.
                                                                (Note: Customs has since developed a historical
                                                                database which contains planned versus actual results
                                                                for schedule, effort, and cost and plans to use this in
                                                                future estimates.)
The database contains a useful set of completed projects.       Customs did not have a historical database at the time      Weakness
                                                                this estimate was developed. (Note: The database
                                                                developed later contains NCAP 0.1 and 0.2 data only.
                                                                No data from other completed projects is included.)
Elements included in (and excluded from) the effort, cost,      Customs did not have a historical database at the time      Weakness
schedule, size, and reuse measures in the database are          this estimate was developed. Elements included in the
clearly identified. (See, for example, the SEI checklists for   effort, cost, and schedule measures in the database
defining effort, schedule, and size measures.)                  Customs has since developed are clearly identified.
                                                                However, size and reuse measures are not identified in
                                                                the database.
Schedule milestones (start and finish dates) are described The schedule milestones (start and finish dates) are             Strength
in terms of criteria for initiation or completion, so that work accompanied by task and subtask descriptions.
accomplished between milestones is clearly bounded.
Records for completed projects indicate whether or not          Not applicable. According to Customs project officials,     N/A
unpaid overtime was used.                                       unpaid overtime was insignificant and did not warrant
                                                                recording for NCAP 0.1.
Unpaid overtime, if used, has been quantified, so that          Not applicable. According to Customs project officials,     N/A
recorded data provide a valid basis for estimating future       unpaid overtime was insignificant and did not warrant
effort.                                                         recording for NCAP 0.1.
Cost models that were used for estimating have been             Customs did not use any cost models for estimating.         Weakness
used also to provide consistent frameworks for recording        The spreadsheets that were used for cost and workload
historical data. (This helps ensure that comparable terms       estimating do not provide a consistent framework for
and parameters are used across all projects, and that           recording historical data for NCAP releases 0.1 and 0.2.
recorded data are suitable for use in the estimating
models.)
The data in the historical database have been examined to       At the time of the estimate, there was no historical        Weakness
identify inconsistencies and anomalies have been                database. (Note: The data in the historical database
corrected or explained. (This is best done with the same        prepared later have not yet been examined to identify
cost models used for estimating.)                               inconsistencies.)
The organization has a structured process for capturing         Customs has a structured process for capturing effort       Strength
effort and cost data from ongoing projects.                     and cost data from the ACE project on a weekly basis.
The producing organization holds postmortems at the             According to project officials, Customs plans to hold       Weakness
completion of its projects to (1) ensure that recorded data     postmortems at the completion of each ACE/NCAP
are valid, and (2) ensure that events that affected costs       software release. NCAP 0.1 was completed but a
get recorded and described while they are still fresh in        postmortem has not yet been performed. No
people’s minds.                                                 documentation exists which describes the postmortem
                                                                process or structure.
                                                                                                                                  (continued)




                                                 Page 61                                      GAO/AIMD-99-41 Customs Service Modernization
                                             Appendix II
                                             Detailed Comparison of SEI’s Checklist for
                                             Reliable Cost Estimates and Customs’
                                             Practices for Estimating ACE Costs




SEI checklist item                                         Finding                                                    Rating
Information on completed projects includes                According to Customs project officials, all this            Weakness
—the life-cycle model used, together with the portion     information has not yet been collected, but they plan to
covered by the recorded cost and schedule;                do so.
—actual (measured) size, cost, and schedule;
—the actual staffing profile;
—an estimate at completion, together with the values for
cost model parameters that map the estimate to the actual
cost and schedule;
—a work breakdown structure or alternative description of
the tasks included in the recorded cost;
—a work-flow schematic that illustrates the software
process used;
—nonlabor costs;
—management costs;
—a summary or list of significant deliverables (software
and documentation) produced by the project; and
—a summary of any unusual issues that affected cost.
Evolution in the organization’s work-flow schematics shows Customs’ historical database on completed software         Weakness
steady improvement in the understanding and                projects represents an effort to understand and measure
measurement of its software processes.                     its software processes. Customs currently has no other
                                                           ongoing software process improvement efforts.




                                             Page 62                                      GAO/AIMD-99-41 Customs Service Modernization
                                               Appendix II
                                               Detailed Comparison of SEI’s Checklist for
                                               Reliable Cost Estimates and Customs’
                                               Practices for Estimating ACE Costs




Table II.7: Has the Situation Remained Unchanged Since the Estimate Was Prepared?
SEI checklist item                                     Finding                                                            Rating
The estimate has not been invalidated by recent events,      The estimate has been invalidated by the following         Weakness
changing requirements, or management action (or              recent actions and events: (1) The approved fiscal year
inaction).                                                   1999 ACE appropriated funding totals $6 million, which is
                                                             significantly less than the $75.1 million fiscal year 1999
                                                             funding requirement stated in the ACE cost estimate;
                                                             (2) NCAP 0.1 was developed in-house and the cost
                                                             estimate for subsequent releases was based on the
                                                             assumption that they would be developed in-house as
                                                             well. Customs has since decided to acquire NCAP 0.2
                                                             and perhaps subsequent releases from a contractor; and
                                                             (3) NCAP 0.2 and subsequent releases will utilize a
                                                             combined mainframe and internet-based client-server
                                                             architecture instead of the NCAP 0.1 mainframe
                                                             architecture upon which the estimate was based.
The estimate is being used as the basis for assigning        The estimate is not being used as the basis for              Weakness
resources, deploying schedules, and making                   assigning resources, deploying schedules, and making
commitments.                                                 commitments. Instead, Customs is using a revised
                                                             estimate that reflects actual fiscal year 1998 and 1999
                                                             ACE appropriated funding, which is significantly less
                                                             than the funding requirements identified in the ACE cost
                                                             estimate for fiscal years 1998 and 1999.
The estimate is the current baseline for project tracking    The estimate is not the current baseline for project         Weakness
and oversight.                                               tracking and oversight. Instead, Customs is using a
                                                             revised estimate that reflects actual fiscal year 1998 and
                                                             1999 ACE appropriated funding, which is significantly
                                                             less than the funding requirements identified in the ACE
                                                             cost estimate for fiscal years 1998 and 1999. Customs is
                                                             currently tracking actual project costs against these
                                                             appropriated funds.




                                               Page 63                                      GAO/AIMD-99-41 Customs Service Modernization
Appendix III

Results of Software Development Capability
Maturity Model (SW-CMM) Evaluation for
NCAP 0.1

Table III.1: Key Process Area: Requirements Management
                      Key practice                                   Finding                                       Rating
Commitment 1       The project follows a written organizational There is no written organizational policy for Weakness
                   policy for managing the system requirements managing the system requirements allocated
                   allocated to software.                       to software.
Ability 1          For each project, responsibility is established   The Process Analysis and Requirements         Strength
                   for analyzing the system requirements and         Team is responsible for analyzing system
                   allocating them to hardware, software, and        requirements and allocating them to
                   other system components.                          hardware, software, and other system
                                                                     components.
Ability 2          The allocated requirements are documented. Allocated requirements are documented.               Strength
Ability 3          Adequate resources and funding are                Adequate resources and funding are            Strength
                   provided for managing the allocated               provided for managing the allocated
                   requirements.                                     requirements.
Ability 4          Members of the software engineering group         Members of the software engineering group     Strength
                   and other software-related groups are             and other software-related groups are
                   trained to perform their requirements             trained to perform their requirements
                   management activities.                            management activities.
Activity 1         The software engineering group reviews the        The software engineering group reviews the    Strength
                   allocated requirements before they are            allocated requirements before they are
                   incorporated into the software project.           incorporated into the software project.
Activity 2         The software engineering group uses the        The software engineering group uses the        Strength
                   allocated requirements as the basis for        allocated requirements as the basis for
                   software plans, work products, and activities. software plans, work products, and activities.
Activity 3         Changes to the allocated requirements are         Changes to the allocated requirements are     Strength
                   reviewed and incorporated into the software       reviewed and incorporated into the software
                   project.                                          project.
Measurement 1      Measurements are made and used to                 Measurements are made and used to             Strength
                   determine the status of the activities for        determine the status of the activities for
                   managing the allocated requirements.              managing the allocated requirements.
Verification 1     The activities for managing the allocated         Periodic meetings with senior management      Strength
                   requirements are reviewed with senior             include reviews of allocated requirements.
                   management on a periodic basis.
Verification 2     The activities for managing the allocated         The activities for managing the allocated   Strength
                   requirements are reviewed with the project        requirements are reviewed with the project
                   manager on both a periodic and event-driven       manager on both a periodic and event-driven
                   basis.                                            basis.
Verification 3     The software quality assurance group              There is no software quality assurance        Weakness
                   reviews and/or audits the activities and work     group; therefore, no reviews and/or audits
                   products for managing the allocated               are done.
                   requirements and reports the results.




                                            Page 64                                      GAO/AIMD-99-41 Customs Service Modernization
                                             Appendix III
                                             Results of Software Development Capability
                                             Maturity Model (SW-CMM) Evaluation for
                                             NCAP 0.1




Table III.2: Key Process Area: Software Project Planning
                      Key practice                                  Finding                                         Rating
Commitment 1        A project software manager is designated to     The NCAP project has a software manager         Strength
                    be responsible for negotiating commitments      designated to be responsible for negotiating
                    and developing the project’s software           commitments and developing the project’s
                    development plan.                               software development plan.
Commitment 2        The project follows a written organizational    The project does not follow a written           Weakness
                    policy for planning a software project.         organizational policy for planning a software
                                                                    project.
Ability 1           A documented and approved statement of          The approved project plan meets the             Strength
                    work exists for the software project.           requirements for a statement of work for the
                                                                    project.
Ability 2           Responsibilities for developing the software    The project manager has been assigned           Strength
                    development plan are assigned.                  responsibility for developing the software
                                                                    development plan.
Ability 3           Adequate resources and funding are              Adequate resources and funding have been        Strength
                    provided for planning the software project.     provided for planning the software project.
Ability 4           The software managers, software engineers, Project personnel are not trained in software        Weakness
                    and other individuals involved in the software project planning and estimating procedures.
                    project planning are trained in the software
                    estimating and planning procedures
                    applicable to their areas of responsibility.
Activity 1          The software engineering group participates     The software engineering group participates     Strength
                    on the project proposal team.                   on the project proposal team.
Activity 2          Software project planning is initiated in the   Software project planning is initiated in the   Strength
                    early stages of, and in parallel with, the      early stages of, and in parallel with, the
                    overall project planning.                       overall project planning.
Activity 3          The software engineering group participates The software engineering group participates Strength
                    with other affected groups in the overall       with other affected groups in the overall
                    project planning throughout the project’s life. project planning throughout the project’s life.
Activity 4          Software project commitments made to            Software project commitments made to            Weakness
                    individuals and groups external to the          individuals and groups external to the
                    organization are reviewed with senior           organization are not reviewed with senior
                    management according to a documented            management and there is no documented
                    procedure.                                      procedure for such reviews.
Activity 5          A software life-cycle with predefined stages    There is no documented evidence that a          Weakness
                    of manageable size is identified or defined.    software life-cycle was selected for the
                                                                    project.
Activity 6          The project’s software development plan is      The project has a software development plan Strength
                    developed according to a documented             that is developed according to a
                    procedure.                                      documented procedure.
Activity 7          The plan for the software project is            The plan for the software project is            Strength
                    documented.                                     documented in the project plan.
Activity 8          Software work products that are needed to      Software work products that are needed to      Strength
                    establish and maintain control of the software establish and maintain control of the software
                    project are identified.                        project are identified in the project plan.
                                                                                                                             (continued)




                                             Page 65                                   GAO/AIMD-99-41 Customs Service Modernization
                                         Appendix III
                                         Results of Software Development Capability
                                         Maturity Model (SW-CMM) Evaluation for
                                         NCAP 0.1




                 Key practice                                    Finding                                         Rating
Activity 9       Estimates for the size of the software work     Estimates for the size of the software work     Strength
                 products (or changes to the size of software    products (or changes to the size of software
                 work products) are derived according to a       work products) are derived according to a
                 documented procedure.                           documented procedure.
Activity 10      Estimates for the software project’s effort and Estimates for the software project’s effort     Weakness
                 costs are derived according to a                and costs are not derived according to a
                 documented procedure.                           documented procedure.
Activity 11      Estimates for the project’s critical computer   Estimates for the project’s critical computer   Weakness
                 resources are derived according to a            resources are not derived according to a
                 documented procedure.                           documented procedure.
Activity 12      The project’s software schedule is derived      There is no documented procedure for            Weakness
                 according to a documented procedure.            deriving the software schedule.
Activity 13      The software risks associated with the cost,    The software risks associated with the cost,    Strength
                 resource, schedule, and technical aspects of    resource, schedule, and technical aspects
                 the project are identified, assessed, and       of the project are identified, assessed, and
                 documented.                                     documented.
Activity 14      Plans for the project’s software engineering    Plans for the project’s software engineering    Strength
                 facilities and support tools are prepared.      facilities and support tools are prepared.
Activity 15      Software planning data are recorded.            Software planning data are recorded.            Strength
Measurement 1    Measurements are made and used to             Measurements are made and used to             Strength
                 determine the status of the software planning determine the status of the software planning
                 activities.                                   activities (status and MS Project reports).
Verification 1   The activities for software project planning    The activities for software project planning Strength
                 are reviewed with senior management on a        are periodically reviewed with senior
                 periodic basis.                                 management, including reports to Treasury
                                                                 and Customs investment review board (IRB).
Verification 2   The activities for software project planning    The activities for software project planning    Strength
                 are reviewed with the project manager on        are reviewed with the project manager on
                 both a periodic and event-driven basis.         both a periodic and event-driven basis
                                                                 through weekly status reports and meetings.
Verification 3   The software quality assurance group            There is no SQA group.                          Weakness
                 reviews and/or audits the activities and work
                 products for software project planning and
                 reports the results.




                                         Page 66                                     GAO/AIMD-99-41 Customs Service Modernization
                                             Appendix III
                                             Results of Software Development Capability
                                             Maturity Model (SW-CMM) Evaluation for
                                             NCAP 0.1




Table III.3: Key Process Area: Software Project Tracking and Oversight
                      Key practice                              Finding                                             Rating
Commitment 1        A project software manager is designated to     The project software manager is designated      Strength
                    be responsible for the project’s software       to be responsible for the project’s software
                    activities and results.                         activities and results.
Commitment 2        The project follows a written organizational    There is no written organizational policy for   Weakness
                    policy for managing the software project.       managing the software project.
Ability 1           A software development plan for the software An approved and documented software                Strength
                    project is documented and approved.          development plan is contained in the project
                                                                 plan.
Ability 2           The project software manager explicitly         The project software manager explicitly         Strength
                    assigns responsibility for software work        assigns responsibility for software work
                    products and activities.                        products and activities.
Ability 3           Adequate resources and funding are              Adequate resources and funding are              Strength
                    provided for tracking the software project.     provided for tracking the software project.
Ability 4           The software managers are trained in            The software managers are not trained in        Weakness
                    managing the technical and personnel            managing the technical and personnel
                    aspects of the software project.                aspects of the software project.
Ability 5           First-line software managers receive            First-line software managers receive            Strength
                    orientation in the technical aspects of the     orientation in the technical aspects of the
                    software project.                               software project.
Activity 1          A documented software development plan is A documented software development plan                Strength
                    used for tracking the software activities and is used for tracking the software activities
                    communicating status.                         and communicating status.
Activity 2          The project’s software development plan is      No documented procedure exists for              Weakness
                    revised according to a documented               revising the software development plan.
                    procedure.
Activity 3          Software project commitments and changes        Software project commitments and changes        Weakness
                    to commitments made to individuals and          to commitments made to individuals and
                    groups external to the organization are         groups external to the organization are not
                    reviewed with senior management according       reviewed with senior management. Also,
                    to a documented procedure.                      there is no documented procedure for such
                                                                    reviews.
Activity 4          Approved changes to commitments that            Approved changes to commitments that         Strength
                    affect the software project are communicated    affect the software project are communicated
                    to the members of the software engineering      to the members of the software engineering
                    group and other software-related groups.        group and other software-related groups
                                                                    through weekly staff meetings.
Activity 5          The sizes of the software work products (or     The size of the software work products (or      Strength
                    sizes of the changes to the software work       size of the changes to the software work
                    products) are tracked, and corrective actions   products) are tracked; however, at the time
                    are taken as necessary.                         of our evaluation, no corrective actions were
                                                                    needed.
Activity 6          The project’s software effort and costs are     The project’s software effort and costs are     Strength
                    tracked and corrective actions are taken as     tracked; however, at the time of our
                    necessary.                                      evaluation, no corrective actions were
                                                                    needed.
                                                                                                                             (continued)




                                             Page 67                                   GAO/AIMD-99-41 Customs Service Modernization
                                          Appendix III
                                          Results of Software Development Capability
                                          Maturity Model (SW-CMM) Evaluation for
                                          NCAP 0.1




                 Key practice                                    Finding                                         Rating
Activity 7       The project’s critical computer resources are The project’s critical computer resources are Strength
                 tracked, and corrective actions are taken as tracked; however, at the time of our
                 necessary.                                    evaluation, no corrective actions were
                                                               needed.
Activity 8       The project’s software schedule is tracked,     The project’s software schedule is tracked;     Strength
                 and corrective actions are taken as             however, at the time of our evaluation, no
                 necessary.                                      corrective actions were needed.
Activity 9       Software engineering technical activities are   Software engineering technical activities are   Strength
                 tracked, and corrective actions are taken as    tracked, and corrective actions are taken as
                 necessary.                                      necessary.
Activity 10      The software risks associated with cost,        The software risks associated with cost,        Strength
                 resource, schedule, and technical aspects       resource, schedule, and technical aspects
                 of the project are tracked.                     of the project are tracked.
Activity 11      Actual measurement data and replanning          Actual measurement data and replanning          Strength
                 data for the software project are recorded.     data for the software project are recorded.
Activity 12      The software engineering group conducts         The software engineering group conducts         Strength
                 periodic internal reviews to track technical    periodic internal reviews to track technical
                 progress, plans, performance, and issues        progress, plans, performance, and issues
                 against the software development plan.          against the software development plan.
Activity 13      Formal reviews to address the                   Formal reviews to address the                   Strength
                 accomplishments and results of the software     accomplishments and results of the software
                 project are conducted at selected project       project are conducted at selected project
                 milestones according to a documented            milestones according to a documented
                 procedure.                                      procedure.
Measurement 1    Measurements are made and used to               Measurements are made and used to               Strength
                 determine the status of the software tracking   determine the status of the software tracking
                 and oversight activities.                       and oversight activities.
Verification 1   The activities for software project tracking    The activities for software project tracking    Strength
                 and oversight are reviewed with senior          and oversight are reviewed with senior
                 management periodically.                        management on a weekly basis.
Verification 2   The activities for software project tracking    The activities for software project tracking Strength
                 and oversight are reviewed with the project     and oversight are reviewed with the project
                 manager on both a periodic and event-driven     manager on both a periodic and event-driven
                 basis.                                          basis.
Verification 3   The software quality assurance group            No software quality assurance group exists;     Weakness
                 reviews and/or audits the activities and work   therefore, there are no reviews and/or audits
                 products for software project tracking and      of the activities and work products for
                 oversight and reports the results.              software project tracking and oversight.




                                          Page 68                                   GAO/AIMD-99-41 Customs Service Modernization
                                             Appendix III
                                             Results of Software Development Capability
                                             Maturity Model (SW-CMM) Evaluation for
                                             NCAP 0.1




Table III.4: Key Process Area: Software Quality Assurance
                      Key practice                                     Finding                                           Rating
Commitment 1        The project follows a written organizational       There is no written organizational policy for     Weakness
                    policy for implementing software quality           implementing SQA.
                    assurance (SQA).
Ability 1           A group that is responsible for coordinating       Although there is no group responsible for        Observation
                    and implementing SQA for the project (the          coordinating and implementing SQA for the
                    SQA group) exists.                                 project, there are plans to establish one and
                                                                       assign responsibility.
Ability 2           Adequate resources and funding are                 There is no SQA group, and no resources           Weakness
                    provided for performing the SQA activities.        and funding are provided for performing
                                                                       SQA activities.
Ability 3           Members of the SQA group are trained to            There is no SQA group or plan to provide          Weakness
                    perform their SQA activities.                      SQA training.
Ability 4           The members of the software project receive        Project staff do not receive orientation on the   Weakness
                    orientation on the role, responsibilities,         role, responsibilities, authority, and value of
                    authority, and value of the SQA group.             the SQA group.
Activity 1          The SQA plan is prepared for the software          There is no SQA plan or documented                Weakness
                    project according to a documented                  procedure for preparing one.
                    procedure.
Activity 2          The SQA group’s activities are performed in        There is no SQA group.                            Weakness
                    accordance with the SQA plan.
Activity 3          The SQA group participates in the                  There is no SQA group.                            Weakness
                    preparation and review of the project’s
                    software development plan, standards, and
                    procedures.
Activity 4          The SQA group reviews the software                 There is no SQA group.                            Weakness
                    engineering activities to verify compliance.
Activity 5          The SQA group audits designated software           There is no SQA group.                            Weakness
                    work products to verify compliance.
Activity 6          The SQA group periodically reports the             There is no SQA group.                            Weakness
                    results of its activities to the software
                    engineering group.
Activity 7          Deviations identified in the software activities   Procedures for handling deviations in             Weakness
                    and software work products are documented          testing activities are documented. However,
                    and handled according to a documented              procedures for handling deviations in other
                    procedure.                                         software development activities (such as
                                                                       compliance with organizational policy and
                                                                       standards and adherence to software
                                                                       development plan) are not documented.
Activity 8          The SQA group conducts periodic reviews of There is no SQA group.                                    Weakness
                    its activities and findings with the customer’s
                    SQA personnel, as appropriate.
Measurement 1       Measurements are made and used to                  There is no SQA group.                            Weakness
                    determine the cost and schedule status of
                    the SQA activities.
Verification 1      The SQA activities are reviewed with senior        There is no SQA group.                            Weakness
                    management on a periodic basis.
                                                                                                                                  (continued)


                                             Page 69                                      GAO/AIMD-99-41 Customs Service Modernization
                                           Appendix III
                                           Results of Software Development Capability
                                           Maturity Model (SW-CMM) Evaluation for
                                           NCAP 0.1




                   Key practice                                   Finding                                        Rating
Verification 2     The SQA activities are reviewed with the       There is no SQA group.                         Weakness
                   project manager on both a periodic and
                   event-driven basis.
Verification 3     Experts independent of the SQA group           There is no SQA group.                         Weakness
                   periodically review the activities and
                   software work products of the project’s SQA
                   group.


Table III.5: Key Process Area: Software Configuration Management
                      Key practice                            Finding                                            Rating
Commitment 1       The project follows a written organizational   The project has no organizational policy for   Weakness
                   policy for implementing software               implementing SCM.
                   configuration management (SCM).
Ability 1          A board having the authority for managing      The project does not have a SCM board with     Weakness
                   the project’s software baselines (i.e., a      authority for managing software baselines.
                   software configuration control board) exists
                   or is established.
Ability 2          A group that is responsible for coordinating   The project does not have a group that is      Weakness
                   and implementing SCM for the project (i.e.,    responsible for coordinating and
                   the SCM group) exists.                         implementing SCM functions.
Ability 3          Adequate resources and funding are             Adequate resources and funding are not         Weakness
                   provided for performing the SCM activities.    provided for performing the SCM activities.
Ability 4          Members of the SCM group are trained in the There is no SCM group.                            Weakness
                   objectives, procedures, and methods for
                   performing their SCM activities.
Ability 5          Members of the software engineering group      Members of the software engineering group      Weakness
                   and other software-related groups are          and other software-related groups are not
                   trained to perform their SCM activities.       trained to perform their SCM activities.
Activity 1         A SCM plan is prepared for each software       A SCM plan was prepared according to           Strength
                   project according to a documented              procedures documented in the October
                   procedure.                                     1996 SDLC.
Activity 2         A documented and approved SCM plan is          The documented and approved SCM plan is Weakness
                   used as the basis for performing the SCM       used as the basis for performing code
                   activities.                                    control; however, the plan is not used as a
                                                                  basis for doing SCM on software
                                                                  documentation and other software
                                                                  engineering products such as cost estimates
                                                                  and schedules.
Activity 3         A configuration management library system       A configuration management library system     Strength
                   is established as a repository for the software is established as a repository for the
                   baselines.                                      software baselines.
Activity 4         The software work products to be placed        The software work products to be placed        Strength
                   under configuration management are             under configuration management are
                   identified.                                    identified.
Activity 5         Change requests and problem reports for all    Change requests and problem items/units        Strength
                   configuration items/risks are initiated,       are initiated, recorded, reviewed, approved,
                   recorded, reviewed, approved, and tracked      and tracked according to a procedure
                   according to a documented procedure.           documented in the SCM plan.
                                                                                                                          (continued)

                                           Page 70                                   GAO/AIMD-99-41 Customs Service Modernization
                                         Appendix III
                                         Results of Software Development Capability
                                         Maturity Model (SW-CMM) Evaluation for
                                         NCAP 0.1




                 Key practice                                    Finding                                        Rating
Activity 6       Changes to baselines are controlled             Changes to baselines are controlled            Strength
                 according to a documented procedure.            according to a documented procedure in
                                                                 the SCM plan.
Activity 7       Products from the software baseline library     Products from the software baseline library    Strength
                 are created, and their release is controlled    are created, and their release is controlled
                 according to a documented procedure.            according to the SCM plan.
Activity 8       The status of software configuration            The status of SCM items/units are not          Weakness
                 items/units is recorded according to a          recorded according to a documented
                 documented procedure.                           procedure.
Activity 9       Standard reports documenting SCM                Standard reports documenting SCM               Weakness
                 activities and the contents of the software     activities and contents of the software
                 baseline are developed and made available       baseline are not developed.
                 to affected groups and individuals.
Activity 10      Software baseline audits are conducted          Software baseline audits are not conducted.    Weakness
                 according to a documented procedure.
Measurement 1    Measurements are made and used to               No measurements are taken to determine the Weakness
                 determine the status of SCM activities.         status of SCM activities.
Verification 1   SCM activities are reviewed with senior         SCM activities are not reviewed with senior    Weakness
                 management periodically.                        management periodically.
Verification 2   SCM activities are reviewed with the project    SCM activities are not reviewed with the       Weakness
                 manager on both a periodic and                  project manager on both a periodic and
                 event-driven basis.                             event-driven basis.
Verification 3   SCM group periodically audits software          No SCM audits are done to verify that          Weakness
                 baselines to verify that they conform to the    software baselines conform to the
                 documentation that defines them.                documentation that defines them.
Verification 4   The software quality assurance group            There is no quality assurance group;           Weakness
                 reviews and/or audits the activities and work   therefore, no one reviews and/or audits the
                 products for SCM and reports the results.       activities and work products for SCM
                                                                 activities performed.




                                         Page 71                                    GAO/AIMD-99-41 Customs Service Modernization
Appendix IV

Results of Software Acquisition Capability
Maturity Model (SA-CMM) Evaluation for
NCAP 0.2

Table IV.1: Key Process Area: Software Acquisition Planning
                  Key practice                                       Finding                                            Rating
Commitment 1     The acquisition organization has a written policy There is no written policy for planning the          Weakness
                 for planning the software acquisition.            software acquisition.
Commitment 2     Responsibility for software acquisition planning    Responsibility for software acquisition planning   Weakness
                 activities is designated.                           is not designated.
Ability 1        The acquisition organization has experienced        The acquisition organization does not have         Weakness
                 software acquisition management personnel.          experienced software acquisition management
                                                                     personnel.
Ability 2        Adequate resources are provided for software        Resources for software acquisition planning are    Weakness
                 acquisition planning activities.                    not adequate.
Activity 1       Software acquisition planning personnel are         Software acquisition planning personnel are not    Weakness
                 involved in system acquisition planning.            involved in system acquisition planning.
Activity 2       The software acquisition strategy for the           A software acquisition strategy for NCAP 0.2       Weakness
                 project is developed and documented.                was not developed and documented.
Activity 3       The project’s software acquisition planning is      Software acquisition planning for NCAP 0.2 has     Weakness
                 documented, and the planning documentation          not been documented.
                 is maintained over the life of the project.
Activity 4       Life-cycle support of the software is included in   Life-cycle support of the software has not been    Weakness
                 software acquisition planning documentation.        included in any document.
Activity 5       Life-cycle cost and schedule estimates for the      Life-cycle cost and schedule estimates were        Observation
                 software products and services being acquired       prepared, but there is no evidence that they
                 are prepared and independently reviewed.            were independently reviewed.
Measurement 1    Measurements are made and used to determine Software acquisition planning activities are not           Weakness
                 the status of the software acquisition planning measured.
                 activities and resultant products.
Verification 1   Software acquisition planning activities are        Software acquisition planning activities are not   Weakness
                 reviewed by acquisition organization                reviewed by management.
                 management periodically.
Verification 2   Software acquisition planning activities are        There is no documented evidence that the           Weakness
                 reviewed by the project manager on both a           project manager reviewed software acquisition
                 periodic and event-driven basis.                    planning activities.




                                            Page 72                                     GAO/AIMD-99-41 Customs Service Modernization
                                               Appendix IV
                                               Results of Software Acquisition Capability
                                               Maturity Model (SA-CMM) Evaluation for
                                               NCAP 0.2




Table IV.2: Key Process Area: Solicitation
                 Key practice                                           Finding                                                Rating
Commitment 1     The acquisition organization has a written policy for There is no written policy for the conduct of the       Weakness
                 the conduct of the software portion of the            software portion of the solicitation.
                 solicitation.
Commitment 2     Responsibility for the software portion of the         Responsibility for the software portion of the      Strength
                 solicitation is designated.                            solicitation was designated to the project
                                                                        manager/contract officer’s technical representative
                                                                        (COTR).
Commitment 3     A selection official has been designated to be         A selection official has been designated.              Strength
                 responsible for the selection process and the
                 decision.
Ability 1        A group that is responsible for coordinating and       There is no documented evidence that a group           Weakness
                 conducting solicitation activities exists.             responsible for coordinating and conducting
                                                                        solicitation activities exists.
Ability 2        Adequate resources are provided for solicitation       Resources for solicitation activities are not          Weakness
                 activities.                                            adequate.
Ability 3        Individuals performing solicitation activities have    There is only one individual performing solicitation   Weakness
                 experience or receive training.                        activities. Although he was trained as a COTR, he
                                                                        has no experience or training in acquisition
                                                                        management or costing methodologies and tools.
Ability 4        The groups supporting the solicitation (e.g.,          No orientation is provided to any other groups. End Weakness
                 end user, systems engineering, software support        users, system engineers, and software support
                 organization, and application domain experts)          organizations did not participate in the solicitation.
                 receive orientation on the solicitation’s objectives
                 and procedures.
Activity 1       The project team performs its activities in        There are no documented solicitation plans to              Weakness
                 accordance with its documented solicitation plans. guide the performance of solicitation activities.
Activity 2       The project team performs its activities in            There are no documented proposal evaluation            Weakness
                 accordance with its documented proposal                plans.
                 evaluation plans.
Activity 3       Cost and schedule estimates for the software           Cost and schedule estimates for the software           Strength
                 products and services being sought are prepared.       products and services being sought are prepared.
Activity 4       Software cost and schedule estimates are               Software cost and schedule estimates are not           Weakness
                 independently reviewed for                             independently reviewed.
                 comprehensiveness and realism.
Activity 5       The project team takes action to ensure the mutual     The project team takes action to ensure the mutual     Strength
                 understanding of software requirements and plans       understanding of software requirements and plans
                 prior to contract award.                               prior to contract award.
Measurement 1    Measurements are made and used to                      No measurements were made to determine the             Weakness
                 determine the status of the solicitation activities    status of solicitation activities and resultant
                 and resultant products.                                products.
Verification 1   Solicitation activities are reviewed periodically by   Solicitation activities were not reviewed by the       Weakness
                 the designated selection official or acquisition       designated selection official or acquisition
                 organization management.                               organization management on a periodic basis.
Verification 2   Solicitation activities are reviewed by the project Solicitation activities are not reviewed by the           Weakness
                 manager on both a periodic and event-driven basis. project manager on both a periodic and
                                                                     event-driven basis.



                                               Page 73                                      GAO/AIMD-99-41 Customs Service Modernization
                                            Appendix IV
                                            Results of Software Acquisition Capability
                                            Maturity Model (SA-CMM) Evaluation for
                                            NCAP 0.2




Table IV.3: Key Process Area: Requirements Development and Management
                  Key practice                              Finding                                                     Rating
Commitment 1     The acquisition organization has a written policy There is no written policy for establishing and      Weakness
                 for establishing and managing the                 managing the software-related contractual
                 software-related contractual requirements.        requirements.
Commitment 2     Responsibility for requirements development        Responsibility for requirements development         Strength
                 and management is designated.                      and management is designated to the Process
                                                                    Analysis and Requirements Team (PART).
Ability 1        A group that is responsible for performing         The PART team is responsible for performing         Strength
                 requirements development and management            requirements development and management.
                 activities exists.
Ability 2        Adequate resources are provided for                The PART team has adequate resources for            Strength
                 requirements development and management            requirements development and management
                 activities.                                        activities.
Ability 3        Individuals performing requirements                Individuals performing requirements                 Strength
                 development and management activities have         development and management activities have
                 experience or receive training.                    experience or receive training.
Activity 1       The project team performs its activities in        There are no documented requirements                Weakness
                 accordance with its documented requirements        development and management plans.
                 development and management plans.
Activity 2       The project team develops and baselines the        The project team develops and baselines the         Strength
                 software-related contractual requirements and      software-related contractual requirements and
                 places them under change control early in the      places them under change control early in the
                 project, but not later than release of the         project, but not later than release of the
                 solicitation package.                              solicitation package.
Activity 3       The project team appraises system                  To date there have been no system                   Observation
                 requirements change requests for their impact      requirements change requests.
                 on the software being acquired.
Activity 4       The project team appraises all changes to the      To date there have been no changes to the           Observation
                 software-related contractual requirements for      software-related contractual requirements.
                 their impact on performance, architecture,
                 supportability, system resource utilization, and
                 contract schedule and cost.
Activity 5       Bidirectional traceability between the             Bidirectional traceability between the              Weakness
                 software-related contractual requirements and      software-related contractual requirements and
                 the contractor’s software work products and        the contractor’s software work products and
                 services is maintained throughout the effort.      services is not maintained throughout the effort.
Measurement 1    Measurements are made and used to determine Measurements are made and used to determine Strength
                 the status of the requirements development and the status of the requirements development and
                 management activities and resultant products.  management activities and resultant products.
Verification 1   Requirements development and management            Requirements development and management             Strength
                 activities are reviewed periodically by            activities are reviewed periodically by
                 acquisition organization management (and the       acquisition management (and the contractor).
                 contractor).
Verification 2   Requirements development and management            Requirements development and management             Weakness
                 activities are reviewed by the project manager     activities are not reviewed by the project
                 on both a periodic and event-driven basis.         manager.




                                            Page 74                                      GAO/AIMD-99-41 Customs Service Modernization
                                             Appendix IV
                                             Results of Software Acquisition Capability
                                             Maturity Model (SA-CMM) Evaluation for
                                             NCAP 0.2




Table IV.4: Key Process Area: Project Management
                  Key practice                                         Finding                                              Rating
Commitment 1     The acquisition organization has a written policy There is no written policy for execution of the          Weakness
                 for execution of the software project.            software project.
Commitment 2     Responsibility for project management is              Responsibility for project management has            Strength
                 designated.                                           been designated.
Ability 1        A team that is responsible for performing the         A team that is responsible for performing the        Strength
                 project’s software acquisition management             project’s software acquisition management
                 activities exists.                                    activities exists.
Ability 2        Adequate resources for the project team and           Adequate resources for the project team and          Weakness
                 matrix support persons are provided for the           matrix support persons are not provided for the
                 duration of the software acquisition project.         duration of the software acquisition project.
Ability 3        When project trade-offs are necessary, the            The project manager, who is also the                 Strength
                 project manager is permitted to alter either the      contracting officer’s technical representative, is
                 performance, cost, or schedule software               permitted to alter either the performance, cost,
                 acquisition baseline.                                 or schedule software acquisition baseline.
Ability 4        The project team and matrix support                   The project team members do not have                 Weakness
                 individual(s) have experience or receive training     experience and have not received training in
                 in project software acquisition management            project software acquisition management
                 activities.                                           activities.
Activity 1       The project team performs its activities in           There is no software acquisition management          Weakness
                 accordance with its documented software               plan; therefore, the project team does not
                 acquisition management plans.                         perform its activities in accordance with a plan.
Activity 2       The organization of the project provides for the      The organization of the project does not provide Weakness.
                 management of all project functions.                  for the management of all the project’s
                                                                       functions. For example, acquisition planning
                                                                       and risk management are not provided.
Activity 3       The software acquisition management activities        The software acquisition management activities       Weakness
                 of the project team are directed to accomplish        of the project team are not directed to
                 the project’s objectives.                             accomplish the project’s objectives.
Activity 4       The software acquisition management activities        The activities of the project team are not           Weakness.
                 of the project team are controlled.                   controlled.
Activity 5       The project team implements a corrective action       There is no corrective action system for the         Weakness
                 system for the identification, recording, tracking,   identification, recording, tracking, and
                 and correction of problems discovered during          correction of problems discovered during the
                 the software acquisition.                             software acquisition.
Activity 6       The project team tracks project status,        The project team tracks project status,                     Observation
                 execution, funding, and expenditures and takes execution, funding, and expenditures. At the
                 action.                                        time of our evaluation, however, no actions
                                                                were needed.
Measurement 1    Measurements are made and used to              Measurements are taken and used to determine Strength
                 determine the status of the project management the status of project management activities and
                 activities and resultant products.             resultant products.
Verification 1   Project management activities are reviewed by         Project management activities are not reviewed       Weakness
                 acquisition organization management                   by acquisition organization management
                 periodically.                                         periodically.
Verification 2   Project management activities are reviewed by         There was no evidence provided to show that          Weakness
                 the project manager on both a periodic and            the project management activities are reviewed
                 event-driven basis.                                   by the project manager.


                                             Page 75                                      GAO/AIMD-99-41 Customs Service Modernization
                                             Appendix IV
                                             Results of Software Acquisition Capability
                                             Maturity Model (SA-CMM) Evaluation for
                                             NCAP 0.2




Table IV.5: Key Process Area: Contract Tracking and Oversight Findings
                  Key practice                                  Finding                                                 Rating
Commitment 1     The acquisition organization has a written policy There is no organizational policy for the            Weakness
                 for the contract tracking and oversight of the    contract tracking and oversight of the
                 contracted software effort.                       contracted software.
Commitment 2     Responsibility for contract tracking and             Responsibility for contract tracking and          Strength
                 oversight activities is designated.                  oversight has been designated to the project
                                                                      manager/COTR.
Commitment 3     The project team is supported by contracting         The contracting officer supports the project      Strength
                 specialists in the execution of the contract.        team in the execution of the contract.

Ability 1        A group that is responsible for managing             The project team is responsible for contract      Strength
                 contract tracking and oversight activities exists.   tracking and oversight activities.
Ability 2        Adequate resources are provided for contract         Adequate resources are not provided for           Weakness
                 tracking and oversight activities.                   contract tracking and oversight activities.
Ability 3        Individuals performing contract tracking and         The project manager has received training         Strength
                 oversight activities have experience or receive      conducting his duties as a COTR.
                 training.
Activity 1       The project team performs its activities in          There is no contract tracking and oversight plan. Weakness
                 accordance with its documented contract
                 tracking and oversight plans.
Activity 2       The project team reviews required contractor         The contractor was not required to submit any     Weakness
                 software planning documents which, when              software planning documents that could be
                 satisfactory, are used to oversee the                used to oversee the contractor’s software
                 contractor’s software engineering effort.            engineering effort.
Activity 3       The project team conducts periodic reviews           The project team conducts periodic reviews and Strength
                 and interchanges with the contractor.                interchanges with the contractor.
Activity 4       The project team reviews and tracks the              Customs provided the software engineering         Not rated
                 development of the software engineering              environment to the contractor; therefore, there
                 environment required to provide life-cycle           was no need to track the development of the
                 support for the acquired software.                   software engineering environment required to
                                                                      provide life-cycle support for the acquired
                                                                      software.
Activity 5       Any problems or issues found by the project     Problems or issues found by the project team           Weakness
                 team during contract tracking and oversight are during contract tracking and oversight are not
                 recorded in the appropriate corrective action   tracked.
                 system and tracked to closure.
Activity 6       The project team maintains the integrity of the      The project team maintains the integrity of the   Strength
                 contract throughout the contract performance         contract throughout the contract performance
                 period.                                              period.
Measurement 1    Measurements are made and used to determine Measurement are not made to determine the                  Weakness
                 the status of the contract tracking and oversight status of contract tracking and oversight
                 activities and resultant products.                activities and resultant products.
Verification 1   Contract tracking and oversight activities are       Contract tracking and oversight activities are    Weakness
                 reviewed by acquisition organization                 not reviewed by the acquisition organization
                 management periodically.                             management.
Verification 2   Contract tracking and oversight activities are       Contract tracking and oversight activities are    Weakness
                 reviewed by the project manager on both a            not reviewed by the project manager.
                 periodic and event-driven basis.


                                             Page 76                                      GAO/AIMD-99-41 Customs Service Modernization
                                              Appendix IV
                                              Results of Software Acquisition Capability
                                              Maturity Model (SA-CMM) Evaluation for
                                              NCAP 0.2




Table IV.6: Key Process Area: Evaluation
                  Key practice                                         Finding                                               Rating
Commitment 1     The acquisition organization has a written            There is no written policy for managing the           Weakness
                 policy for managing the evaluation of the             evaluation of the acquired software products
                 acquired software products and services.              and services.
Commitment 2     Responsibility for evaluation activities is clearly   Responsibility for evaluation activities is clearly   Strength
                 designated.                                           designated.
Ability 1        A group that is responsible for planning,             A group that is responsible for planning,             Strength
                 managing, and performing evaluation activities        managing, and performing evaluation activities
                 for the project exists.                               for the project exists.
Ability 2        Adequate resources are provided for evaluation We found no evidence that a lack of resources                Observation
                 activities.                                    precluded the performance of evaluation
                                                                activities.
Ability 3        Individuals performing evaluation activities have Individuals performing evaluation activities have Strength
                 experience or receive training.                   experience or receive training.
Ability 4        Members of the project team and groups                No orientation is provided to members of the          Weakness
                 supporting the software acquisition receive           project team and groups supporting the
                 orientation on the objectives of the evaluation       software acquisition.
                 approach.
Activity 1       The project team performs its activities in           There are no documented evaluation plans.             Weakness
                 accordance with its documented evaluation
                 plans.
Activity 2       The project’s evaluation requirements are             The project’s evaluation requirements are not         Weakness
                 developed in conjunction with the development         developed in conjunction with the development
                 of the system or software technical                   of the system or software technical
                 requirements.                                         requirements.
Activity 3       The evaluation requirements are incorporated          Some evaluation requirements have been              Weakness
                 into the solicitation package and resulting           incorporated into the solicitation package and
                 contract.                                             resulting contract. Others, such as a
                                                                       mechanism that provides the project team
                                                                       visibility into the contractor’s evaluation program
                                                                       and requirements for the contractor to establish
                                                                       a corrective action system, have not.
Activity 4       The project team assesses contractor’s                The project team does not assess the                  Weakness
                 performance for compliance with evaluation            contractor’s performance for compliance with
                 requirements.                                         evaluation requirements.
Activity 5       Planned evaluations are performed on the              No products or services have yet been            Observation
                 acquired software products and services prior         accepted for operational use. The project team
                 to acceptance for operational use.                    plans to evaluate acquired software products
                                                                       and services prior to acceptance for operational
                                                                       use.
Activity 6       Results of the evaluations are analyzed and           No products or services have yet been                 Observation
                 compared to the contract’s requirements to            delivered by the contractor. The project team
                 establish an objective basis to support the           plans to analyze the results of the evaluations
                 decision to accept the products and services          and compare the results to contract
                 or to take further action.                            requirements as an objective basis to support
                                                                       the decision to accept the products and
                                                                       services or to take further action.
                                                                                                                                 (continued)



                                              Page 77                                      GAO/AIMD-99-41 Customs Service Modernization
                                            Appendix IV
                                            Results of Software Acquisition Capability
                                            Maturity Model (SA-CMM) Evaluation for
                                            NCAP 0.2




                 Key practice                                        Finding                                         Rating
Measurement 1    Measurements are made and used to determine No measurements are made to determine the               Weakness
                 the status of the evaluation activities and status of evaluation activities and resultant
                 resultant products.                         products.
Verification 1   Evaluation activities are reviewed by acquisition Evaluation activities are not reviewed by         Weakness
                 organization management periodically.             acquisition organization management.
Verification 2   Evaluation activities are reviewed by the project   Evaluation activities are not reviewed by the   Weakness
                 manager on both a periodic and event-driven         project manager.
                 basis.




                                            Page 78                                      GAO/AIMD-99-41 Customs Service Modernization
                                              Appendix IV
                                              Results of Software Acquisition Capability
                                              Maturity Model (SA-CMM) Evaluation for
                                              NCAP 0.2




Table IV.7: Key Process Area: Transition to Support
                  Key practice                                         Finding                                               Rating
Commitment 1     The acquisition organization has a written            There is no policy for the transitioning of           Weakness
                 policy for transitioning software products to the     software products to the software support
                 software support organization.                        organization.
Commitment 2     The acquisition organization ensures that the         A software support organization has not been          Weakness
                 software support organization is involved in          established.
                 planning for transition to support.
Commitment 3     Responsibility for transition to support activities   Responsibility for transition to support activities   Weakness
                 is designated.                                        has not been designated.
Ability 1        A group that is responsible for coordinating the      No group is responsible for coordinating              Weakness
                 transition to support activities exists.              transition to support activities.
Ability 2        Adequate resources are provided for transition        Adequate resources have not been provided             Weakness
                 to support activities.                                for transition to support activities.
Ability 3        The organization responsible for providing            The organization responsible for providing            Weakness
                 support of the software products is identified no     support of the software products was not
                 later than initiation of the solicitation package’s   identified before initiation of the solicitation
                 development.                                          package’s development.
Ability 4        The software support organization, prior to           At the time of the audit, the system was still in     Not rated
                 transition, has a complete inventory of all           development and had not reached the
                 software and related items that are to be             transition stage.
                 transitioned.
Ability 5        Individuals performing transition to support          Individuals responsible for transition to support     Weakness
                 activities have experience or receive training.       activities have not been selected.
Ability 6        The members of organizations interfacing with         The members of organizations interfacing with         Weakness
                 the transition to support activities receive          the transition to support activities have not
                 orientation on the salient aspects of transition to   received orientation on the salient aspects of
                 support activities.                                   transition to support activities.
Activity 1       The project team performs its activities in           There are no transition to support plans.             Weakness
                 accordance with its documented transition to
                 support plans.
Activity 2       Responsibility for the software products is           No products have been transferred yet.                Not rated
                 transferred only after the software support
                 organization demonstrates its capability to
                 modify and support the software products.
Activity 3       The project team oversees the configuration           Transition has not yet occurred.                      Not rated
                 control of the software products throughout the
                 transition.
Measurement 1    Measurements are made and used to                     There are no plans to take measurements to            Weakness
                 determine the status of the transition to support     determine the status of the transition to support
                 activities and resultant products.                    activities and resultant products.
Verification 1   Transition to support activities are reviewed by      There are no plans for acquisition and software       Weakness
                 acquisition and software support organizations’       support organizations’ managements to review
                 managements periodically.                             transition to support activities periodically.
Verification 2   Transition to support activities are reviewed by      No evidence was provided that transition to           Weakness
                 the project manager on both a periodic and            support activities are reviewed by the project
                 event-driven basis.                                   manager.




                                              Page 79                                       GAO/AIMD-99-41 Customs Service Modernization
                                             Appendix IV
                                             Results of Software Acquisition Capability
                                             Maturity Model (SA-CMM) Evaluation for
                                             NCAP 0.2




Table IV.8: Key Process Area: Acquisition Risk Management
                  Key practice                                      Finding                                           Rating
Commitment 1     The acquisition organization has a written         There is no policy for the management of          Weakness
                 policy for the management of software              software acquisition risk.
                 acquisition risk.
Commitment 2     Responsibility for software acquisition risk       Responsibility for software acquisition risk      Weakness
                 management activities is designated.               management activities is not designated.
Ability 1        A group that is responsible for coordinating       No group responsible for coordinating software    Weakness
                 software acquisition risk management activities    acquisition risk management activities exists.
                 exists.
Ability 2        Adequate resources are provided for software       Adequate resources are not provided for          Weakness
                 acquisition risk management activities.            software acquisition risk management activities.
Ability 3        Individuals performing software acquisition risk   No individual or group is performing software     Weakness
                 management activities have experience or           acquisition risk management.
                 receive required training.
Activity 1       Software acquisition risk management activities Software acquisition risk management activities      Weakness
                 are integrated into software acquisition planning. are not integrated into software acquisition
                                                                    planning.
Activity 2       The software acquisition risk management plan      There is no software acquisition risk             Weakness
                 is developed in accordance with the project’s      management plan.
                 defined software acquisition process.
Activity 3       The project team performs its software             No documented software acquisition risk           Weakness
                 acquisition risk management activities in          management plans exists. No risk management
                 accordance with its documented plans.              activities are being performed.
Activity 4       Risk management is conducted as an integral        Risk management is not conducted as an            Weakness
                 part of the solicitation, project performance      integral part of the solicitation, project
                 management, and contract performance               performance management, and contract
                 management processes.                              performance management processes.
Activity 5       Software acquisition risk handling actions are     There is no record of tracking and controlling    Weakness
                 tracked and controlled until the risks are         software acquisition risk handling actions.
                 mitigated.
Measurement 1    Measurements are made and used to                  Measurements are not made and used to             Weakness
                 determine the status of the acquisition risk       determine the status of the acquisition risk
                 management activities and resultant products.      management activities and resultant products.
Verification 1   Acquisition risk management activities are         There are no acquisition risk management          Weakness
                 reviewed by acquisition organization               activities for management to review.
                 management periodically.
Verification 2   Acquisition risk management activities are         There are no acquisition risk management          Weakness
                 reviewed by the project manager on both a          activities for the project manager to review.
                 periodic and event-driven basis.




                                             Page 80                                      GAO/AIMD-99-41 Customs Service Modernization
Appendix V

Major Contributors to This Report


                       Dr. Rona Stillman, Chief Scientist for Computers and
Accounting and           Telecommunications
Information            Jack L. Brock, Jr., Director, Governmentwide and Defense Information
Management Division,     Systems
                       Mark T. Bird, Assistant Director
Washington, D.C.       Madhav S. Panwar, Technical Assistant Director
                       Dr. Nabajyoti Barkakati, Technical Assistant Director
                       Prithviraj Mukerji, Assistant Director
                       Carl M. Urie, Assistant Director
                       Bernard R. Anderson, Senior Information Systems Analyst
                       Suzanne M. Burns, Senior Information Systems Analyst
                       Timothy D. Hopkins, Senior Information Systems Analyst
                       Ona M. Noble, Senior Information Systems Analyst
                       Karen A. Richey, Senior Information Systems Analyst
                       Cristina T. Chaplain, Communications Analyst


                       Harold J. Brumm, Economist
Office of the Chief
Economist,
Washington, D.C.
                       Teresa F. Tucker, Senior Information Systems Analyst
Atlanta Field Office




(511114)               Page 81                          GAO/AIMD-99-41 Customs Service Modernization
Ordering Information

The first copy of each GAO report and testimony is free.
Additional copies are $2 each. Orders should be sent to the
following address, accompanied by a check or money order
made out to the Superintendent of Documents, when
necessary. VISA and MasterCard credit cards are accepted, also.
Orders for 100 or more copies to be mailed to a single address
are discounted 25 percent.

Orders by mail:

U.S. General Accounting Office
P.O. Box 37050
Washington, DC 20013

or visit:

Room 1100
700 4th St. NW (corner of 4th and G Sts. NW)
U.S. General Accounting Office
Washington, DC

Orders may also be placed by calling (202) 512-6000
or by using fax number (202) 512-6061, or TDD (202) 512-2537.

Each day, GAO issues a list of newly available reports and
testimony. To receive facsimile copies of the daily list or any
list from the past 30 days, please call (202) 512-6000 using a
touchtone phone. A recorded menu will provide information on
how to obtain these lists.

For information on how to access GAO reports on the INTERNET,
send an e-mail message with "info" in the body to:

info@www.gao.gov

or visit GAO’s World Wide Web Home Page at:

http://www.gao.gov




PRINTED ON    RECYCLED PAPER
United States                       Bulk Rate
General Accounting Office      Postage & Fees Paid
Washington, D.C. 20548-0001           GAO
                                 Permit No. G100
Official Business
Penalty for Private Use $300

Address Correction Requested