oversight

Customs Service Modernization: Ineffective Software Development Processes Increase Customs System Development Risks

Published by the Government Accountability Office on 1999-02-11.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                 United States General Accounting Office

GAO              Report to Congressional Requesters




February 1999
                 CUSTOMS SERVICE
                 MODERNIZATION
                 Ineffective Software
                 Development
                 Processes Increase
                 Customs System
                 Development Risks




GAO/AIMD-99-35
      United States
GAO   General Accounting Office
      Washington, D.C. 20548

      Accounting and Information
      Management Division

      B-280417

      February 11, 1999

      The Honorable Ben Nighthorse Campbell
      Chairman, Subcommittee on Treasury,
        General Government, and Civil Service
      Committee on Appropriations
      United States Senate

      The Honorable Jim Kolbe
      Chairman, Subcommittee on Treasury,
        Postal, and General Government
      Committee on Appropriations
      House of Representatives

      This report responds to your requests that we review the U. S. Customs Service’s software
      development maturity and improvement activities. Customs plans to spend hundreds of millions
      of dollars developing and maintaining software-intensive systems. We found that Customs’
      software development processes are immature and are making recommendations to the
      Commissioner of Customs for strengthening them.

      We are sending copies of this report to the Secretary of the Treasury; Commissioner of
      Customs; Director of the Office of Management and Budget; Ranking Minority Members of the
      Subcommittee on Treasury and General Government of the Senate Committee on
      Appropriations and the Subcommittee on Treasury, Postal Service, and General Government of
      the House Committee on Appropriations; and other congressional committees. We will also
      make copies available to other interested parties upon request. If you have questions or wish to
      discuss the issues in this report, please contact me at (202) 512-6240. Major contributors to this
      report are listed in appendix II.




      Randolph C. Hite
      Associate Director, Governmentwide
      and Defense Information Systems
Executive Summary


             The Customs Service relies extensively on software intensive systems to
Purpose      perform its core missions of enforcing laws governing the flow of goods
             and persons across U.S. borders, and assessing and collecting billions of
             dollars annually in duties, taxes, and fees on imported merchandise.
             Because software is a complex and expensive component of these
             systems, Customs must use defined and disciplined processes if it expects
             to develop and maintain software effectively.

             Recognizing software’s importance to Customs’ mission effectiveness, the
             Chairman, Subcommittee on Treasury and General Government, Senate
             Committee on Appropriations, and the Chairman, Subcommittee on
             Treasury, Postal Service and General Government, House Committee on
             Appropriations, asked GAO to determine (1) the maturity of Customs’
             software development processes, and (2) whether Customs has an
             effective software process improvement program.


             During fiscal year 1997, Customs collected $22.1 billion in revenue at more
Background   than 300 ports of entry, and it processed nearly 450 million passengers
             who entered the U.S. during the year. Each year Customs also provides
             trade statistics used in developing trade policy and negotiating trade
             agreements with various countries. Customs expects its workload to
             burgeon. Accordingly, Customs plans to spend hundreds of millions of
             dollars over the next 5 years developing software for new and existing
             information systems.

             Software quality is governed largely by the quality of the processes
             involved in developing or acquiring, and maintaining it. Carnegie Mellon
             University’s Software Engineering Institute (SEI), recognized for its
             expertise in software processes, has developed models and methods that
             define and determine organizations’ software process maturity. Together,
             they provide a logical framework for baselining an organization’s current
             process capabilities (i.e., strengths and weaknesses) and providing a
             structured plan for incremental process improvement.

             Using SEI’s Software Capability Maturity ModelSM (SW-CMM)1 and the SEI
             software capability evaluation method2, GAO staff trained at SEI evaluated
             Customs’ software development/maintenance maturity in five of the six
             key process areas (KPA) that are necessary to attain a “repeatable” level of

             1
              Capability Maturity ModelSM is the service mark of Carnegie Mellon University, and CMM is
             registered in the U.S. Patent and Trademark Office.
             2
              We used the latest version (version 1.1) of the software development CMM for our evaluation.



             Page 2                                         GAO/AIMD-99-35 Customs Service Modernization
                   Executive Summary




                   process maturity.3 GAO did not evaluate Customs in the sixth repeatable
                   level KPA, software subcontract management, because Customs did not use
                   subcontractors on any of the projects that GAO evaluated.4 An organization
                   at the repeatable level of process maturity has the necessary process
                   discipline in place to repeat earlier successes on projects in similar
                   applications. The repeatable level of process maturity is the second level
                   on SEI’s five-level scale. Organizations that do not satisfy the requirements
                   for the “repeatable” level are by default judged to be at the “initial” level of
                   maturity, meaning that their processes are ad hoc, sometimes even
                   chaotic, with few of the processes defined and success dependent mainly
                   on the heroic efforts of individuals. To aid such organizations in maturing
                   their processes, SEI has also published a software process improvement
                   model called IDEALSM5 that defines a systematic, five-phase process
                   improvement approach.6

                   As part of its evaluation, GAO examined three software projects. These
                   were (1) development of the first software release of the National Customs
                   Automation Program (NCAP 0.1), which is the first phase of the Automated
                   Commercial Environment (ACE); (2) software maintenance on the
                   Automated Export System (AES); and (3) software maintenance on the
                   Administrative Security System.7


                   Because of the number and severity of Customs’ software development
Results in Brief   process weaknesses, Customs did not fully satisfy any of the key process

                   3
                    The five KPAs are requirements management, software project planning, software project tracking
                   and oversight, software quality assurance, and software configuration management. According to the
                   SW-CMM, requirements management is the process for establishing a common understanding between
                   the customer and the software developer of the customer’s requirements; software project planning is
                   the process for establishing reasonable plans for engineering the software and managing the software
                   project; software project tracking and oversight is the process of providing adequate visibility into the
                   software project’s progress to permit effective action when deviations from plans occur; software
                   quality assurance is the process of verifying for management that software project plans, standards,
                   and procedures are being followed; and software configuration management is the process of
                   establishing and maintaining the integrity of the software products throughout the project’s life cycle.
                   4
                     According to the SW-CMM, software subcontract management is the process of selecting a software
                   subcontractor, establishing commitments with the subcontractor, and tracking and reviewing the
                   subcontractor’s performance and results.
                   5
                    IDEALSM is a service mark of Carnegie Mellon University.
                   6
                    IDEAL: A User’s Guide for Software Process Improvement (CMU/SEI-96-HB-001).
                   7
                    The Subcommittee Chairmen requested that we evaluate ACE, which is the largest and most
                   important system that Customs is developing. The other two projects were selected by Customs on the
                   basis of the following GAO specified criteria: each project should be managed by a different software
                   team, at least one project should involve a legacy system, at least one project should involve Year 2000
                   software conversion, and each project should be relatively large and important to accomplishing
                   Customs’ mission.



                   Page 3                                           GAO/AIMD-99-35 Customs Service Modernization
                        Executive Summary




                        areas (KPAs) necessary to achieve the “repeatable” level of process
                        maturity. As a result, its processes for developing software, a complex and
                        expensive component of Customs’ systems, are ad hoc, sometimes
                        chaotic, and not repeatable across projects.

                        Customs had some practice strengths in all but one of the five KPAs
                        evaluated (i.e., requirements management, software project planning,
                        software project tracking and oversight, software quality assurance, and
                        software configuration management); however, GAO also found extensive
                        and significant weaknesses in each of these KPAs. Some of these
                        weaknesses were systemic, recurring in each of the KPAs. For example,
                        Customs had no written policy for managing or implementing any of the
                        KPAs; and none of the projects had (1) an approved quality assurance plan;
                        (2) documented procedures for determining the project cost, schedule, or
                        effort; or (3) any outside group reviewing or reporting on the project’s
                        compliance with defined processes. These weaknesses are some of the
                        reasons for Customs’ limited success, for example, in delivering promised
                        ACE capabilities on time.


                        Currently, Customs does not have a software development process
                        improvement program, and it has not taken the basic steps to initiate one.
                        These steps, many of which are described in SEI’s IDEAL8 model for
                        process improvement, include assigning responsibility and authority for
                        process improvement, establishing a process improvement management
                        structure, defining a plan of action, and committing needed resources.
                        Until Customs establishes an effective process improvement program, its
                        software processes will remain poorly defined and undisciplined, and its
                        software projects are likely to suffer cost, schedule, and performance
                        shortfalls.


Principal Findings

Customs Software        GAO evaluated how effectively each of the three Customs software projects
Development Processes   implemented five of the six level 2 KPAs: requirements management,
Are Immature            software project planning, software project tracking and oversight,
                        software quality assurance, and software configuration management. To
                        attain a level 2 or repeatable maturity rating, Customs would have to
                        effectively implement all of the key practices for all five relevant KPAs.


                        8
                         IDEAL stands for the five phases of the model—Initiating, Diagnosing, Establishing, Acting, and
                        Leveraging.



                        Page 4                                          GAO/AIMD-99-35 Customs Service Modernization
                                     Executive Summary




                                     GAO found that while Customs had some strengths (i.e., practices that are
                                     effectively implemented), it had too many weaknesses (i.e., practices that
                                     are ineffectively implemented) to satisfy any of the level 2 KPAs. The
                                     strengths and weaknesses for all three projects are tallied in table 1. In
                                     summary, of the total number of KPA practices rated, 35 percent
                                     constituted strengths, 61 percent were weaknesses, and 4 percent were
                                     observations. An observation indicates that the evidence was inconclusive
                                     and did not clearly support a determination of either strength or
                                     weakness. To reach the repeatable level of maturity, Customs must
                                     eliminate the key practice weaknesses identified in this report.9

Table 1: Collective Number of KPA
Strengths, Weaknesses, and                                                                  Number of   Number of   Number of
Observations on the Three Projects   Key process area                                        strengths weaknesses observations
                                     Requirements management                                          23               13                 0
                                     Software project planning                                        32               38                 5
                                     Software project tracking and oversight                          28               40                 4
                                     Software quality assurance                                        3               46                 2
                                     Software configuration management                                17               46                 0
                                     Total                                                           103              183                11

                                     Also, GAO found that while the three projects varied as to the number of
                                     key practice strengths, weaknesses, and observations under each of the
                                     five “common features” or practice groupings (commitment to perform,
                                     ability to perform, activities performed, measurement and analysis, and
                                     verification of implementation), the NCAP 0.1 project displayed a better
                                     strengths to weaknesses ratio across all KPAs (about 1:1) than either AES or
                                     the Administrative Security System (about 1:2 and 1:4, respectively). By
                                     increasing its software maturity, Customs will reduce both the number of
                                     weaknesses on individual projects and the variability in process discipline
                                     among projects.


Customs Does Not Have a              SEI’s IDEAL model for software process improvement defines five
Software Process                     sequential phases. The phases are (1) initiating the program, including
Improvement Program                  assigning organizational roles and responsibility, establishing a program
                                     management structure, developing an action plan, and allocating needed
                                     resources; (2) assessing the organization’s current level of process
                                     maturity; (3) establishing a plan for addressing the identified process
                                     weaknesses; (4) developing and implementing new or improved processes;

                                     9
                                      SEI groups each of its KPA practices into one of five “common features” or practice attributes. These
                                     are “commitment to perform, ability to perform, activities performed, measurement and analysis, and
                                     verifying implementation.”



                                     Page 5                                          GAO/AIMD-99-35 Customs Service Modernization
                      Executive Summary




                      and (5) learning from process improvement experiences to strengthen the
                      program.

                      In 1996, Customs initiated efforts to improve its software processes. As
                      part of this effort, Customs had a contractor assess its software
                      development capabilities and develop a process improvement plan.
                      Customs then established process improvement teams for strengthening
                      two level-2 KPAs (software project planning and project tracking and
                      oversight).

                      In 1997, Customs discontinued its process improvement program, deciding
                      at that time to redirect its process improvement resources to Year 2000
                      conversion. As a result, Customs’ software development capability, which
                      is fundamental to its ability to effectively develop new systems like ACE,
                      and maintain existing systems like AES, remains ad hoc and undisciplined.
                      Customs officials stated their commitment to using the results of GAO’s
                      software capability evaluation to baseline its strengths and weaknesses
                      and address its weaknesses, but Customs has not yet established a
                      software process improvement program. In particular, it has not assigned
                      organizational responsibility and authority for process improvement,
                      established a program management structure, defined a plan of action, or
                      committed the necessary resources (trained staff and funding) to execute
                      the plan.


                      GAO recommends that, after ensuring that its mission-critical systems are
Recommendations       Year 2000 compliant, but before investing in major software development
                      efforts like ACE, the Commissioner of Customs direct the Customs Chief
                      Information Officer to:

                  •   assign responsibility and authority for software process improvement;
                  •   develop and implement a formal plan for software development process
                      improvement that is based on the software capability evaluation results
                      contained in this report and specifies measurable goals and time frames,
                      prioritizes initiatives, estimates resource requirements (trained staff and
                      funding), and defines a process improvement management structure;
                  •   ensure that every new software development effort in Customs adopts
                      processes that satisfy at least SW-CMM level 2 requirements; and
                  •   ensure that process improvement activities are initiated for all ongoing
                      essential software maintenance projects.




                      Page 6                              GAO/AIMD-99-35 Customs Service Modernization
                       Executive Summary




                       In its written comments on a draft of this report, Customs acknowledged
Agency Comments        the importance of software process improvement and maturity. Also, it
and GAO’s Evaluation   agreed with GAO’s overall findings, including that Customs’ software
                       development processes have not attained SW-CMM level 2 maturity.

                       Customs stated that it has taken the first step toward implementing GAO’s
                       recommendations by assigning responsibility and authority for software
                       process improvement as part of a reorganization of the Office of
                       Information and Technology that it plans to implement in early 1999.
                       Customs commented that once the reorganization is implemented, a
                       formal software process improvement program will be established,
                       including definition of an action plan, commitment of resources, and
                       specification of goals for achieving CMM levels 2 and 3. According to
                       Customs, these improvement activities are in their early stages. When they
                       are successfully implemented, they should address many of GAO’s
                       recommendations.

                       Customs also stated that, because its legacy systems are aging and need to
                       be enhanced and replaced, software process improvement must occur in
                       parallel with continued software development investments. History has
                       shown that attempting to modernize without first instituting disciplined
                       software processes has been a characteristic of failed modernization
                       programs.10 Until it implements disciplined software processes (at least
                       level 2 process maturity), Customs cannot prudently manage major
                       software investments, such as ACE with an estimated life cycle cost
                       exceeding $1 billion.

                       In its comments, Customs also asked to meet with GAO to discuss
                       system-specific KPA practice strength and weakness determinations. GAO
                       met with Customs prior to requesting comments on a draft of this report to
                       discuss system-specific determinations, and then again after receiving
                       Customs’ comments to ensure that all findings were clear. GAO is prepared
                       to continue assisting Customs as it improves its software processes.

                       Customs provided other comments on a draft of this report. Each of
                       Customs’ comments, along with GAO’s responses, is discussed in detail in
                       chapter 8 and appendix I of this report.



                       10
                        Tax Systems Modernization: Management and Technical Weaknesses Must Be Corrected If
                       Modernization Is To Succeed (GAO/AIMD-95-156, July 26, 1995), Tax Systems Modernization: Actions
                       Underway But IRS Has Not Yet Corrected Management and Technical Weaknesses (GAO/AIMD-96-106,
                       June 7, 1996), and Air Traffic Control: Immature Software Acquisition Processes Increase FAA System
                       Acquisition Risks (GAO/AIMD-97-47, March 21, 1997).



                       Page 7                                        GAO/AIMD-99-35 Customs Service Modernization
Contents



Executive Summary                                                                                2


Chapter 1                                                                                       12
                    Current Projects: A Brief Description                                       14
Introduction        Objectives, Scope, and Methodology                                          15


Chapter 2                                                                                       21
                    Customs’ Requirements Management Process Does Not Satisfy                   21
Requirements          SEI’s Criteria
Management          Conclusions                                                                 26


Chapter 3                                                                                       27
                    Customs Is Not Performing Software Project Planning Effectively             27
Software Project    Conclusions                                                                 36
Planning
Chapter 4                                                                                       37
                    Customs’ Software Project Tracking and Oversight Process Is                 37
Software Project      Immature
Tracking and        Conclusions                                                                 45
Oversight
Chapter 5                                                                                       46
                    Customs Lacks a Software Quality Assurance Process                          46
Software Quality    Conclusions                                                                 53
Assurance
Chapter 6                                                                                       54
                    Customs’ Software Configuration Management Process Is                       54
Software              Immature
Configuration       Conclusions                                                                 62
Management




                    Page 8                             GAO/AIMD-99-35 Customs Service Modernization
                       Contents




Chapter 7                                                                                         63
                       SEI Has Defined a Five-Phase Model for Software Process                    63
Customs Lacks a          Improvement
Software Process       Customs Does Not Currently Have a Software Process                         64
                         Improvement Program
Improvement            Conclusions                                                                65
Program
Chapter 8                                                                                         66
                       Recommendations                                                            66
Overall Conclusions,   Agency Comments and Our Evaluation                                         66
Recommendations,
and Agency
Comments
Appendixes             Appendix I: Comments From the Department of the Treasury                   68
                       Appendix II: Major Contributors to This Report                             71


Tables                 Table 1: Collective Number of KPA Strengths, Weaknesses, and                5
                         Observations on the Three Projects
                       Table 1.1: SW-CMM KPAs Used to Assess Customs’ Software                    18
                         Development Maturity
                       Table 2.1: Requirements Management                                         22
                       Table 2.2: Requirements Management Findings for NCAP 0.1                   23
                       Table 2.3: Requirements Management Findings for AES                        24
                       Table 2.4: Requirements Management Findings for Administrative             25
                         Security System
                       Table 3.1: Software Project Planning                                       28
                       Table 3.2: Software Project Planning Findings for NCAP 0.1                 30
                       Table 3.3: Software Project Planning Findings for AES                      32
                       Table 3.4: Software Project Planning Findings for Administrative           34
                         Security System
                       Table 4.1: Software Project Tracking and Oversight Summary                 38
                       Table 4.2: Software Project Tracking and Oversight Findings for            40
                         NCAP 0.1
                       Table 4.3: Software Project Tracking and Oversight Findings for            42
                         AES
                       Table 4.4: Software Project Tracking and Oversight Findings for            44
                         Administrative Security System
                       Table 5.1: Software Quality Assurance                                      47




                       Page 9                            GAO/AIMD-99-35 Customs Service Modernization
         Contents




         Table 5.2: Software Quality Assurance Findings for NCAP 0.1                48
         Table 5.3: Software Quality Assurance Findings for AES                     50
         Table 5.4: Software Quality Assurance Findings for                         52
           Administrative Security System
         Figure 6.1: Software Configuration Management                              55
         Table 6.2: Software Configuration Management Findings for                  57
           NCAP 0.1
         Table 6.3: Software Configuration Management Findings for AES              59
         Table 6.4: Software Configuration Management Findings for                  61
           Administrative Security System


Figure   Figure 1.1: SW-CMM Levels and Descriptions                                 17




         Abbreviations

         ACE        Automated Commercial Environment
         ACS        Automated Commercial System
         AES        Automated Export System
         CMM       Capability Maturity ModelSM
         GAO        General Accounting Office
         IDEALSM    Initiating, Diagnosing, Establishing, Acting, Leveraging
         KPA        key process area
         NAFTA      North American Free Trade Agreement
         NCAP       National Customs Automation Program
         NCAP/P     National Customs Automation Program Prototype
         RM         requirements management
         SCE        Software Capability Evaluation
         SCM        software configuration management
         SDLC       Software Development Life Cycle
         SEI        Software Engineering Institute
         SEPG       Software Engineering Process Group
         SPP        software project planning
         SPTO       software project tracking and oversight
         SQA        software quality assurance
         SW-CMM     Software Capability Maturity Model


         Page 10                           GAO/AIMD-99-35 Customs Service Modernization
Page 11   GAO/AIMD-99-35 Customs Service Modernization
Chapter 1

Introduction


                   The mission of the Customs Service is to ensure that all goods and persons
                   entering and exiting the United States do so in compliance with all U.S.
                   laws and regulations. It does this by (1) enforcing the laws governing the
                   flow of goods and persons across the borders of the United States and
                   (2) assessing and collecting duties, taxes, and fees on imported
                   merchandise. During fiscal year 1997, Customs collected $22.1 billion in
                   revenue1 at more than 300 ports of entry and reported that it processed
                   nearly 450 million passengers who entered the United States during the
                   year.

                   To accomplish its mission, Customs is organized into six lines of
                   business—trade compliance, outbound, passenger, finance, human
                   resources, and investigations. Each business area is described below.

               •   Trade compliance includes enforcement of laws and regulations
                   associated with the importation of goods into the United States. To do so,
                   Customs (1) works with the trade community to promote understanding of
                   applicable laws and regulations, (2) selectively examines cargo to ensure
                   that only eligible goods enter the country, (3) reviews documentation
                   associated with cargo entries to ensure that they are properly valued and
                   classified, (4) collects billions of dollars annually in duties, taxes, and fees
                   associated with imported cargo, (5) assesses fines and penalties for
                   noncompliance with trade laws and regulations, (6) seizes and accounts
                   for illegal cargo, and (7) manages the collection of these moneys to ensure
                   that all trade-related debts due to Customs are paid and properly
                   accounted for.
               •   Outbound includes Customs enforcement of laws and regulations
                   associated with the movement of merchandise and conveyances from the
                   United States. To do so, Customs (1) selectively inspects cargo at U.S.
                   ports to guard against the exportation of illegal goods, such as protected
                   technologies, stolen vehicles, and illegal currency, (2) collects,
                   disseminates, and uses intelligence to identify high-risk cargo and
                   passengers, (3) seizes and accounts for illegal cargo, (4) assesses and
                   collects fines and penalties associated with the exportation of illegal
                   cargo, and (5) physically examines baggage and cargo at airport facilities
                   for explosive and nuclear materials. In addition, the outbound business
                   includes collecting and disseminating trade data within the federal
                   government. Accurate trade data are crucial to establishing accurate trade
                   statistics on which to base trade policy decisions and negotiate trade
                   agreements with other countries. By the year 2000, Customs estimates that


                   1
                    Includes tariff duty, user fees, Internal Revenue Service excise taxes, and other assessments.



                   Page 12                                          GAO/AIMD-99-35 Customs Service Modernization
    Chapter 1
    Introduction




    exports will be valued at $1.2 trillion, compared to a reported $696 million
    in 1994.
•   Passenger includes processing all passengers and crew of arriving and
    departing (1) air and sea conveyances and (2) land vehicles and
    pedestrians. In fiscal year 1997, Customs reported it processed nearly
    450 million travelers and, by the year 2000, expects almost 500 million
    passengers to arrive in the United States annually. Many of Customs’
    passenger activities focus on illegal immigration and drug smuggling and
    are coordinated with other federal agencies, such as the Immigration and
    Naturalization Service and the Department of Agriculture’s Animal and
    Plant Health Inspection Service. Activities include targeting high-risk
    passengers, which requires timely and accurate information, and
    physically inspecting selected passengers, baggage, and vehicles to
    determine compliance with laws and regulations.
•   Finance includes asset and revenue management activities. Asset
    management consists of activities to (1) formulate Customs’ budget,
    (2) properly allocate and distribute funds, and (3) acquire, manage, and
    account for personnel, goods, and services. Revenue management
    encompasses all Customs activities to identify and establish amounts
    owed Customs, collect these amounts, and accurately report the status of
    revenue from all sources. Sources of revenue include duties, fees, taxes,
    other user fees, and forfeited currency and property. The revenue
    management activities interrelate closely with the revenue collection
    activities in the trade compliance, outbound, and passenger business
    areas.
•   Human resources is responsible for filling positions, providing employee
    benefits and services, training employees, facilitating workforce
    effectiveness, and processing personnel actions for Customs’ 18,000
    employees and managers.
•   Investigations includes activities to detect and eliminate narcotics and
    money laundering operations. Customs works with other agencies and
    foreign governments to reduce drug-related activity by interdicting
    (seizing and destroying) narcotics, investigating organizations involved in
    drug smuggling, and deterring smuggling efforts through various other
    methods. Customs also develops and provides information to the trade
    and carrier communities to assist them in their efforts to prevent
    smuggling organizations from using cargo containers and commercial
    conveyances to introduce narcotics into the United States.

    To carry out its responsibilities, Customs relies on information systems
    and processes to assist its staff in (1) documenting, inspecting, and
    accounting for the movement and disposition of imported goods and



    Page 13                             GAO/AIMD-99-35 Customs Service Modernization
                      Chapter 1
                      Introduction




                      (2) collecting and accounting for the related revenues. Customs expects its
                      reliance on information systems to increase as a result of its burgeoning
                      workload. For 1995 through 2001, Customs estimates that the annual
                      volume of import trade between the United States and other countries will
                      increase from $761 billion to $1.1 trillion. This will result in Customs
                      processing an estimated increase of 7.5 million commercial entries—from
                      13.1 million to 20.6 million annually—during the same period. Recent trade
                      agreements, such as the North American Free Trade Agreement (NAFTA),
                      have also increased the number and complexity of trade provisions that
                      Customs must enforce.

                      Customs recognizes that its ability to process the growing volume of
                      imports while improving compliance with trade laws depends heavily on
                      successfully modernizing its trade compliance process and its supporting
                      automated systems. To speed the processing of imports and improve
                      compliance with trade laws, the Congress enacted legislation2 that
                      eliminated certain legislatively mandated paper requirements and required
                      Customs to establish the National Customs Automation Program (NCAP).
                      The legislation also specified certain functions that NCAP must provide,
                      including giving members of the trade community the capability to
                      electronically file import entries at remote locations and enabling Customs
                      to electronically process “drawback” claims.3 In response to the
                      legislation, Customs began in 1994 to modernize the information systems
                      that support operations.


                      Customs has several projects underway to develop and acquire new
Current Projects: A   software and evolve (i.e., maintain) existing software to support its six
Brief Description     business areas. Customs’ fiscal year 1998 budget for information
                      management and technology activities was about $147 million.

                      Customs’ major information technology effort is its Automated
                      Commercial Environment (ACE) system. In 1994, Customs began to
                      develop ACE to replace its existing automated import system, the
                      Automated Commercial System (ACS). ACE is intended to provide an
                      integrated, automated information system for collecting, disseminating,
                      and analyzing import-related data and ensuring the proper collection and
                      allocation of revenues, totaling about $19 billion annually. According to


                      2
                       North American Free Trade Agreement Implementation Act, Public Law 103-182, 19 U.S.C. 1411 et seq.
                      3
                       Drawbacks are refunds of duties and taxes paid on imported goods that are subsequently exported or
                      destroyed.



                      Page 14                                       GAO/AIMD-99-35 Customs Service Modernization
                     Chapter 1
                     Introduction




                     Customs, ACE is planned to automate critical functions that the Congress
                     specified when it established NCAP.

                     Customs reported that it spent $47.8 million on ACE as of the end of fiscal
                     year 1997. In November 1997, Customs estimated it would cost
                     $1.05 billion to develop, operate, and maintain ACE over the 15 years from
                     fiscal year 1994 through fiscal year 2008. Customs plans to deploy ACE to
                     more than 300 ports that handle commercial cargo imports.

                     Customs plans to develop and deploy ACE in multiple phases. According to
                     Customs, the first phase, known as NCAP, is an ACE prototype. Customs
                     currently plans to deploy NCAP in four releases. The first release was
                     deployed for field evaluation at three locations in May 1998,4 and the
                     fourth is scheduled for 1999. Customs, however, has not adhered to
                     previous NCAP deployment schedules. Specifically, implementation of the
                     NCAP prototype slipped from January 1997 to August 1997 and then again to
                     a series of four releases beginning in October 1997, with the fourth release
                     starting in June 1998.

                     Customs also has several other projects underway to modify or enhance
                     existing systems that support its six business areas. For example, in fiscal
                     year 1998, Customs planned to spend about $3.7 million to enhance its
                     Automated Export System (AES), which supports the outbound business
                     area and is designed to improve Customs’ collection and reporting of
                     export statistics and to enforce export regulations. In addition, Customs
                     planned to spend another $4.6 million to maintain its administrative
                     systems supporting its finance and human resource business areas.


                     The Chairman, Subcommittee on Treasury and General Government,
Objectives, Scope,   Senate Committee on Appropriations, and the Chairman, Subcommittee
and Methodology      on Treasury, Postal Service and General Government, House Committee
                     on Appropriations, requested that we review Customs’ ability to develop
                     software for its computer systems. Our objectives were to determine
                     (1) the maturity of Customs’ software development processes and (2) the
                     effectiveness of Customs’ software process improvement program.

                     To determine Customs’ software development process maturity, we
                     applied the Software Engineering Institute’s (SEI) Software Capability



                     4
                     The first release of the NCAP prototype was deployed to Detroit, Michigan; Laredo, Texas; and Port
                     Huron, Michigan.



                     Page 15                                        GAO/AIMD-99-35 Customs Service Modernization
Chapter 1
Introduction




Maturity ModelSM (SW-CMM)5 and its Software Capability Evaluation (SCE)
method. SEI’s expertise in software process maturity as well as its
capability maturity models and evaluation methods are widely accepted
throughout the software industry. All our specialists were SEI-trained.

The SW-CMM ranks organizational maturity according to five levels. (See
figure 1.1.) Maturity levels 2 through 5 require the verifiable existence and
use of certain software development processes, known as key process
areas (KPA). According to SEI, an organization that has these processes in
place is in a much better position to successfully develop software than an
organization that does not have these processes in place. We evaluated
Customs’ software development processes against five of the six level 2
KPAs.




5
 Capability Maturity ModelSM is the service mark of Carnegie Mellon University, and CMM is
registered in the U.S. Patent and Trademark Office.



Page 16                                       GAO/AIMD-99-35 Customs Service Modernization
                                            Chapter 1
                                            Introduction




Figure 1.1: SW-CMM Levels and Descriptions

                                                                                   Level 5 - Optimizing
                                     Continuously
                                                                       Continuous process improvement is
                                       Improving
                                                                       enabled by quantitative feedback from the
                                          Process                      process and from piloting innovative ideas
                                                                       and technologies.



                                                                           Level 4 - Managed
                          Predictable                         Detailed measures of the software process
                             Process                          and product quality are collected. Both the
                                                              software process and products are
                                                              quantitatively understood and controlled.




                                                               Level 3 - Defined
               Standard
                                                 The software process for both management
              Consistent                         and engineering activities is documented,
                Process                          standardized, and integrated into a standard
                                                 software process for the organization. All
                                                 projects use an approved, tailored version of
                                                 the organization's standard software process
                                                 for developing and maintaining software.




                                             Level 2 - Repeatable
 Disciplined
                              Basic project management processes are
    Process                   established to track cost, schedule, and
                              functionality. The necessary process discipline
                              is in place to repeat earlier successes on
                              projects with similar applications.



                             Level 1 - Initial
                The software process is characterized as ad
                hoc, and occasionally even chaotic. Few
                processes are defined, and success depends
                on individual effort.




                                            Page 17                                      GAO/AIMD-99-35 Customs Service Modernization
                                     Chapter 1
                                     Introduction




                                     The sixth level 2 KPA, software subcontract management, was not
                                     evaluated because Customs did not use subcontractors on any of the
                                     projects that we evaluated. (See table 1.1.)

Table 1.1: SW-CMM KPAs Used to
Assess Customs’ Software             CMM Level 2 KPAs             Summary description
Development Maturity                 Requirements management      Defining, validating, and prioritizing requirements, such
                                                                  as functions, performance, and delivery dates.
                                     Software project planning    Developing estimates for the work to be performed,
                                                                  establishing the necessary commitments, and defining
                                                                  the plan to perform the work.
                                     Software project tracking and Tracking and reviewing software accomplishments and
                                     oversight                     results against documented estimates, commitments, and
                                                                   plans and adjusting these based on the actual
                                                                   accomplishments and results.
                                     Software quality assurance   Reviewing and auditing the software products and
                                                                  activities to ensure that they comply with the applicable
                                                                  processes, standards, and procedures and providing the
                                                                  staff and managers with the results of their reviews and
                                                                  audits.
                                     Software configuration       Selecting project baseline items, such as specifications;
                                     management                   systematically controlling these items and changes to
                                                                  them; and recording and reporting status and change
                                                                  activity for these items.

                                     As established by the model, each KPA contains five common attributes
                                     that indicate whether the implementation and institutionalization of a KPA
                                     can be effective, repeatable, and lasting. The five common features are:

                                 •   Commitment to perform: The actions that the organization must take to
                                     establish the process and ensure that it can endure. Commitment to
                                     perform typically involves establishing organizational policies and senior
                                     management sponsorship.
                                 •   Ability to perform: The preconditions that must exist in the project or
                                     organization to implement the software development process competently.
                                     Ability to perform typically involves resources, organizational structures,
                                     and training.
                                 •   Activities performed: The roles and procedures necessary to implement a
                                     KPA. Activities performed typically involve establishing plans and
                                     procedures, performing the work, tracking it, and taking appropriate
                                     management actions.
                                 •   Measurement and analysis: Activities performed to measure the process
                                     and analyze the measurements. Measurement and analysis typically
                                     includes defining the measurements to be taken and the analyses to be




                                     Page 18                                GAO/AIMD-99-35 Customs Service Modernization
    Chapter 1
    Introduction




    conducted to determine the status and effectiveness of the activities
    performed.
•   Verifying implementation: The steps to ensure that the activities are
    performed in compliance with the process that has been established.
    Verification typically encompasses reviews by management.

    In accordance with SEI’s SCE method and, for five of the six KPAs in level 2,
    we evaluated Customs’ institutional policies and practices and compared
    project-specific guidance and practices against the five common
    attributes. This project-specific comparison can result in one of four
    possible outcomes: (1) project strength—an effective implementation of
    the key practice, (2) project weakness—ineffective implementation of a
    key practice or failure to implement a key practice, (3) project
    observation—key practice evaluated but evidence inconclusive and cannot
    be characterized as either strength or weakness, and (4) not rated—key
    practice not currently relevant to project, therefore, not evaluated.

    We performed the project-specific evaluations on three ongoing Customs
    software development projects, each of which is described below. As
    requested by the Subcommittee Chairmen, one of the projects evaluated
    was ACE, which is the largest and most important system that Customs is
    developing. The other two projects were selected by Customs on the basis
    of the following GAO specified criteria: (1) each project should be managed
    by a different software team, (2) at least one project should involve a
    legacy system, (3) at least one project should involve Year 2000 software
    conversion, and (4) each project should be relatively large and important
    to accomplishing Customs’ mission. The projects we evaluated are:

•   National Customs Automation Program (NCAP 0.1): NCAP 0.1 was the first
    component of the National Customs Automation Program Prototype
    (NCAP/P). NCAP/P, in turn, is the first phase of the Automated Commercial
    Environment (ACE). Customs began developing ACE in 1994 to address the
    new import processing requirements established by the National Customs
    Automation Program. ACE is also intended to replace the agency’s legacy
    automated import system, the Automated Commercial System (ACS). NCAP
    0.1 was installed at three field locations in May 1998.
•   Automated Export System (AES): AES is an export information gathering
    and processing system, developed through cooperative efforts by
    Customs, the Bureau of Census, other federal agencies with export
    missions, and the export trade community. AES is designed to improve the
    collection of trade statistics; assist in the creation of a paperless export
    environment; facilitate the release of exports subject to licensing



    Page 19                             GAO/AIMD-99-35 Customs Service Modernization
    Chapter 1
    Introduction




    requirements; and consolidate export data required by several government
    agencies, easing the data filing burden for exporters while streamlining the
    federal data collection process.

    Customs installed AES in all U.S. vessel ports in October 1996, and
    currently it is operational in all ports, including air, rail, and truck transit
    ports. Customs and Census officials estimate that they spent
    approximately $12.9 million to develop and implement AES from fiscal year
    1992 to 1997. These costs included, among other things, expenses for
    contractors, travel, and training. According to Customs’ and Census’
    figures, both agencies estimate that together they will spend an additional
    $32.2 million through fiscal year 2002 on AES implementation and
    maintenance.
•   Administrative Security System: The Administrative Security System
    assists users in requesting access to administrative systems. Users’
    requests are electronically submitted to the appropriate official for
    approvals. In addition, other portions of the Administrative Security
    System provide functionality to allow the System Administrators the
    ability to prepare and maintain user profiles, request logs, and electronic
    approval and disapproval reports.

    To assess the effectiveness of Customs’ software process improvement
    program, we interviewed the Director, Technical Architecture Group,
    Office of Information and Technology, to determine: (1) process
    improvements that are planned and underway, (2) the rationale for each
    initiative, (3) the relative priority of each, (4) progress made on each
    initiative, and (5) obstacles, if any, impeding progress. We also reviewed
    past process improvement plans, meeting minutes, and related
    documentation. Further, we reviewed SEI’s model for software process
    improvement, known as IDEALSM.6 IDEAL defines five sequential phases of
    software process improvement that can be used to develop a long range,
    integrated plan for initiating and managing a software process
    improvement program.7

    Customs provided written comments on a draft of this report. These
    comments are presented and evaluated in chapter 8, and are reprinted in
    appendix I. We performed our work at Customs’ Newington, Virginia, Data
    Center from February 1998 through November 1998, in accordance with
    generally accepted government auditing standards.


    6
     IDEALSM is a service mark of Carnegie Mellon University.
    7
     IDEAL: A User’s Guide for Software Process Improvement (CMU/SEI-96-HB-001).



    Page 20                                        GAO/AIMD-99-35 Customs Service Modernization
Chapter 2

Requirements Management


                         The purpose of requirements management is to establish agreement
                         between the customer and the software developers of the customer’s
                         requirements that will be implemented by the software developers. This
                         agreement typically is referred to as the “system requirements allocated to
                         the software.” The agreement covers both technical and nontechnical (e.g.,
                         delivery dates) requirements. The agreement forms the basis for
                         estimating, planning, performing, and tracking the software developer’s
                         activities throughout the software life cycle.

                         According to the SW-CMM, a repeatable requirements management process,
                         among other things, includes (1) documenting the system requirements
                         allocated to software, (2) providing adequate resources and funding for
                         managing the allocated requirements, (3) following a written
                         organizational policy for requirements management, (4) having a quality
                         assurance group that reviews the activities and work products for
                         managing allocated requirements and reports the results, (5) using the
                         allocated requirements as the basis for software plans, work products, and
                         activities, and (6) training members of the software engineering group to
                         perform their requirements management activities.


                         All three projects had practice strengths in this KPA. For example, each
Customs’                 project documented the system requirements allocated to software and
Requirements             ensured that adequate resources and funding for managing the allocated
Management Process       requirements were provided. One of the projects, NCAP 0.1, had strengths in
                         all but two practices under this KPA; however, each practice weakness is
Does Not Satisfy SEI’s   significant.
Criteria
                         Collectively, the projects had many weaknesses in this KPA, and thus
                         Customs’ requirements management processes do not meet “repeatable”
                         maturity level criteria. For example, none of the projects had a written
                         organizational policy governing requirements management, and none had
                         a quality assurance group for reviewing and reporting on the activities and
                         work products associated with managing the allocated requirements. In
                         the absence of these two practices, management is missing two means for
                         ensuring that software requirements are managed in a prescribed manner.
                         Also, two of the projects did not use the allocated software requirements
                         as the basis for software plans, work products, and activities, which
                         increases the risk that the software developed will not fully satisfy
                         requirements. Further, members of two projects’ software engineering
                         groups were not trained to perform requirements management activities,
                         thus increasing the chances of mismanagement.



                         Page 21                            GAO/AIMD-99-35 Customs Service Modernization
                                           Chapter 2
                                           Requirements Management




                                           Table 2.1 provides a comprehensive list of the three projects’ strengths and
                                           weaknesses for the requirements management KPA. The specific findings
                                           supporting the practice ratings cited in table 2.1 are in tables 2.2 through
                                           2.4.


Table 2.1: Requirements Management
                    Key practice                                            NCAP 0.1        AES              Administrative
Commitment 1      The project follows a written organizational policy for   Weakness        Weakness         Weakness
                  managing the system requirements allocated to
                  software.
Ability 1         For each project, responsibility is established for       Strength        Strength         Strength
                  analyzing the system requirements and allocating them
                  to hardware, software, and other system components.
Ability 2         The allocated requirements are documented.                Strength        Strength         Strength
Ability 3         Adequate resources and funding are provided for           Strength        Strength         Strength
                  managing the allocated requirements.
Ability 4         Members of the software engineering group and other       Strength        Weakness         Weakness
                  software-related groups are trained to perform their
                  requirements management activities.
Activity 1        The software engineering group reviews the allocated      Strength        Strength         Weakness
                  requirements before they are incorporated into the
                  software project.
Activity 2        The software engineering group uses the allocated         Strength        Weakness         Weakness
                  requirements as the basis for software plans, work
                  products, and activities.
Activity 3        Changes to the allocated requirements are reviewed        Strength        Strength         Weakness
                  and incorporated into the software project.
Measurement 1     Measurements are made and used to determine the           Strength        Strength         Strength
                  status of the activities for managing the allocated
                  requirements.
Verification 1    The activities for managing the allocated requirements    Strength        Weakness         Strength
                  are reviewed with senior management on a periodic
                  basis.
Verification 2    The activities for managing the allocated requirements    Strength        Strength         Strength
                  are reviewed with the project manager on both a
                  periodic and event-driven basis.
Verification 3    The software quality assurance group reviews and/or      Weakness         Weakness         Weakness
                  audits the activities and work products for managing the
                  allocated requirements and reports the results.




                                           Page 22                                 GAO/AIMD-99-35 Customs Service Modernization
                                            Chapter 2
                                            Requirements Management




Table 2.2: Requirements Management Findings for NCAP 0.1
                                                 National Customs Automation Program 0.1
                   Key practice                                      Finding                                          Rating
Commitment 1       The project follows a written organizational      There is no written organizational policy for    Weakness
                   policy for managing the system requirements       managing the system requirements allocated
                   allocated to software.                            to software.
Ability 1          For each project, responsibility is established   The Process Analysis and Requirements Team Strength
                   for analyzing the system requirements and         is responsible for analyzing system
                   allocating them to hardware, software, and        requirements and allocating them to hardware,
                   other system components.                          software, and other system components.
Ability 2          The allocated requirements are documented.        Allocated requirements are documented.           Strength
Ability 3          Adequate resources and funding are provided Adequate resources and funding are provided Strength
                   for managing the allocated requirements.    for managing the allocated requirements.
Ability 4          Members of the software engineering group         Members of the software engineering group        Strength
                   and other software-related groups are trained     and other software-related groups are trained
                   to perform their requirements management          to perform their requirements management
                   activities.                                       activities.
Activity 1         The software engineering group reviews the        The software engineering group reviews the       Strength
                   allocated requirements before they are            allocated requirements before they are
                   incorporated into the software project.           incorporated into the software project.
Activity 2         The software engineering group uses the           The software engineering group uses the          Strength
                   allocated requirements as the basis for           allocated requirements as the basis for
                   software plans, work products, and activities.    software plans, work products, and activities.
Activity 3         Changes to the allocated requirements are         Changes to the allocated requirements are        Strength
                   reviewed and incorporated into the software       reviewed and incorporated into the software
                   project.                                          project.
Measurement 1      Measurements are made and used to                 Measurements are made and used to                Strength
                   determine the status of the activities for        determine the status of the activities for
                   managing the allocated requirements.              managing the allocated requirements.
Verification 1     The activities for managing the allocated         Periodic meetings with senior management         Strength
                   requirements are reviewed with senior             include reviews of allocated requirements.
                   management on a periodic basis.
Verification 2     The activities for managing the allocated         The activities for managing the allocated        Strength
                   requirements are reviewed with the project        requirements are reviewed with the project
                   manager on both a periodic and event-driven       manager on both a periodic and event-driven
                   basis.                                            basis.
Verification 3     The software quality assurance group reviews      There is no software quality assurance group,    Weakness
                   and/or audits the activities and work products    therefore, no reviews and/or audits are done.
                   for managing the allocated requirements and
                   reports the results.




                                            Page 23                                    GAO/AIMD-99-35 Customs Service Modernization
                                            Chapter 2
                                            Requirements Management




Table 2.3: Requirements Management Findings for AES
                                                                Automated Export System
                   Key practice                                      Finding                                           Rating
Commitment 1       The project follows a written organizational      There is no written organizational policy for     Weakness
                   policy for managing the system requirements       managing systems requirements allocated to
                   allocated to software.                            software.
Ability 1          For each project, responsibility is established   The project team established responsibility for   Strength
                   for analyzing the system requirements and         analyzing the system requirements and
                   allocating them to hardware, software, and        allocating them to hardware, software, and
                   other system components.                          other system components.
Ability 2          The allocated requirements are documented.        The allocated requirements are documented in Strength
                                                                     the system functional requirements document
                                                                     and in the change requests document.
Ability 3          Adequate resources and funding are provided Adequate resources and funding are provided Strength
                   for managing the allocated requirements.    for managing allocated requirements.
Ability 4          Members of the software engineering group         One member of the software engineering            Weakness
                   and other software-related groups are trained     group has been trained to perform
                   to perform their requirements management          requirements management activities, but
                   activities.                                       others have not.
Activity 1         The software engineering group reviews the        The software engineering group reviews the        Strength
                   allocated requirements before they are            allocated requirements before they are
                   incorporated into the software project.           incorporated into the software project.
Activity 2         The software engineering group uses the           The software engineering group does not use       Weakness
                   allocated requirements as the basis for           the allocated requirements as the basis for
                   software plans, work products, and activities.    software plans, work products, and activities.
Activity 3         Changes to the allocated requirements are         The Change Request Board reviews and              Strength
                   reviewed and incorporated into the software       approves all changes made to allocated
                   project.                                          requirements and incorporates them into the
                                                                     software project.
Measurement 1      Measurements are made and used to                 Measurements are made and used to track           Strength
                   determine the status of the activities for        changes to requirements and hours spent on
                   managing the allocated requirements.              requirements management. A performance
                                                                     measurement plan is available.
Verification 1     The activities for managing the allocated         The activities for managing the allocated         Weakness
                   requirements are reviewed with senior             requirements are not reviewed with senior
                   management on a periodic basis.                   management on a periodic basis.
Verification 2     The activities for managing the allocated         The activities for managing the allocated         Strength
                   requirements are reviewed with the project        requirements are reviewed with the project
                   manager on both a periodic and event-driven       manager on both a periodic and event-driven
                   basis.                                            basis.
Verification 3     The software quality assurance group reviews      The individual responsible for software quality   Weakness
                   and/or audits the activities and work products    assurance did not review and/or audit the
                   for managing the allocated requirements and       activities and work products for managing the
                   reports the results.                              allocated requirements.




                                            Page 24                                   GAO/AIMD-99-35 Customs Service Modernization
                                            Chapter 2
                                            Requirements Management




Table 2.4: Requirements Management Findings for Administrative Security System
                                                      Administrative Security System
                   Key practice                                      Finding                                         Rating
Commitment 1       The project follows a written organizational      There is no written organizational policy on    Weakness
                   policy for managing the system requirements       requirements management.
                   allocated to software.
Ability 1          For each project, responsibility is established   Project leaders are responsible for analyzing   Strength
                   for analyzing the system requirements and         systems requirements and allocating them to
                   allocating them to hardware, software, and        software, hardware or other components.
                   other system components.
Ability 2          The allocated requirements are documented.        Allocated requirements for the system are       Strength
                                                                     documented.
Ability 3          Adequate resources and funding are provided Adequate resources and funding are available Strength
                   for managing the allocated requirements.    for managing the allocated requirements.
Ability 4          Members of the software engineering group         There was no evidence to show that members      Weakness
                   and other software-related groups are trained     of the software engineering group received
                   to perform their requirements management          training in requirements management.
                   activities.
Activity 1         The software engineering group reviews the        The software engineering group did not review Weakness
                   allocated requirements before they are            the allocated requirements before they were
                   incorporated into the software project.           incorporated into the software project.
Activity 2         The software engineering group uses the           Members of the software engineering group       Weakness
                   allocated requirements as the basis for           did not use the allocated requirements as the
                   software plans, work products, and activities.    basis for software plans, work products, and
                                                                     activities.
Activity 3         Changes to the allocated requirements are         Changes to the allocated requirements are not Weakness
                   reviewed and incorporated into the software       reviewed by the software engineering group
                   project.                                          and incorporated into the software project.
Measurement 1      Measurements are made and used to                 Measurements are made and used to               Strength
                   determine the status of the activities for        determine the status of the activities for
                   managing the allocated requirements.              managing the allocated requirements.
Verification 1     The activities for managing the allocated         Requirements management activities are          Strength
                   requirements are reviewed with senior             discussed at periodic meetings with senior
                   management on a periodic basis.                   management.
Verification 2     The activities for managing the allocated         The activities for managing the allocated       Strength
                   requirements are reviewed with the project        requirements are reviewed with the project
                   manager on both a periodic and event-driven       manager on both a periodic and event-driven
                   basis.                                            basis.

Verification 3     The software quality assurance group reviews      There is no software quality assurance (SQA)    Weakness
                   and/or audits the activities and work products    group.
                   for managing the allocated requirements and
                   reports the results.




                                            Page 25                                    GAO/AIMD-99-35 Customs Service Modernization
              Chapter 2
              Requirements Management




              While Customs’ projects had several practice strengths in this KPA, the
Conclusions   number and significance of their practice weaknesses mean that Customs’
              ability to manage software requirements is not repeatable. As a result,
              Customs is at risk of producing systems that fail to provide promised
              capabilities, and cost more and take longer than necessary.




              Page 26                          GAO/AIMD-99-35 Customs Service Modernization
Chapter 3

Software Project Planning


                      The purpose of software project planning is to establish reasonable plans
                      for performing the software engineering and for managing the software
                      project. According to the SW-CMM, a repeatable software project planning
                      process, among other things, includes (1) documenting the software
                      project plan, and preparing plans for software engineering facilities and
                      support tools, (2) identifying the work products needed to establish and
                      maintain control of the software project, (3) following a written
                      organizational policy for planning a software project, (4) having a quality
                      assurance group that reviews the activities and work products for
                      software project planning and reports the results, (5) estimating the
                      software project’s efforts and costs, and estimating its critical computer
                      resources according to a documented procedure, (6) making and using
                      measurements to determine the status of planning activities, and
                      (7) training personnel in software project planning and estimating.


                      All of the projects that we evaluated had key practice strengths in this KPA.
Customs Is Not        For example, all had strengths in (1) documenting a software project plan
Performing Software   and preparing plans for the software engineering facilities and support
Project Planning      tools needed to develop the software and (2) identifying the work
                      products needed to control the software project. NCAP 0.1, in particular,
Effectively           had many additional practice strengths.

                      However, many significant practice weaknesses were found in all three
                      projects. None of the projects followed an organizational software project
                      planning policy, and none had a quality assurance group conducting
                      reviews and/or audits. As a result, the projects performed these practices
                      differently and inconsistently, and controls were unreliable. For example,
                      while the NCAP 0.1 project followed a documented procedure for
                      estimating the size of software work products (or changes to the size of
                      work products), and made and used measurements to determine the status
                      of software planning activities, neither of the other two projects
                      performed these practices and none of the projects had personnel trained
                      in software project planning and estimating. Such project planning
                      weaknesses mean that management has no assurance that it will get the
                      consistent, complete, and reliable information about the projects’
                      expected costs and schedules needed to make expeditious and informed
                      investment decisions.

                      Table 3.1 provides a comprehensive list of the three projects’ strengths,
                      weaknesses, and observations for the software project planning KPA. The




                      Page 27                             GAO/AIMD-99-35 Customs Service Modernization
                                             Chapter 3
                                             Software Project Planning




                                             specific findings supporting the practice ratings cited in table 3.1 are in
                                             tables 3.2 through 3.4.


Table 3.1: Software Project Planning
                     Key practice                                                NCAP 0.1        AES              Administrative
Commitment 1        A project software manager is designated to be               Strength        Strength         Weakness
                    responsible for negotiating commitments and
                    developing the project’s software development plan.
Commitment 2        The project follows a written organizational policy for      Weakness        Weakness         Weakness
                    planning a software project.
Ability 1           A documented and approved statement of work exists           Strength        Observation      Observation
                    for the software project.
Ability 2           Responsibilities for developing the software                 Strength        Observation      Strength
                    development plan are assigned.
Ability 3           Adequate resources and funding are provided for              Strength        Weakness         Strength
                    planning the software project.
Ability 4           The software managers, software engineers, and other         Weakness        Weakness         Weakness
                    individuals involved in the software project planning are
                    trained in the software estimating and planning
                    procedures applicable to their areas of responsibility.
Activity 1          The software engineering group participates on the           Strength        Weakness         Weakness
                    project proposal team.
Activity 2          Software project planning is initiated in the early stages   Strength        Weakness         Strength
                    of, and in parallel with, the overall project planning.
Activity 3          The software engineering group participates with other       Strength        Weakness         Observation
                    affected groups in the overall project planning
                    throughout the project’s life.
Activity 4          Software project commitments made to individuals and         Weakness        Observation      Weakness
                    groups external to the organization are reviewed with
                    senior management according to a documented
                    procedure.
Activity 5          A software life cycle with predefined stages of              Weakness        Strength         Strength
                    manageable size is identified or defined.
Activity 6          The project’s software development plan is developed         Strength        Strength         Weakness
                    according to a documented procedure.
Activity 7          The plan for the software project is documented.             Strength        Strength         Strength
Activity 8          Software work products that are needed to establish          Strength        Strength         Strength
                    and maintain control of the software project are
                    identified.
Activity 9          Estimates for the size of the software work products (or     Strength        Weakness         Weakness
                    changes to the size of software work products) are
                    derived according to a documented procedure.
Activity 10         Estimates for the software project’s effort and costs are    Weakness        Weakness         Weakness
                    derived according to a documented procedure.
Activity 11         Estimates for the project’s critical computer resources      Weakness        Weakness         Weakness
                    are derived according to a documented procedure.
                                                                                                                         (continued)


                                             Page 28                                    GAO/AIMD-99-35 Customs Service Modernization
                                         Chapter 3
                                         Software Project Planning




                 Key practice                                                NCAP 0.1        AES              Administrative
Activity 12      The project’s software schedule is derived according to     Weakness        Weakness         Weakness
                 a documented procedure.
Activity 13      The software risks associated with the cost, resource,      Strength        Strength         Weakness
                 schedule, and technical aspects of the project are
                 identified, assessed, and documented.
Activity 14      Plans for the project’s software engineering facilities and Strength        Strength         Strength
                 support tools are prepared.
Activity 15      Software planning data are recorded.                        Strength        Weakness         Weakness
Measurement 1    Measurements are made and used to determine the             Strength        Weakness         Weakness
                 status of the software planning activities.
Verification 1   The activities for software project planning are reviewed   Strength        Weakness         Weakness
                 with senior management on a periodic basis.
Verification 2   The activities for software project planning are reviewed   Strength        Strength         Weakness
                 with the project manager on both a periodic and
                 event-driven basis.
Verification 3   The software quality assurance group reviews and/or         Weakness        Weakness         Weakness
                 audits the activities and work products for software
                 project planning and reports the results.




                                         Page 29                                    GAO/AIMD-99-35 Customs Service Modernization
                                            Chapter 3
                                            Software Project Planning




Table 3.2: Software Project Planning Findings for NCAP 0.1
                                                    National Customs Automation Program 0.1
                   Key practice                                      Finding                                            Rating
Commitment 1       A project software manager is designated to       The NCAP project has a software manager            Strength
                   be responsible for negotiating commitments        designated to be responsible for negotiating
                   and developing the project’s software             commitments and developing the project’s
                   development plan.                                 software development plan.
Commitment 2       The project follows a written organizational      The project does not follow a written              Weakness
                   policy for planning a software project.           organizational policy for planning a software
                                                                     project.
Ability 1          A documented and approved statement of            The approved project plan meets the                Strength
                   work exists for the software project.             requirements for a statement of work for the
                                                                     project.
Ability 2          Responsibilities for developing the software      The project manager has been assigned              Strength
                   development plan are assigned.                    responsibility for developing the software
                                                                     development plan.
Ability 3          Adequate resources and funding are provided Adequate resources and funding have been                 Strength
                   for planning the software project.          provided for planning the software project.
Ability 4          The software managers, software engineers,        Project personnel are not trained in software      Weakness
                   and other individuals involved in the software    project planning and estimating procedures.
                   project planning are trained in the software
                   estimating and planning procedures
                   applicable to their areas of responsibility.
Activity 1         The software engineering group participates       The software engineering group participates        Strength
                   on the project proposal team.                     on the project proposal team.
Activity 2         Software project planning is initiated in the      Software project planning is initiated in the      Strength
                   early stages of, and in parallel with, the overall early stages of, and in parallel with, the overall
                   project planning.                                  project planning.
Activity 3         The software engineering group participates       The software engineering group participates        Strength
                   with other affected groups in the overall         with other affected groups in the overall
                   project planning throughout the project’s life.   project planning throughout the project’s life.
Activity 4         Software project commitments made to              Software project commitments made to               Weakness
                   individuals and groups external to the            individuals and groups external to the
                   organization are reviewed with senior             organization are not reviewed with senior
                   management according to a documented              management and there is no documented
                   procedure.                                        procedure for such reviews.
Activity 5         A software life cycle with predefined stages of   There is no documented evidence that a            Weakness
                   manageable size is identified or defined.         software life cycle was selected for the project.
Activity 6         The project’s software development plan is        The project has a software development plan        Strength
                   developed according to a documented               that is developed according to a documented
                   procedure.                                        procedure.
Activity 7         The plan for the software project is              The plan for the software project is               Strength
                   documented.                                       documented in the project plan.
Activity 8         Software work products that are needed to         Software work products that are needed to          Strength
                   establish and maintain control of the software    establish and maintain control of the software
                   project are identified.                           project are identified in the project plan.
                                                                                                                             (continued)




                                            Page 30                                    GAO/AIMD-99-35 Customs Service Modernization
                                         Chapter 3
                                         Software Project Planning




                                                    National Customs Automation Program 0.1
                 Key practice                                       Finding                                            Rating
Activity 9       Estimates for the size of the software work        Estimates for the size of the software work        Strength
                 products (or changes to the size of software       products (or changes to the size of software
                 work products) are derived according to a          work products) are derived according to a
                 documented procedure.                              documented procedure.
Activity 10      Estimates for the software project’s effort and    Estimates for the software project’s effort and    Weakness
                 costs are derived according to a documented        costs are not derived according to a
                 procedure.                                         documented procedure.
Activity 11      Estimates for the project’s critical computer      Estimates for the project’s critical computer      Weakness
                 resources are derived according to a               resources are not derived according to a
                 documented procedure.                              documented procedure.
Activity 12      The project’s software schedule is derived         There is no documented procedure for               Weakness
                 according to a documented procedure.               deriving the software schedule.
Activity 13      The software risks associated with the cost,       The software risks associated with the cost,       Strength
                 resource, schedule, and technical aspects of       resource, schedule, and technical aspects of
                 the project are identified, assessed, and          the project are identified, assessed, and
                 documented.                                        documented.
Activity 14      Plans for the project’s software engineering       Plans for the project’s software engineering       Strength
                 facilities and support tools are prepared.         facilities and support tools are prepared.
Activity 15      Software planning data are recorded.               Software planning data are recorded.               Strength
Measurement 1    Measurements are made and used to                  Measurements are made and used to                  Strength
                 determine the status of the software planning      determine the status of the software planning
                 activities.                                        activities.
Verification 1   The activities for software project planning are   The activities for software project planning are   Strength
                 reviewed with senior management on a               reviewed with senior management, including
                 periodic basis.                                    reports to Treasury and Customs Investment
                                                                    Review Boards (IRB), on a periodic basis.
Verification 2   The activities for software project planning are   The activities for software project planning are   Strength
                 reviewed with the project manager on both a        reviewed with the project manager on both a
                 periodic and event-driven basis.                   periodic and event-driven basis through
                                                                    weekly status reports and meetings.
Verification 3   The software quality assurance group reviews       There is no SQA group.                             Weakness
                 and/or audits the activities and work products
                 for software project planning and reports the
                 results.




                                         Page 31                                      GAO/AIMD-99-35 Customs Service Modernization
                                             Chapter 3
                                             Software Project Planning




Table 3.3: Software Project Planning Findings for AES
                                                                Automated Export System
                    Key practice                                      Finding                                           Rating
Commitment 1        A project software manager is designated to       A project software manager is designated to       Strength
                    be responsible for negotiating commitments        be responsible for negotiating commitments
                    and developing the project’s software             and developing the project’s software
                    development plan.                                 development plan.
Commitment 2        The project follows a written organizational      The project does not follow a written             Weakness
                    policy for planning a software project.           organizational policy for planning a software
                                                                      project.
Ability 1           A documented and approved statement of            The AES System and Functional Requirements Observation
                    work exists for the software project.             Table documents the statement of work for this
                                                                      project. However, this document is not
                                                                      approved.
Ability 2           Responsibilities for developing the software      Responsibilities for developing the software      Observation
                    development plan are assigned.                    development plan have not been assigned;
                                                                      however, a plan has been developed.
Ability 3           Adequate resources and funding are provided Adequate resources and funding are not                  Weakness
                    for planning the software project.          provided for planning the software project.
Ability 4           The software managers, software engineers,        Individuals involved in the software project      Weakness
                    and other individuals involved in the software    planning are not trained in the software
                    project planning are trained in the software      estimating and planning procedures.
                    estimating and planning procedures
                    applicable to their areas of responsibility.
Activity 1          The software engineering group participates       The software engineering group does not           Weakness
                    on the project proposal team.                     participate in project proposal preparation.
Activity 2          Software project planning is initiated in the      Software project planning is not initiated in the Weakness
                    early stages of, and in parallel with, the overall early stages of, and in parallel with, the overall
                    project planning.                                  project planning.
Activity 3          The software engineering group participates       The software engineering group does not           Weakness
                    with other affected groups in the overall         participate in the overall project planning
                    project planning throughout the project’s life.   throughout the project’s life cycle.
Activity 4          Software project commitments made to              Software project commitments made to              Observation
                    individuals and groups external to the            individuals and groups external to the
                    organization are reviewed with senior             organization are reviewed with senior
                    management according to a documented              management; however, there is no
                    procedure.                                        documented procedure for these reviews.
Activity 5          A software life cycle with predefined stages of   The project uses the waterfall model identified   Strength
                    manageable size is identified or defined.         in the Customs software development life
                                                                      cycle document.
Activity 6          The project’s software development plan is        The project’s software development plan was       Strength
                    developed according to a documented               developed according to a documented
                    procedure.                                        procedure in the SDLC.
Activity 7          The plan for the software project is              The plan for the software project is              Strength
                    documented.                                       documented.
Activity 8          Software work products that are needed to         Software work products that are needed to         Strength
                    establish and maintain control of the software    establish and maintain control of the software
                    project are identified.                           project have been identified.
                                                                                                                            (continued)


                                             Page 32                                    GAO/AIMD-99-35 Customs Service Modernization
                                         Chapter 3
                                         Software Project Planning




                                                              Automated Export System
                 Key practice                                       Finding                                            Rating
Activity 9       Estimates for the size of the software work        There is no documented procedure for               Weakness
                 products (or changes to the size of software       estimating the size of software work products
                 work products) are derived according to a          (or changes to the size of software work
                 documented procedure.                              products).
Activity 10      Estimates for the software project’s effort and    There is no documented procedure for               Weakness
                 costs are derived according to a documented        estimating project effort and cost.
                 procedure.
Activity 11      Estimates for the project’s critical computer      There is no documented procedure for               Weakness
                 resources are derived according to a               estimating the project’s computer resources.
                 documented procedure.
Activity 12      The project’s software schedule is derived         There is no documented procedure for               Weakness
                 according to a documented procedure.               deriving the software schedule.
Activity 13      The software risks associated with the cost,       The software risks associated with the cost,       Strength
                 resource, schedule, and technical aspects of       resource, schedule, and technical aspects of
                 the project are identified, assessed, and          the project are identified, assessed, and
                 documented.                                        documented.
Activity 14      Plans for the project’s software engineering       AES is an ongoing project. The software          Strength
                 facilities and support tools are prepared.         engineering facilities and support tools already
                                                                    exist, and are documented in the software
                                                                    development plan.
Activity 15      Software planning data are recorded.               Software planning data, such as estimating the Weakness
                                                                    project’s critical computer resources, are not
                                                                    recorded.
Measurement 1    Measurements are made and used to                  Measurements are not made and used to              Weakness
                 determine the status of the software planning      determine the status of the software planning
                 activities.                                        activities.
Verification 1   The activities for software project planning are   The activities for software project planning are   Weakness
                 reviewed with senior management on a               not reviewed with senior management on a
                 periodic basis.                                    periodic basis.
Verification 2   The activities for software project planning are   The activities for software project planning are   Strength
                 reviewed with the project manager on both a        reviewed with the project manager on both a
                 periodic and event-driven basis.                   periodic and event-driven basis.
Verification 3   The software quality assurance group reviews       The individual performing software quality         Weakness
                 and/or audits the activities and work products     assurance activities does not conduct reviews
                 for software project planning and reports the      and/or audits.
                 results.




                                         Page 33                                      GAO/AIMD-99-35 Customs Service Modernization
                                             Chapter 3
                                             Software Project Planning




Table 3.4: Software Project Planning Findings for Administrative Security System
                                                          Administrative Security System
                    Key practice                                      Finding                                            Rating
Commitment 1        A project software manager is designated to       A project software manager has not been            Weakness
                    be responsible for negotiating commitments        designated to be responsible for negotiating
                    and developing the project’s software             commitments and developing the project’s
                    development plan.                                 software development plan. The software
                                                                      engineering group was not sure who was
                                                                      responsible for negotiating commitments.
Commitment 2        The project follows a written organizational      The project does not follow a written              Weakness
                    policy for planning a software project.           organizational policy for planning a software
                                                                      project.
Ability 1           A documented and approved statement of            A documented statement of work exists for the Observation
                    work exists for the software project.             software project, but has not been approved.
Ability 2           Responsibilities for developing the software      Responsibilities for developing the software Strength
                    development plan are assigned.                    development plan were assigned to a team of
                                                                      seven whose names appear on the document.
Ability 3           Adequate resources and funding are provided Adequate resources and funding are available Strength
                    for planning the software project.          for planning the software project.

Ability 4           The software managers, software engineers,        The software managers, software engineers,       Weakness
                    and other individuals involved in the software    and other individuals involved in the software
                    project planning are trained in the software      project planning are not trained in the software
                    estimating and planning procedures                estimating and planning procedures
                    applicable to their areas of responsibility.      applicable to their areas of responsibility.
Activity 1          The software engineering group participates       The software engineering group did not             Weakness
                    on the project proposal team.                     participate on the project proposal team.
Activity 2          Software project planning is initiated in the      Software project planning is initiated in the      Strength
                    early stages of, and in parallel with, the overall early stages of, and in parallel with, the overall
                    project planning.                                  project planning.
Activity 3          The software engineering group participates       The software engineering group occasionally        Observation
                    with other affected groups in the overall         participates with other affected groups in the
                    project planning throughout the project’s life.   overall project planning throughout the
                                                                      project’s life.
Activity 4          Software project commitments made to              There is no documented procedure to ensure         Weakness
                    individuals and groups external to the            that software project commitments made to
                    organization are reviewed with senior             individuals and groups external to the
                    management according to a documented              organization are reviewed with senior
                    procedure.                                        management.
Activity 5          A software life cycle with predefined stages of   A software life cycle with predefined stages of    Strength
                    manageable size is identified or defined.         manageable size has been identified in the
                                                                      SDLC.
Activity 6          The project’s software development plan is        The project has a software development plan;       Weakness
                    developed according to a documented               but it was not developed according to a
                    procedure.                                        documented procedure.
Activity 7          The plan for the software project is              The plan for the software project is               Strength
                    documented.                                       documented.
                                                                                                                              (continued)



                                             Page 34                                    GAO/AIMD-99-35 Customs Service Modernization
                                         Chapter 3
                                         Software Project Planning




                                                          Administrative Security System
                 Key practice                                       Finding                                          Rating
Activity 8       Software work products that are needed to          Software work products that are needed to        Strength
                 establish and maintain control of the software     establish and maintain control of the software
                 project are identified.                            project are identified.
Activity 9       Estimates for the size of the software work        No documented procedure is used for              Weakness
                 products (or changes to the size of software       estimating the size of software work products
                 work products) are derived according to a          (or changes to the size of software work
                 documented procedure.                              products).
Activity 10      Estimates for the software project’s effort and    No documented procedure is used for              Weakness
                 costs are derived according to a documented        estimating the project’s effort and cost.
                 procedure.
Activity 11      Estimates for the project’s critical computer      No documented procedure is used for              Weakness
                 resources are derived according to a               estimating the project’s computer resources.
                 documented procedure.
Activity 12      The project’s software schedule is derived         No documented procedure is used for              Weakness
                 according to a documented procedure.               deriving the project schedule.
Activity 13      The software risks associated with the cost,       The software risks associated with the cost,     Weakness
                 resource, schedule, and technical aspects of       resource, schedule, and technical aspects of
                 the project are identified, assessed, and          the project have been identified and
                 documented.                                        documented, but have not been assessed for
                                                                    impact and mitigation options.
Activity 14      Plans for the project’s software engineering       Existing tools and the current software          Strength
                 facilities and support tools are prepared.         engineering environment have been identified
                                                                    for use.
Activity 15      Software planning data are recorded.               Software planning data are not recorded.         Weakness
Measurement 1    Measurements are made and used to                  No measurements are made to determine the        Weakness
                 determine the status of the software planning      status of software planning activities.
                 activities.
Verification 1   The activities for software project planning are   While periodic meetings are held with senior     Weakness
                 reviewed with senior management on a               management, there was no evidence that
                 periodic basis.                                    activities for software project planning are
                                                                    addressed.
Verification 2   The activities for software project planning are   While periodic meetings are held with the        Weakness
                 reviewed with the project manager on both a        project manager, there was no evidence that
                 periodic and event-driven basis.                   activities for software project planning are
                                                                    addressed.
Verification 3   The software quality assurance group reviews       There is no software quality assurance group.    Weakness
                 and/or audits the activities and work products
                 for software project planning and reports the
                 results.




                                         Page 35                                      GAO/AIMD-99-35 Customs Service Modernization
              Chapter 3
              Software Project Planning




              Effective planning is the cornerstone of successful software development
Conclusions   project management. While Customs showed some strengths in this KPA,
              its many weaknesses render its software project planning processes
              unrepeatable. Therefore, Customs has no assurance that the projects are
              effectively establishing plans, including reliable projections of costs and
              schedules, and effectively measuring and monitoring progress and taking
              needed corrective actions expeditiously.




              Page 36                            GAO/AIMD-99-35 Customs Service Modernization
Chapter 4

Software Project Tracking and Oversight


                       The purpose of software project tracking and oversight is to provide
                       adequate visibility into the progress of the software development so that
                       management can act effectively when the software project’s performance
                       deviates significantly from the software plans. Software project tracking
                       and oversight involves tracking and reviewing the software
                       accomplishments and results against documented estimates,
                       commitments, and plans, and adjusting these plans based on the actual
                       accomplishments and results.

                       According to the SW-CMM, effective software project tracking and oversight,
                       among other things, includes (1) designating a project software manager
                       to be responsible for the project’s software activities and results,
                       (2) having a documented software development plan for tracking software
                       activities and communicating status, (3) following a written organizational
                       policy for managing the project, (4) conducting periodic internal reviews
                       to track technical progress, plans, performance, and issues against the
                       software development plan, (5) tracking the software risks associated with
                       the cost, resource, schedule, and technical aspects of the project,
                       (6) explicitly assigning responsibility for software work products and
                       activities, (7) tracking the sizes of the software work products (or sizes of
                       the changes to the software work products) and taking corrective actions
                       as necessary, and (8) periodically reviewing the activities for software
                       project tracking and oversight with senior management.


                       The projects evaluated exhibited some software project tracking and
Customs’ Software      oversight practice strengths. For example, all three of the projects had a
Project Tracking and   project software manager designated to be responsible for the project’s
Oversight Process Is   software activities and results, and all had a documented software
                       development plan for tracking software activities and communicating
Immature               status. Also, NCAP 0.1 had strengths in all but five of this KPA’s 24 key
                       practices.

                       However, the three projects collectively had many weaknesses, and these
                       weaknesses, including the five for NCAP 0.1, were significant and thus
                       preclude Customs from meeting SEI’s repeatable maturity level criteria. For
                       example, none of the projects followed a written organizational policy for
                       managing the software project. With no established policy, Customs
                       increases the risk that key tracking and oversight activities will not be
                       performed effectively. For example, for two of the three projects, the
                       project managers did not (1) conduct periodic internal reviews to track
                       technical progress, plans, performance, and issues against the software



                       Page 37                             GAO/AIMD-99-35 Customs Service Modernization
                                            Chapter 4
                                            Software Project Tracking and Oversight




                                            development plan, (2) track software risks associated with cost, resource,
                                            schedule, and technical aspects of the project, (3) explicitly assign
                                            responsibility to individuals for software work products and activities,
                                            (4) track the sizes of the software work products (or sizes of the changes
                                            to the software work products) and take corrective actions, or
                                            (5) periodically review software project tracking and oversight activities
                                            with senior management.

                                            Table 4.1 provides a comprehensive list of the three projects’ strengths,
                                            weaknesses, and observations for the software project tracking and
                                            oversight KPA. The specific findings supporting the practice ratings cited in
                                            table 4.1 are in tables 4.2 through 4.4.


Table 4.1: Software Project Tracking and Oversight Summary
                     Key practice                                              NCAP 0.1        AES              Administrative
Commitment 1       A project software manager is designated to be              Strength        Strength         Strength
                   responsible for the project’s software activities and
                   results.
Commitment 2       The project follows a written organizational policy for     Weakness        Weakness         Weakness
                   managing the software project.
Ability 1          A software development plan for the software project is     Strength        Observation      Observation
                   documented and approved.
Ability 2          The project software manager explicitly assigns             Strength        Weakness         Weakness
                   responsibility for software work products and activities.
Ability 3          Adequate resources and funding are provided for             Strength        Weakness         Weakness
                   tracking the software project.
Ability 4          The software managers are trained in managing the        Weakness           Strength         Weakness
                   technical and personnel aspects of the software project.
Ability 5          First-line software managers receive orientation in the     Strength        Strength         Weakness
                   technical aspects of the software project.
Activity 1         A documented software development plan is used for          Strength        Strength         Strength
                   tracking the software activities and communicating
                   status.
Activity 2         The project’s software development plan is revised          Weakness        Weakness         Weakness
                   according to a documented procedure.
Activity 3         Software project commitments and changes to                 Weakness        Observation      Observation
                   commitments made to individuals and groups external
                   to the organization are reviewed with senior
                   management according to a documented procedure.
Activity 4         Approved changes to commitments that affect the             Strength        Strength         Weakness
                   software project are communicated to the members of
                   the software engineering group and other
                   software-related groups.
                                                                                                                       (continued)




                                            Page 38                                   GAO/AIMD-99-35 Customs Service Modernization
                                          Chapter 4
                                          Software Project Tracking and Oversight




                 Key practice                                                NCAP 0.1        AES              Administrative
Activity 5       The sizes of the software work products (or sizes of the    Strength        Weakness         Weakness
                 changes to the software work products) are tracked,
                 and corrective actions are taken as necessary.
Activity 6       The project’s software effort and costs are tracked, and    Strength        Strength         Weakness
                 corrective actions are taken as necessary.
Activity 7       The project’s critical computer resources are tracked,      Strength        Weakness         Weakness
                 and corrective actions are taken as necessary.
Activity 8       The project’s software schedule is tracked, and             Strength        Weakness         Strength
                 corrective actions are taken as necessary.
Activity 9       Software engineering technical activities are tracked,      Strength        Weakness         Weakness
                 and corrective actions are taken as necessary.
Activity 10      The software risks associated with cost, resource,          Strength        Weakness         Weakness
                 schedule, and technical aspects of the project are
                 tracked.
Activity 11      Actual measurement data and replanning data for the         Strength        Weakness         Weakness
                 software project are recorded.
Activity 12      The software engineering group conducts periodic            Strength        Weakness         Weakness
                 internal reviews to track technical progress, plans,
                 performance, and issues against the software
                 development plan.
Activity 13      Formal reviews to address the accomplishments and         Strength          Weakness         Weakness
                 results of the software project are conducted at selected
                 project milestones according to a documented
                 procedure.
Measurement 1    Measurements are made and used to determine the             Strength        Weakness         Weakness
                 status of the software tracking and oversight activities.
Verification 1   The activities for software project tracking and oversight Strength         Weakness         Weakness
                 are reviewed with senior management on a periodic
                 basis.
Verification 2   The activities for software project tracking and oversight Strength         Weakness         Weakness
                 are reviewed with the project manager on both a
                 periodic and event-driven basis.
Verification 3   The software quality assurance group reviews and/or         Weakness        Weakness         Weakness
                 audits the activities and work products for software
                 project tracking and oversight and reports the results.




                                          Page 39                                   GAO/AIMD-99-35 Customs Service Modernization
                                            Chapter 4
                                            Software Project Tracking and Oversight




Table 4.2: Software Project Tracking and Oversight Findings for NCAP 0.1
                                                    National Customs Automation Program 0.1
                   Key practice                                     Finding                                          Rating
Commitment 1       A project software manager is designated to      The project software manager is designated to Strength
                   be responsible for the project’s software        be responsible for the project’s software
                   activities and results.                          activities and results.
Commitment 2       The project follows a written organizational     There is no written organizational policy for    Weakness
                   policy for managing the software project.        managing the software project.
Ability 1          A software development plan for the software     An approved and documented software              Strength
                   project is documented and approved.              development plan is contained in the project
                                                                    plan.
Ability 2          The project software manager explicitly          The project software manager explicitly          Strength
                   assigns responsibility for software work         assigns responsibility for software work
                   products and activities.                         products and activities.
Ability 3          Adequate resources and funding are provided Adequate resources and funding are provided Strength
                   for tracking the software project.          for tracking the software project.
Ability 4          The software managers are trained in             The software managers are not trained in         Weakness
                   managing the technical and personnel             managing the technical and personnel
                   aspects of the software project.                 aspects of the software project.
Ability 5          First-line software managers receive             First-line software managers receive             Strength
                   orientation in the technical aspects of the      orientation in the technical aspects of the
                   software project.                                software project
Activity 1         A documented software development plan is        A documented software development plan is        Strength
                   used for tracking the software activities and    used for tracking the software activities and
                   communicating status.                            communicating status.
Activity 2         The project’s software development plan is   No documented procedure exists for revising          Weakness
                   revised according to a documented procedure. the software development plan.
Activity 3         Software project commitments and changes to      Software project commitments and changes to Weakness
                   commitments made to individuals and groups       commitments made to individuals and groups
                   external to the organization are reviewed with   external to the organization are not reviewed
                   senior management according to a                 with senior management. Also, there is no
                   documented procedure.                            documented procedure for such reviews.
Activity 4         Approved changes to commitments that affect      Approved changes to commitments that affect Strength
                   the software project are communicated to the     the software project are communicated to the
                   members of the software engineering group        members of the software engineering group
                   and other software-related groups.               and other software-related groups through
                                                                    weekly staff meetings.
Activity 5         The sizes of the software work products (or      The sizes of the software work products (or      Strength
                   sizes of the changes to the software work        sizes of the changes to the software work
                   products) are tracked, and corrective actions    products) are tracked; however, at the time of
                   are taken as necessary.                          our evaluation, no corrective actions were
                                                                    needed.
Activity 6         The project’s software effort and costs are      The project’s software effort and costs are      Strength
                   tracked, and corrective actions are taken as     tracked; however, at the time of our evaluation,
                   necessary.                                       no corrective actions were needed.
Activity 7         The project’s critical computer resources are    The project’s critical computer resources are    Strength
                   tracked, and corrective actions are taken as     tracked; however, at the time of our evaluation,
                   necessary.                                       no corrective actions were needed.
                                                                                                                         (continued)


                                            Page 40                                   GAO/AIMD-99-35 Customs Service Modernization
                                         Chapter 4
                                         Software Project Tracking and Oversight




                                                    National Customs Automation Program 0.1
                 Key practice                                       Finding                                            Rating
Activity 8       The project’s software schedule is tracked,    The project’s software schedule is tracked;            Strength
                 and corrective actions are taken as necessary. however, at the time of our evaluation, no
                                                                corrective actions were needed.
Activity 9       Software engineering technical activities are      Software engineering technical activities are      Strength
                 tracked, and corrective actions are taken as       tracked, and corrective actions are taken as
                 necessary.                                         necessary.
Activity 10      The software risks associated with cost,           The software risks associated with cost,           Strength
                 resource, schedule, and technical aspects of       resource, schedule, and technical aspects of
                 the project are tracked.                           the project are tracked.
Activity 11      Actual measurement data and replanning data Actual measurement data and replanning data Strength
                 for the software project are recorded.      for the software project are recorded.
Activity 12      The software engineering group conducts            The software engineering group conducts            Strength
                 periodic internal reviews to track technical       periodic internal reviews to track technical
                 progress, plans, performance, and issues           progress, plans, performance, and issues
                 against the software development plan.             against the software development plan.
Activity 13      Formal reviews to address the                      Formal reviews to address the                      Strength
                 accomplishments and results of the software        accomplishments and results of the software
                 project are conducted at selected project          project are conducted at selected project
                 milestones according to a documented               milestones according to a documented
                 procedure.                                         procedure.
Measurement 1    Measurements are made and used to                  Measurements are made and used to                  Strength
                 determine the status of the software tracking      determine the status of the software tracking
                 and oversight activities.                          and oversight activities.
Verification 1   The activities for software project tracking and   The activities for software project tracking and   Strength
                 oversight are reviewed with senior                 oversight are reviewed with senior
                 management on a periodic basis.                    management on a weekly basis.
Verification 2   The activities for software project tracking and   The activities for software project tracking and   Strength
                 oversight are reviewed with the project            oversight are reviewed with the project
                 manager on both a periodic and event-driven        manager on both a periodic and event-driven
                 basis.                                             basis.
Verification 3   The software quality assurance group reviews       No software quality assurance group exists;      Weakness
                 and/or audits the activities and work products     therefore, there are no reviews and/or audits of
                 for software project tracking and oversight and    the activities and work products for software
                 reports the results.                               project tracking and oversight.




                                         Page 41                                      GAO/AIMD-99-35 Customs Service Modernization
                                            Chapter 4
                                            Software Project Tracking and Oversight




Table 4.3: Software Project Tracking and Oversight Findings for AES
                                                            Automated Export System
                   Key practice                                     Finding                                            Rating
Commitment 1       A project software manager is designated to      A project software manager is designated to        Strength
                   be responsible for the project’s software        be responsible for the project’s software
                   activities and results.                          activities and results.
Commitment 2       The project follows a written organizational     There is no written organizational policy for      Weakness
                   policy for managing the software project.        managing the software project.
Ability 1          A software development plan for the software     A software development plan for the project        Observation
                   project is documented and approved.              exists. However, this plan has not been
                                                                    approved.
Ability 2          The project software manager explicitly          The project software manager does not              Weakness
                   assigns responsibility for software work         explicitly assign responsibility for software
                   products and activities.                         work products and activities.
Ability 3          Adequate resources and funding are provided Adequate resources and funding are not                  Weakness
                   for tracking the software project.          provided for tracking the software project.
Ability 4          The software managers are trained in             The software managers are trained in               Strength
                   managing the technical and personnel             managing the technical and personnel
                   aspects of the software project.                 aspects of the software projects through
                                                                    training given at off-site retreats and guidance
                                                                    provided in the AES users’ guide.
Ability 5          First-line software managers receive             First-line software managers receive               Strength
                   orientation in the technical aspects of the      orientation in the technical aspects of the
                   software project.                                software project.
Activity 1         A documented software development plan is        A documented software development plan is          Strength
                   used for tracking the software activities and    used for tracking the software activities and
                   communicating status.                            communicating status.
Activity 2         The project’s software development plan is   No documented procedure exists for revising            Weakness
                   revised according to a documented procedure. the software development plan.
Activity 3         Software project commitments and changes to      Software project commitments and changes to Observation
                   commitments made to individuals and groups       commitments made to individuals and groups
                   external to the organization are reviewed with   external to the organization are reviewed in
                   senior management according to a                 periodic meetings. However, there is no
                   documented procedure.                            documented procedure for these reviews.
Activity 4         Approved changes to commitments that affect      Approved changes to commitments that affect Strength
                   the software project are communicated to the     the software project are communicated to the
                   members of the software engineering group        members of the software engineering group
                   and other software-related groups.               and other software-related groups through
                                                                    e-mail and weekly status meetings.
Activity 5         The sizes of the software work products (or      The sizes of the software work products (or        Weakness
                   sizes of the changes to the software work        sizes of the changes to the software work
                   products) are tracked, and corrective actions    products) are not tracked.
                   are taken as necessary.
Activity 6         The project’s software effort and costs are      The project’s software effort and costs are        Strength
                   tracked, and corrective actions are taken as     tracked, and corrective action is taken as
                   necessary.                                       necessary.
                                                                                                                           (continued)




                                            Page 42                                   GAO/AIMD-99-35 Customs Service Modernization
                                         Chapter 4
                                         Software Project Tracking and Oversight




                                                                Automated Export System
                 Key practice                                       Finding                                            Rating
Activity 7       The project’s critical computer resources are      The project’s critical computer resources are      Weakness
                 tracked, and corrective actions are taken as       not tracked.
                 necessary.
Activity 8       The project’s software schedule is tracked,    The project’s software schedule is not tracked. Weakness
                 and corrective actions are taken as necessary.
Activity 9       Software engineering technical activities are      Software engineering technical activities are      Weakness
                 tracked, and corrective actions are taken as       not tracked.
                 necessary.
Activity 10      The software risks associated with cost,           The software risks associated with cost,           Weakness
                 resource, schedule, and technical aspects of       resource, schedule, and technical aspects of
                 the project are tracked.                           the project are not tracked.
Activity 11      Actual measurement data and replanning data Actual measurement data and replanning data Weakness
                 for the software project are recorded.      for the software project are not recorded.
Activity 12      The software engineering group conducts            The software engineering group does not       Weakness
                 periodic internal reviews to track technical       conduct periodic internal reviews to track
                 progress, plans, performance, and issues           technical progress, plans, performance, and
                 against the software development plan.             issues against the software development plan.
Activity 13      Formal reviews to address the                      No documented procedure exists for                 Weakness
                 accomplishments and results of the software        conducting formal reviews to address the
                 project are conducted at selected project          accomplishments and results of the software
                 milestones according to a documented               project at selected project milestones.
                 procedure.
Measurement 1    Measurements are made and used to                  Measurements are not made and used to              Weakness
                 determine the status of the software tracking      determine the status of software tracking and
                 and oversight activities.                          oversight activities.
Verification 1   The activities for software project tracking and   The activities for software project tracking and   Weakness
                 oversight are reviewed with senior                 oversight are not reviewed with senior
                 management on a periodic basis.                    management on a periodic basis.
Verification 2   The activities for software project tracking and   The activities for software project tracking and   Weakness
                 oversight are reviewed with the project            oversight are not reviewed with the project
                 manager on both a periodic and event-driven        manager.
                 basis.
Verification 3   The software quality assurance group reviews       The software quality assurance group does          Weakness
                 and/or audits the activities and work products     not review and/or audit the activities and work
                 for software project tracking and oversight and    products for software project tracking and
                 reports the results.                               oversight and report the results.




                                         Page 43                                      GAO/AIMD-99-35 Customs Service Modernization
                                             Chapter 4
                                             Software Project Tracking and Oversight




Table 4.4: Software Project Tracking and Oversight Findings for Administrative Security System
                                                         Administrative Security System
                    Key practice                                     Finding                                           Rating
Commitment 1        A project software manager is designated to      A project software manager is designated to       Strength
                    be responsible for the project’s software        be responsible for the project’s software
                    activities and results.                          activities and results.
Commitment 2        The project follows a written organizational     There is no organizational policy for managing Weakness
                    policy for managing the software project.        the software project.
Ability 1           A software development plan for the software     The project plan contains the software            Observation
                    project is documented and approved.              development plan; however, this plan has not
                                                                     been approved.
Ability 2           The project software manager explicitly          The project software manager does not             Weakness
                    assigns responsibility for software work         explicitly assign responsibility for software
                    products and activities.                         work products and activities.
Ability 3           Adequate resources and funding are provided Adequate resources and funding are not                 Weakness
                    for tracking the software project.          provided for tracking the software project.

Ability 4           The software managers are trained in             The software managers are not trained in          Weakness
                    managing the technical and personnel             managing the technical and personnel
                    aspects of the software project.                 aspects of the software project.
Ability 5           First-line software managers receive             First-line software managers did not receive      Weakness
                    orientation in the technical aspects of the      orientation on the technical aspects of the
                    software project.                                software project.
Activity 1          A documented software development plan is        A documented software development plan is         Strength
                    used for tracking the software activities and    used for tracking the software activities and
                    communicating status.                            communicating status.
Activity 2          The project’s software development plan is   No documented procedure exists for revising           Weakness
                    revised according to a documented procedure. the software development plan.
Activity 3          Software project commitments and changes to      Software project commitments and changes to Observation
                    commitments made to individuals and groups       commitments made to individuals and groups
                    external to the organization are reviewed with   external to the organization are reviewed in
                    senior management according to a                 periodic meetings. However, there is no
                    documented procedure.                            documented procedure for these reviews.
Activity 4          Approved changes to commitments that affect      There is no evidence to show that changes to      Weakness
                    the software project are communicated to the     commitments are either approved by project
                    members of the software engineering group        managers or communicated to the software
                    and other software-related groups.               engineers and other software-related groups.
Activity 5          The sizes of the software work products (or      The sizes of the software work products (or       Weakness
                    sizes of the changes to the software work        sizes of the changes to the software work
                    products) are tracked, and corrective actions    products) are not tracked.
                    are taken as necessary.
Activity 6          The project’s software effort and costs are      The project’s software effort and costs are not   Weakness
                    tracked, and corrective actions are taken as     tracked.
                    necessary.
Activity 7          The project’s critical computer resources are    The project’s critical computer resources are     Weakness
                    tracked, and corrective actions are taken as     not tracked.
                    necessary.
                                                                                                                           (continued)



                                             Page 44                                   GAO/AIMD-99-35 Customs Service Modernization
                                         Chapter 4
                                         Software Project Tracking and Oversight




                                                          Administrative Security System
                 Key practice                                       Finding                                            Rating
Activity 8       The project’s software schedule is tracked,    The project’s software schedule is tracked,    Strength
                 and corrective actions are taken as necessary. and corrective actions are taken as necessary.
Activity 9       Software engineering technical activities are      Software engineering technical activities are      Weakness
                 tracked, and corrective actions are taken as       not tracked.
                 necessary.
Activity 10      The software risks associated with cost,           The software risks associated with cost,           Weakness
                 resource, schedule, and technical aspects of       resource, schedule, and technical aspects of
                 the project are tracked.                           the project are not tracked.
Activity 11      Actual measurement data and replanning data Actual measurement data and replanning data Weakness
                 for the software project are recorded.      for the software project are not recorded.
Activity 12      The software engineering group conducts            The software engineering group does            Weakness
                 periodic internal reviews to track technical       not conduct periodic internal reviews to track
                 progress, plans, performance, and issues           technical progress, plans, performance, and
                 against the software development plan.             issues against the software development plan.
Activity 13      Formal reviews to address the                      No documented procedure exists for                 Weakness
                 accomplishments and results of the software        conducting formal reviews to address the
                 project are conducted at selected project          accomplishments and results of the software
                 milestones according to a documented               project at selected project milestones.
                 procedure.
Measurement 1    Measurements are made and used to                  Measurements are not made and used to              Weakness
                 determine the status of the software tracking      determine the status of software tracking and
                 and oversight activities.                          oversight activities.
Verification 1   The activities for software project tracking and   The activities for software project tracking and   Weakness
                 oversight are reviewed with senior                 oversight are not reviewed with senior
                 management on a periodic basis.                    management on a periodic basis.
Verification 2   The activities for software project tracking and   The activities for software project tracking and   Weakness
                 oversight are reviewed with the project            oversight are not reviewed with the project
                 manager on both a periodic and event-driven        manager.
                 basis.
Verification 3   The software quality assurance group reviews       No software quality assurance group exists;      Weakness
                 and/or audits the activities and work products     therefore, there are no reviews and/or audits of
                 for software project tracking and oversight and    the activities and work products for software
                 reports the results.                               project tracking and oversight.


                                         Despite several practice strengths in this KPA, the number and significance
Conclusions                              of the practice weaknesses that we found mean that Customs’ current
                                         process for tracking and overseeing its projects is not repeatable, thereby
                                         increasing the chances of its software projects being late, costing more
                                         than expected, and not performing as intended.




                                         Page 45                                      GAO/AIMD-99-35 Customs Service Modernization
Chapter 5

Software Quality Assurance


                    The purpose of software quality assurance is to independently review and
                    audit the software products and activities to verify that they comply with
                    the applicable procedures and standards and to provide the software
                    project and higher-level managers with the results of these independent
                    reviews and audits.

                    According to the SW-CMM, a repeatable software quality assurance process,
                    among other things, includes (1) preparing a software quality assurance
                    plan for the project according to a documented procedure, (2) having a
                    written organizational policy for implementing software quality assurance,
                    (3) conducting audits of designated work processes and products to verify
                    compliance, (4) documenting deviations identified in the software
                    activities and software work products and handling them according to a
                    documented procedure, and (5) having experts independent of the
                    software quality assurance group periodically review the activities and
                    work products of the project’s software quality assurance group.


                    All of the projects evaluated had extensive and significant software quality
Customs Lacks a     assurance practice weaknesses. For example, two of the projects did not
Software Quality    have a software quality assurance plan; and none of the projects (1) had a
Assurance Process   written organizational policy for implementing software quality assurance,
                    (2) conducted audits of designated work products to verify compliance,
                    (3) documented deviations identified in the software activities and
                    software work products and handled them according to a documented
                    procedure, or (4) had experts independent of the software quality
                    assurance group periodically review the group’s work products. In fact,
                    only one of the projects, AES, had any software quality assurance practice
                    strengths, and these strengths were limited to only a few practices. In this
                    case, the project had assigned responsibility for software quality assurance
                    to a single individual and, for example, a software quality assurance plan
                    had been drafted, although not according to a documented procedure.
                    This virtual absence of software quality assurance on Customs’ software
                    projects increases greatly the risk of software process and product
                    standards not being met, which in turn increases the risk of software not
                    performing as intended, and costing more and taking longer to develop
                    than necessary.

                    Table 5.1 provides a comprehensive list of the three projects’ strengths,
                    weaknesses, and observations for the software quality assurance KPA. The
                    specific findings supporting the practice ratings cited in table 5.1 are in
                    tables 5.2 through 5.4.



                    Page 46                            GAO/AIMD-99-35 Customs Service Modernization
                                             Chapter 5
                                             Software Quality Assurance




Table 5.1: Software Quality Assurance
                     Key practice                                                NCAP 0.1       AES              Administrative
Commitment 1        The project follows a written organizational policy for      Weakness       Weakness         Weakness
                    implementing software quality assurance (SQA).
Ability 1           A group that is responsible for coordinating and             Observation    Strength         Weakness
                    implementing SQA for the project (i.e., the SQA group)
                    exists.
Ability 2           Adequate resources and funding are provided for              Weakness       Weakness         Weakness
                    performing the SQA activities.
Ability 3           Members of the SQA group are trained to perform their        Weakness       Strength         Weakness
                    SQA activities.
Ability 4           The members of the software project receive orientation      Weakness       Strength         Weakness
                    on the role, responsibilities, authority, and value of the
                    SQA group.
Activity 1          A SQA plan is prepared for the software project              Weakness       Observation      Weakness
                    according to a documented procedure.
Activity 2          The SQA group’s activities are performed in accordance Weakness             Weakness         Weakness
                    with the SQA plan.
Activity 3          The SQA group participates in the preparation and            Weakness       Weakness         Weakness
                    review of the project’s software development plan,
                    standards, and procedures.
Activity 4          The SQA group reviews the software engineering               Weakness       Weakness         Weakness
                    activities to verify compliance.
Activity 5          The SQA group audits designated software work                Weakness       Weakness         Weakness
                    products to verify compliance.
Activity 6          The SQA group periodically reports the results of its        Weakness       Weakness         Weakness
                    activities to the software engineering group.
Activity 7          Deviations identified in the software activities and         Weakness       Weakness         Weakness
                    software work products are documented and handled
                    according to a documented procedure.
Activity 8          The SQA group conducts periodic reviews of its               Weakness       Weakness         Weakness
                    activities and findings with the customer’s SQA
                    personnel, as appropriate.
Measurement 1       Measurements are made and used to determine the              Weakness       Weakness         Weakness
                    cost and schedule status of the SQA activities.
Verification 1      The SQA activities are reviewed with senior                  Weakness       Weakness         Weakness
                    management on a periodic basis.
Verification 2      The SQA activities are reviewed with the project             Weakness       Weakness         Weakness
                    manager on both a periodic and event-driven basis.
Verification 3      Experts independent of the SQA group periodically            Weakness       Weakness         Weakness
                    review the activities and software work products of the
                    project’s SQA group.




                                             Page 47                                   GAO/AIMD-99-35 Customs Service Modernization
                                            Chapter 5
                                            Software Quality Assurance




Table 5.2: Software Quality Assurance Findings for NCAP 0.1
                                                    National Customs Automation Program 0.1
                   Key practice                                       Finding                                           Rating
Commitment 1       The project follows a written organizational       There is no written organizational policy for     Weakness
                   policy for implementing software quality           implementing SQA.
                   assurance (SQA).
Ability 1          A group that is responsible for coordinating    Although there is no group responsible for           Observation
                   and implementing SQA for the project (i.e., the coordinating and implementing SQA for the
                   SQA group) exists.                              project, there are plans to establish one and
                                                                   assign responsibility.
Ability 2          Adequate resources and funding are provided There is no SQA group, and no resources and              Weakness
                   for performing the SQA activities.          funding are provided for performing the SQA
                                                               activities.
Ability 3          Members of the SQA group are trained to            There is no SQA group or plan to provide SQA Weakness
                   perform their SQA activities.                      training.
Ability 4          The members of the software project receive        Project staff do not receive orientation on the   Weakness
                   orientation on the role, responsibilities,         role, responsibilities, authority, and value of
                   authority, and value of the SQA group.             the SQA group.
Activity 1         The SQA plan is prepared for the software    There is no SQA plan or documented                      Weakness
                   project according to a documented procedure. procedure for preparing one.
Activity 2         The SQA group’s activities are performed in        There is no SQA group.                            Weakness
                   accordance with the SQA plan.
Activity 3         The SQA group participates in the preparation There is no SQA group.                                 Weakness
                   and review of the project’s software
                   development plan, standards, and procedures.
Activity 4         The SQA group reviews the software                 There is no SQA group.                            Weakness
                   engineering activities to verify compliance.
Activity 5         The SQA group audits designated software           There is no SQA group.                            Weakness
                   work products to verify compliance.
Activity 6         The SQA group periodically reports the results There is no SQA group.                                Weakness
                   of its activities to the software engineering
                   group.
Activity 7         Deviations identified in the software activities   Procedures for handling deviations in testing     Weakness
                   and software work products are documented          activities are documented. However,
                   and handled according to a documented              procedures for handling deviations in other
                   procedure.                                         software development activities (such as
                                                                      compliance with organizational policy and
                                                                      standards and adherence to software
                                                                      development plan) are not documented.
Activity 8         The SQA group conducts periodic reviews of         There is no SQA group.                            Weakness
                   its activities and findings with the customer’s
                   SQA personnel, as appropriate.
Measurement 1      Measurements are made and used to                  There is no SQA group.                            Weakness
                   determine the cost and schedule status of the
                   SQA activities.
Verification 1     The SQA activities are reviewed with senior        There is no SQA group.                            Weakness
                   management on a periodic basis.
                                                                                                                            (continued)


                                            Page 48                                     GAO/AIMD-99-35 Customs Service Modernization
                                         Chapter 5
                                         Software Quality Assurance




                                                    National Customs Automation Program 0.1
                 Key practice                                      Finding                                     Rating
Verification 2   The SQA activities are reviewed with the          There is no SQA group.                      Weakness
                 project manager on both a periodic and
                 event-driven basis.
Verification 3   Experts independent of the SQA group              There is no SQA group.                      Weakness
                 periodically review the activities and software
                 work products of the project’s SQA group.




                                         Page 49                                   GAO/AIMD-99-35 Customs Service Modernization
                                             Chapter 5
                                             Software Quality Assurance




Table 5.3: Software Quality Assurance Findings for AES
                                                                 Automated Export System
                    Key practice                                       Finding                                         Rating
Commitment 1        The project follows a written organizational       There is no written organizational policy for   Weakness
                    policy for implementing software quality           implementing SQA.
                    assurance (SQA).
Ability 1           A group that is responsible for coordinating    A single individual is responsible for             Strength
                    and implementing SQA for the project (i.e., the coordinating and implementing SQA for the
                    SQA group) exists.                              project.

Ability 2           Adequate resources and funding are provided Adequate resources and funding are not                 Weakness
                    for performing the SQA activities.          provided for performing the SQA activities.
Ability 3           Members of the SQA group are trained to            Training has been provided to the single        Strength
                    perform their SQA activities.                      individual responsible for SQA to prepare him
                                                                       to perform his activities.
Ability 4           The members of the software project receive        Members of the software project are oriented    Strength
                    orientation on the role, responsibilities,         on the role, responsibilities, authority, and
                    authority, and value of the SQA group.             value of SQA. Briefings are held during
                                                                       retreats.
Activity 1          A SQA plan is prepared for the software      The SQA plan is not prepared according to a           Observation
                    project according to a documented procedure. documented procedure; however, there is a
                                                                 draft plan.
Activity 2          The SQA group’s activities are performed in        The SQA group’s activities are not performed    Weakness
                    accordance with the SQA plan.                      in accordance with the SQA plan.
Activity 3          The SQA group participates in the preparation The individual responsible for SQA does not      Weakness
                    and review of the project’s software          participate in the preparation and review of the
                    development plan, standards, and procedures. project’s software development plan,
                                                                  standards, and procedures.
Activity 4          The SQA group reviews the software                 The individual responsible for SQA does not     Weakness
                    engineering activities to verify compliance.       review the software engineering activities to
                                                                       verify compliance.
Activity 5          The SQA group audits designated software           The individual responsible for SQA does not     Weakness
                    work products to verify compliance.                audit software work products to verify
                                                                       compliance.
Activity 6          The SQA group periodically reports the results The individual responsible for SQA periodically Weakness
                    of its activities to the software engineering  reports the results of testing activities to the
                    group.                                         software engineering group. Other activities,
                                                                   such as the results of reviews of work products
                                                                   and audits, are not reported.
Activity 7          Deviations identified in the software activities   Procedures for handling deviations in testing   Weakness
                    and software work products are documented          activities are documented. However,
                    and handled according to a documented              procedures for handling deviations in other
                    procedure.                                         software development activities (such as
                                                                       compliance with organizational policy and
                                                                       standards and adherence to the software
                                                                       development plan) are not documented.
                                                                                                                           (continued)




                                             Page 50                                     GAO/AIMD-99-35 Customs Service Modernization
                                         Chapter 5
                                         Software Quality Assurance




                                                             Automated Export System
                 Key practice                                      Finding                                           Rating
Activity 8       The SQA group conducts periodic reviews of        The SQA group does not conduct periodic           Weakness
                 its activities and findings with the customer’s   reviews of its activities and findings with the
                 SQA personnel, as appropriate.                    customer’s SQA personnel, as appropriate.
Measurement 1    Measurements are made and used to                 Measurements are not made to determine the Weakness
                 determine the cost and schedule status of the     cost and schedule status of the SQA activities.
                 SQA activities.
Verification 1   The SQA activities are reviewed with senior       Senior management is not made aware of the        Weakness
                 management on a periodic basis.                   SQA activities.
Verification 2   The SQA activities are reviewed with the          The project manager and team members are          Weakness
                 project manager on both a periodic and            not made aware of SQA activities.
                 event-driven basis.
Verification 3   Experts independent of the SQA group              SQA’s activities and software work products       Weakness
                 periodically review the activities and software   are not reviewed by experts independent of
                 work products of the project’s SQA group.         the SQA group.




                                         Page 51                                     GAO/AIMD-99-35 Customs Service Modernization
                                             Chapter 5
                                             Software Quality Assurance




Table 5.4: Software Quality Assurance Findings for Administrative Security System
                                                         Administrative Security System
                    Key practice                                       Finding                                           Rating
Commitment 1        The project follows a written organizational       There is no written organizational policy for     Weakness
                    policy for implementing software quality           implementing SQA.
                    assurance (SQA).
Ability 1           A group that is responsible for coordinating    No group is responsible for coordinating and         Weakness
                    and implementing SQA for the project (i.e., the implementing SQA for the project.
                    SQA group) exists.
Ability 2           Adequate resources and funding are provided Adequate resources and funding are not                   Weakness
                    for performing the SQA activities.          provided for performing the SQA activities.
Ability 3           Members of the SQA group are trained to            There is no SQA group.                            Weakness
                    perform their SQA activities.
Ability 4           The members of the software project receive        Project staff do not receive orientation on the   Weakness
                    orientation on the role, responsibilities,         role, responsibilities, authority, and value of
                    authority, and value of the SQA group.             the SQA group.
Activity 1          A SQA plan is prepared for the software      There is no documented procedure for                    Weakness
                    project according to a documented procedure. preparing a SQA plan.
Activity 2          The SQA group’s activities are performed in        There is no SQA group.                            Weakness
                    accordance with the SQA plan.
Activity 3          The SQA group participates in the preparation There is no SQA group.                                 Weakness
                    and review of the project’s software
                    development plan, standards, and procedures.
Activity 4          The SQA group reviews the software                 There is no SQA group.                            Weakness
                    engineering activities to verify compliance.
Activity 5          The SQA group audits designated software           There is no SQA group.                            Weakness
                    work products to verify compliance.
Activity 6          The SQA group periodically reports the results There is no SQA group.                                Weakness
                    of its activities to the software engineering
                    group.
Activity 7          Deviations identified in the software activities   Procedures for handling deviations in testing     Weakness
                    and software work products are documented          activities are documented. However,
                    and handled according to a documented              procedures for handling deviations in other
                    procedure.                                         software development activities (such as
                                                                       compliance with organizational policy and
                                                                       standards and adherence to the software
                                                                       development plan) are not documented.
Activity 8          The SQA group conducts periodic reviews of         There is no SQA group.                            Weakness
                    its activities and findings with the customer’s
                    SQA personnel, as appropriate.
Measurement 1       Measurements are made and used to                  There is no SQA group.                            Weakness
                    determine the cost and schedule status of the
                    SQA activities.
Verification 1      The SQA activities are reviewed with senior        The SQA activities are not reviewed with senior Weakness
                    management on a periodic basis.                    management on a periodic basis.
                                                                                                                             (continued)




                                             Page 52                                     GAO/AIMD-99-35 Customs Service Modernization
                                         Chapter 5
                                         Software Quality Assurance




                                                          Administrative Security System
                 Key practice                                      Finding                                        Rating
Verification 2   The SQA activities are reviewed with the          The SQA activities are not reviewed with the   Weakness
                 project manager on both a periodic and            project manager on both a periodic and
                 event-driven basis.                               event-driven basis.
Verification 3   Experts independent of the SQA group              There is no SQA group.                         Weakness
                 periodically review the activities and software
                 work products of the project’s SQA group.


                                         Customs’ software quality assurance process has many weaknesses and is,
Conclusions                              therefore, undefined and undisciplined. As a result, Customs cannot
                                         provide management with independent information about adherence to
                                         software process and product standards. To develop and maintain
                                         software effectively, Customs must adopt a structured and rigorous
                                         approach to software quality assurance.




                                         Page 53                                    GAO/AIMD-99-35 Customs Service Modernization
Chapter 6

Software Configuration Management


                     The purpose of software configuration management is to establish and
                     maintain the integrity of the products of the software project throughout
                     the project’s software life-cycle. Software configuration management
                     involves establishing product baselines and systematically controlling
                     changes to them.

                     According to the SW-CMM, a repeatable software configuration management
                     process, among other things, includes (1) preparing a software
                     configuration management plan according to a documented procedure,
                     (2) establishing a configuration management library system as a repository
                     for the software baselines, (3) identifying software work products to be
                     placed under configuration management, (4) controlling the release of
                     products from the software baseline library according to a documented
                     procedure, (5) following a written organizational policy for implementing
                     software configuration management, (6) recording the status of
                     configuration items/units according to a documented procedure,
                     (7) making and using measurements to determine the status of the
                     software configuration management activities, and (8) reviewing software
                     configuration management activities with senior management on a
                     periodic basis.


                     Customs’ processes for software configuration management show
Customs’ Software    strengths in several activities. For example, all three projects had
Configuration        developed software configuration management plans according to a
Management Process   documented procedure. Also, two of the projects (NCAP 0.1 and AES)
                     established configuration management library systems as repositories for
Is Immature          the software baselines, identified software work products to be placed
                     under configuration management, and controlled the release of products
                     from the software baseline library according to a documented procedure.

                     However, the projects had many practice weakness that collectively
                     jeopardize Customs’ ability to maintain the integrity of the projects’
                     software products. For example, none of the projects had a written
                     organizational policy for implementing software configuration
                     management, and none had documented procedures for recording the
                     status of configuration items (e.g., code, documents). Moreover, none of
                     the projects made or used measurements to determine the status of the
                     software configuration management activities, or reviewed software
                     configuration management activities with senior management on a
                     periodic basis.




                     Page 54                            GAO/AIMD-99-35 Customs Service Modernization
                                            Chapter 6
                                            Software Configuration Management




                                            Table 6.1 provides a comprehensive list of the three projects’ strengths and
                                            weaknesses for the software configuration management KPA. The specific
                                            findings supporting the practice ratings cited in table 6.1 are in tables 6.2
                                            through 6.4.


Figure 6.1: Software Configuration Management
                     Key practice                                               NCAP 0.1        AES              Administrative
Commitment 1       The project follows a written organizational policy for      Weakness        Weakness         Weakness
                   implementing software configuration management
                   (SCM).
Ability 1          A board having the authority for managing the project’s      Weakness        Strength         Weakness
                   software baselines (i.e., a software configuration control
                   board) exists or is established.
Ability 2          A group that is responsible for coordinating and             Weakness        Strength         Strength
                   implementing SCM for the project (i.e., the SCM group)
                   exists.
Ability 3          Adequate resources and funding are provided for              Weakness        Weakness         Weakness
                   performing the SCM activities.
Ability 4          Members of the SCM group are trained in the                  Weakness        Strength         Weakness
                   objectives, procedures, and methods for performing
                   their SCM activities.
Ability 5          Members of the software engineering group and other          Weakness        Strength         Weakness
                   software-related groups are trained to perform their
                   SCM activities.
Activity 1         A SCM plan is prepared for each software project             Strength        Strength         Strength
                   according to a documented procedure.
Activity 2         A documented and approved SCM plan is used as the            Weakness        Weakness         Weakness
                   basis for performing the SCM activities.
Activity 3         A configuration management library system is                 Strength        Strength         Weakness
                   established as a repository for the software baselines.
Activity 4         The software work products to be placed under                Strength        Strength         Weakness
                   configuration management are identified.
Activity 5         Change requests and problem reports for all                  Strength        Weakness         Weakness
                   configuration items/risks are initiated, recorded,
                   reviewed, approved, and tracked according to a
                   documented procedure.
Activity 6         Changes to baselines are controlled according to a           Strength        Weakness         Weakness
                   documented procedure.
Activity 7         Products from the software baseline library are created      Strength        Strength         Weakness
                   and their release is controlled according to a
                   documented procedure.
Activity 8         The status of configuration items/units is recorded          Weakness        Weakness         Weakness
                   according to a documented procedure.
Activity 9         Standard reports documenting the SCM activities and     Weakness             Weakness         Weakness
                   the contents of the software baseline are developed and
                   made available to affected groups and individuals.
                                                                                                                        (continued)


                                            Page 55                                    GAO/AIMD-99-35 Customs Service Modernization
                                        Chapter 6
                                        Software Configuration Management




                 Key practice                                            NCAP 0.1       AES              Administrative
Activity 10      Software baseline audits are conducted according to a   Weakness       Weakness         Weakness
                 documented procedure.
Measurement 1    Measurements are made and used to determine the         Weakness       Weakness         Weakness
                 status of the SCM activities.
Verification 1   The SCM activities are reviewed with senior             Weakness       Weakness         Weakness
                 management on a periodic basis.
Verification 2   The SCM activities are reviewed with the project        Weakness       Strength         Weakness
                 manager on both a periodic and event-driven basis.
Verification 3   The SCM group periodically audits software baselines to Weakness       Weakness         Weakness
                 verify that they conform to the documentation that
                 defines them.
Verification 4   The software quality assurance group reviews and/or     Weakness       Weakness         Weakness
                 audits the activities and work products for SCM and
                 reports the results.




                                        Page 56                                GAO/AIMD-99-35 Customs Service Modernization
                                          Chapter 6
                                          Software Configuration Management




Table 6.2: Software Configuration Management Findings for NCAP 0.1
                                                  National Customs Automation Program 0.1
                   Key practice                                     Finding                                        Rating
Commitment 1       The project follows a written organizational     The project has no organizational policy for   Weakness
                   policy for implementing software configuration   implementing SCM.
                   management (SCM).
Ability 1          A board having the authority for managing the    The project does not have a SCM board with     Weakness
                   project’s software baselines (i.e., a software   authority for managing software baselines.
                   configuration control board) exists or is
                   established.
Ability 2          A group that is responsible for coordinating    The project does not have a group that is     Weakness
                   and implementing SCM for the project (i.e., the responsible for coordinating and implementing
                   SCM group) exists.                              SCM functions.
Ability 3          Adequate resources and funding are provided Adequate resources and funding are not              Weakness
                   for performing the SCM activities.          provided for performing the SCM activities.
Ability 4          Members of the SCM group are trained in the      There is no SCM group.                         Weakness
                   objectives, procedures, and methods for
                   performing their SCM activities.
Ability 5          Members of the software engineering group        Members of the software engineering group      Weakness
                   and other software-related groups are trained    and other software-related groups are not
                   to perform their SCM activities.                 trained to perform their SCM activities
Activity 1         A SCM plan is prepared for each software     A SCM plan was prepared according to               Strength
                   project according to a documented procedure. procedures documented in the October 1996
                                                                SDLC.
Activity 2         A documented and approved SCM plan is            The documented and approved SCM plan is        Weakness
                   used as the basis for performing the SCM         used as the basis for performing code control;
                   activities.                                      however, the plan is not used as a basis for
                                                                    doing SCM on software documentation and
                                                                    other software engineering products such as
                                                                    cost estimates and schedules.
Activity 3         A configuration management library system is     A configuration management library system is   Strength
                   established as a repository for the software     established as a repository for the software
                   baselines.                                       baselines.
Activity 4         The software work products to be placed          The software work products to be placed        Strength
                   under configuration management are               under configuration management are
                   identified.                                      identified.
Activity 5         Change requests and problem reports for all      Change requests and problem items/units are    Strength
                   configuration items/risks are initiated,         initiated, recorded, reviewed, approved, and
                   recorded, reviewed, approved, and tracked        tracked according to a procedure
                   according to a documented procedure.             documented in the SCM plan.
Activity 6         Changes to baselines are controlled              Changes to baselines are controlled            Strength
                   according to a documented procedure.             according to a documented procedure in the
                                                                    SCM plan.
Activity 7         Products from the software baseline library are Products from the software baseline library are Strength
                   created and their release is controlled         created and their release is controlled
                   according to a documented procedure.            according to the SCM plan.
                                                                                                                       (continued)




                                          Page 57                                    GAO/AIMD-99-35 Customs Service Modernization
                                          Chapter 6
                                          Software Configuration Management




                                                    National Customs Automation Program 0.1
                 Key practice                                     Finding                                        Rating
Activity 8       The status of configuration items/units is       The status of SCM items/units are not          Weakness
                 recorded according to a documented               recorded according to a documented
                 procedure.                                       procedure.
Activity 9       Standard reports documenting the SCM         Standard reports documenting the SCM             Weakness
                 activities and the contents of the software  activities and contents of the software baseline
                 baseline are developed and made available to are not developed.
                 affected groups and individuals.
Activity 10      Software baseline audits are conducted           Software baseline audits are not conducted.    Weakness
                 according to a documented procedure.
Measurement 1    Measurements are made and used to                No measurements are taken to determine the     Weakness
                 determine the status of the SCM activities.      status of SCM activities.
Verification 1   The SCM activities are reviewed with senior      The SCM activities are not reviewed with       Weakness
                 management on a periodic basis.                  senior management on a periodic basis.
Verification 2   The SCM activities are reviewed with the         The SCM activities are not reviewed with the   Weakness
                 project manager on both a periodic and           project manager on both a periodic and
                 event-driven basis.                              event-driven basis.
Verification 3   The SCM group periodically audits software       No SCM audits are done to verify that software Weakness
                 baselines to verify that they conform to the     baselines conform to the documentation that
                 documentation that defines them.                 defines them.
Verification 4   The software quality assurance group reviews     There is no quality assurance group; therefore, Weakness
                 and/or audits the activities and work products   no one reviews and/or audits the activities and
                 for SCM and reports the results.                 work products for SCM activities performed.




                                          Page 58                                  GAO/AIMD-99-35 Customs Service Modernization
                                           Chapter 6
                                           Software Configuration Management




Table 6.3: Software Configuration Management Findings for AES
                                                          Automated Export System
                   Key practice                                    Finding                                        Rating
Commitment 1       The project follows a written organizational    The project does not have a written            Weakness
                   policy for implementing software                organizational policy for SCM.
                   configuration management (SCM).
Ability 1          A board having the authority for managing       AES has a Configuration Control Board. This    Strength
                   the project’s software baselines (i.e., a       board has the authority for managing the
                   software configuration control board) exists    project’s software baselines. The board
                   or is established.                              consists of program support and user
                                                                   personnel.
Ability 2          A group that is responsible for coordinating    A group that is responsible for coordinating   Strength
                   and implementing SCM for the project (i.e.,     and implementing SCM for the project (i.e.,
                   the SCM group) exists.                          the SCM group) exists.
Ability 3          Adequate resources and funding are              Adequate resources and funding are not         Weakness
                   provided for performing the SCM activities.     provided for performing the SCM activities.

Ability 4          Members of the SCM group are trained in the Members of the SCM group have extensive            Strength
                   objectives, procedures, and methods for     experience in the objectives, procedures,
                   performing their SCM activities.            and methods for performing their SCM
                                                               activities and receive on-the-job training.
Ability 5          Members of the software engineering group       Members of the software engineering group Strength
                   and other software-related groups are           and other software-related groups have been
                   trained to perform their SCM activities.        briefed on performing their SCM duties at
                                                                   periodic retreat meetings.
Activity 1         A SCM plan is prepared for each software        A SCM plan was prepared according to      Strength
                   project according to a documented               procedures documented in the October 1996
                   procedure.                                      SDLC.
Activity 2         A documented and approved SCM plan is           The documented and approved SCM plan is Weakness
                   used as the basis for performing the SCM        used as the basis for performing code
                   activities.                                     control; however, the plan is not used as a
                                                                   basis for doing SCM on software
                                                                   documentation and other software
                                                                   engineering products such as cost estimates
                                                                   and schedules.
Activity 3         A configuration management library system       The AES library is used as a repository for    Strength
                   is established as a repository for the software software baselines.
                   baselines.
Activity 4         The software work products to be placed         Software work products to be placed under      Strength
                   under configuration management are              SCM are identified in the SCM plan.
                   identified.
Activity 5         Change requests and problem reports for all     Change requests and problem reports for all    Weakness
                   configuration items/units are initiated,        configuration items are not initiated,
                   recorded, reviewed, approved, and tracked       recorded, reviewed, approved, and tracked
                   according to a documented procedure.            according to a documented procedure.
                                                                                                                           (continued)




                                           Page 59                                    GAO/AIMD-99-35 Customs Service Modernization
                                          Chapter 6
                                          Software Configuration Management




                                                               Automated Export System
                 Key practice                                     Finding                                          Rating
Activity 6       Changes to baselines are controlled              The procedures for controlling changes to   Weakness
                 according to a documented procedure.             baselines are documented in the SCM and
                                                                  SQA plans. However, there was no evidence
                                                                  provided to show that these procedures were
                                                                  being followed.
Activity 7       Products from the software baseline library      Products from the baseline library are           Strength
                 are created and their release is controlled      created, and releases are controlled
                 according to a documented procedure.             according to procedures in the SCM plan.
Activity 8       The status of configuration items/units is       The status of configuration items/units is not   Weakness
                 recorded according to a documented               recorded according to a documented
                 procedure.                                       procedure.
Activity 9       Standard reports documenting the SCM             No reports documenting SCM activities and        Weakness
                 activities and the contents of the software      contents of the software baseline are made
                 baseline are developed and made available        available to affected groups and individuals.
                 to affected groups and individuals.
Activity 10      Software baseline audits are conducted           Software baseline audits are not conducted.      Weakness
                 according to a documented procedure.
Measurement 1    Measurements are made and used to                Measurements are not made and used to            Weakness
                 determine the status of the SCM activities.      determine the status of the SCM activities.
Verification 1   The SCM activities are reviewed with senior      The SCM activities are not reviewed with         Weakness
                 management on a periodic basis.                  senior management on a periodic basis.
Verification 2   The SCM activities are reviewed with the         Project management is updated on SCM           Strength
                 project manager on both a periodic and           activities on a weekly basis and bi-monthly at
                 event-driven basis.                              the retreat meetings, and as events warrant.
Verification 3   The SCM group periodically audits software       The SCM group does not verify that the           Weakness
                 baselines to verify that they conform to the     software baseline conforms to the
                 documentation that defines them.                 documentation that defines it.
Verification 4   The software quality assurance group             The SQA group does not perform reviews           Weakness
                 reviews and/or audits the activities and work    and/or audits of the activities and work
                 products for SCM and reports the results.        products for SCM and does not report the
                                                                  results.




                                          Page 60                                    GAO/AIMD-99-35 Customs Service Modernization
                                           Chapter 6
                                           Software Configuration Management




Table 6.4: Software Configuration Management Findings for Administrative Security System
                                                        Administrative Security System
                    Key practice                                     Finding                                           Rating
Commitment 1        The project follows a written organizational     The project does not have a written               Weakness
                    policy for implementing software configuration   organizational policy for implementing
                    management (SCM).                                software configuration management.
Ability 1           A board having the authority for managing the    The project does not have a board (software       Weakness
                    project’s software baselines (i.e., a software   configuration control board) with the authority
                    configuration control board) exists or is        for managing the software baselines.
                    established.
Ability 2           A group that is responsible for coordinating    The project manager is responsible for       Strength
                    and implementing SCM for the project (i.e., the coordinating and implementing SCM functions.
                    SCM group) exists.
Ability 3           Adequate resources and funding are provided Adequate resources and funding are not                 Weakness
                    for performing the SCM activities.          provided for performing the SCM activities.
Ability 4           Members of the SCM group are trained in the      There is no evidence that the personnel           Weakness
                    objectives, procedures, and methods for          assigned SCM functions on the project are
                    performing their SCM activities.                 trained in objectives, procedures, and
                                                                     methods for doing SCM.
Ability 5           Members of the software engineering group        Members of the software engineering group       Weakness
                    and other software-related groups are trained    are not trained to perform their SCM functions.
                    to perform their SCM activities.
Activity 1          A SCM plan is prepared for each software     A SCM plan was prepared according to                  Strength
                    project according to a documented procedure. procedures documented in the October 1996
                                                                 SDLC.
Activity 2          A documented and approved SCM plan is            Although there is a SCM plan for the project,     Weakness
                    used as the basis for performing the SCM         there is no evidence that it is used to perform
                    activities.                                      SCM functions.
Activity 3          A configuration management library system is     A configuration management library system is      Weakness
                    established as a repository for the software     established and used as a repository for
                    baselines.                                       software baselines for source code. However,
                                                                     other items in the software baseline (such as
                                                                     plans and schedules) are not identified or
                                                                     controlled.
Activity 4          The software work products to be placed          Software work products to be placed under         Weakness
                    under configuration management are               configuration management (other than source
                    identified.                                      code) are not identified.
Activity 5          Change requests and problem reports for all      The Automated Request For Service system          Weakness
                    configuration items/risks are initiated,         handles change requests and problem
                    recorded, reviewed, approved, and tracked        reports. However, there is no documented
                    according to a documented procedure.             procedure for its use and there was no
                                                                     evidence that the system was being used to
                                                                     initiate, record, review, approve, or track
                                                                     change requests and problem reports.
Activity 6          Changes to baselines are controlled              There is no documented procedure to control       Weakness
                    according to a documented procedure.             changes to baselines.

                                                                                                                           (continued)




                                           Page 61                                     GAO/AIMD-99-35 Customs Service Modernization
                                          Chapter 6
                                          Software Configuration Management




                                                          Administrative Security System
                 Key practice                                     Finding                                         Rating
Activity 7       Products from the software baseline library are There is no documented procedure that            Weakness
                 created and their release is controlled         defines how products from the software
                 according to a documented procedure.            baseline library should be created and
                                                                 released.
Activity 8       The status of configuration items/units is       The status of SCM items/units is not recorded   Weakness
                 recorded according to a documented               according to a documented procedure.
                 procedure.
Activity 9       Standard reports documenting the SCM         Affected groups and individuals are not           Weakness
                 activities and the contents of the software  notified of the SCM activities or informed of the
                 baseline are developed and made available to contents of software baselines.
                 affected groups and individuals.
Activity 10      Software baseline audits are conducted           Software baseline audits are not conducted.     Weakness
                 according to a documented procedure.
Measurement 1    Measurements are made and used to                No measurements are made to determine the       Weakness
                 determine the status of the SCM activities.      status of the SCM activities.
Verification 1   The SCM activities are reviewed with senior      The SCM activities are not reviewed with        Weakness
                 management on a periodic basis.                  senior management on a periodic basis.
Verification 2   The SCM activities are reviewed with the         Project managers are not updated on SCM         Weakness
                 project manager on both a periodic and           activities on a periodic basis.
                 event-driven basis.
Verification 3   The SCM group periodically audits software       The SCM does not periodically audit the         Weakness
                 baselines to verify that they conform to the     software baselines.
                 documentation that defines them.
Verification 4   The software quality assurance group reviews     There is no quality assurance group.            Weakness
                 and/or audits the activities and work products
                 for SCM and reports the results.


                                          Customs has many configuration management process weaknesses, and
Conclusions                               thus its capability to establish and maintain the integrity of the wide range
                                          of software products is nonrepeatable and ineffective. Without a mature
                                          configuration management process, Customs can lose control of the
                                          current software product baseline, potentially producing and using
                                          inconsistent product versions and creating operational problems.




                                          Page 62                                  GAO/AIMD-99-35 Customs Service Modernization
Chapter 7

Customs Lacks a Software Process
Improvement Program

                           To consistently develop software with specified functionality on time and
                           within budget, Customs must improve its software development
                           processes. According to SEI, an effective process improvement program
                           includes (1) establishing a process improvement management structure,
                           (2) developing a process improvement plan, (3) determining the
                           organization’s baseline capability and using this as a basis for targeting
                           process initiatives, and (4) dedicating adequate resources for
                           implementing the plan.

                           Although it has attempted in the past to initiate and sustain process
                           improvement activities, these activities were terminated without having
                           improved Customs processes. Currently, Customs has no software process
                           improvement program.


                           In 1996, SEI published a software process improvement model, called
SEI Has Defined a          IDEAL.1 This model has five phases: Initiating, Diagnosing, Establishing,
Five-Phase Model for       Acting, and Leveraging—IDEAL. Each of the phases is summarized below.
Software Process
                       •   Initiating phase: During this phase, an organization establishes the
Improvement                management structure of the process improvement program, defines and
                           assigns roles and responsibilities, allocates initial resources, develops a
                           plan to guide the organization through the first three phases of the
                           program, and obtains management approval and funding for the program.
                           Two key organizational components of the program management structure
                           established during this phase are a management steering group and a
                           software engineering process group (SEPG). Responsibility for this phase
                           rests with senior management.
                       •   Diagnosing phase: During this phase, the SEPG appraises the current level
                           of software process maturity to establish a baseline of the organization’s
                           process capability, and identifies any ongoing process improvement
                           initiatives. The SEPG then uses the baseline to identify weaknesses and
                           target process improvement activities. It also compares these targeted
                           activities with any ongoing process improvement activities and reconciles
                           any differences. Responsibility for this phase rests primarily with line
                           managers and practitioners.
                       •   Establishing phase: During this phase, the SEPG prioritizes the software
                           process improvement activities and develops strategies for pursuing them.
                           It then develops a process improvement action plan that details the
                           activities and strategies and includes measurable goals for the activities
                           and metrics for monitoring progress against the goals. Also during this

                           1
                            IDEAL: A User’s Guide for Software Process Improvement (CMU/SEI-96-HB-001).



                           Page 63                                     GAO/AIMD-99-35 Customs Service Modernization
                       Chapter 7
                       Customs Lacks a Software Process
                       Improvement Program




                       phase, the resources needed to implement the plan are committed and
                       training is provided for SEPG’s technical working groups, who will be
                       responsible for developing and testing new or improved processes.
                       Responsibility for this phase resides primarily with line managers and
                       practitioners.
                   •   Acting phase: In this phase, the work groups create and evaluate new and
                       improved processes. Evaluation of the processes is based on pilot tests
                       that are formally planned and executed. If the pilots are successful, the
                       work groups develop plans for organization-wide adoption and
                       institutionalization, and once approved, execute them. Responsibility for
                       this phase resides primarily with line managers and practitioners.
                   •   Leveraging phase: During this phase, results and lessons learned from
                       earlier phases are assessed and applied, as appropriate, to enhance the
                       process improvement program’s structure and plans. Responsibility for
                       this phase rests primarily with senior management.


                       In 1996, Customs initiated some limited software process improvement
Customs Does Not       activities. Specifically, it hired a contractor to develop a process
Currently Have a       improvement plan, which was completed in September 1996. According to
Software Process       the plan, Customs was to reach CMM level 2 process maturity (the
                       repeatable level) by 1998 and CMM level 3 (the defined level) by 2002.
Improvement            Customs began limited implementation of the plan in May of 1997, when it
Program                established process improvement teams for two KPAs—software project
                       planning and project tracking and oversight. Generally, the teams were
                       tasked with defining, implementing, and maintaining CMM-based
                       processes for their respective KPAs. Customs did not staff or fund any
                       other KPA improvement activities at this time. In August 1997, Customs
                       discontinued all process improvement activities. Customs officials stated
                       that this decision was based on the need to focus staff and resources on
                       the agency’s Year 2000 conversion program.

                       Currently, Customs does not have a software development process
                       improvement program, and it has not taken steps to initiate one. Although
                       it has assigned two people part-time to process improvement, it has not
                       assigned organizational responsibility and authority, established a
                       program management structure, developed a plan of action, and
                       committed resources needed (trained staff and funding) to execute the
                       plan.




                       Page 64                            GAO/AIMD-99-35 Customs Service Modernization
              Chapter 7
              Customs Lacks a Software Process
              Improvement Program




              Customs does not have an effective software development process
Conclusions   improvement program. As a result, it cannot expect to improve its
              immature software development processes.




              Page 65                            GAO/AIMD-99-35 Customs Service Modernization
Chapter 8

Overall Conclusions, Recommendations,
and Agency Comments

                         Customs develops and maintains software for systems that are critical to
                         its ability to fulfill its mission. However, its software development
                         processes are ad hoc and sometimes chaotic, and are not repeatable even
                         on a project-by-project basis. As a result, Customs’ success or failure in
                         developing software depends largely on specific individuals, rather than
                         on well-defined and disciplined software management practices. This
                         greatly reduces the probability that its software projects, whether new
                         developments or maintenance of existing software, will consistently
                         perform as intended and be delivered on schedule and within budget. For
                         Customs software projects to mature beyond this initial level, the agency
                         must implement basic management controls and instill self-discipline in its
                         software projects.

                         Customs acknowledges the importance of software process maturity and
                         the need to improve its software development processes. However, it does
                         not have a program for improving its software development processes and
                         has not begun to establish one. Until it does, Customs has no assurance
                         that its large investment in software development and maintenance will
                         produce systems that perform needed functions, on time, and within
                         budget.


                         We recommend that, after ensuring that its mission-critical systems are
Recommendations          Year 2000 compliant but before investing in major software development
                         efforts like ACE, the Commissioner of Customs direct the Chief Information
                         Officer to

                     •   assign responsibility and authority for software development process
                         improvement;
                     •   develop and implement a formal plan for software development process
                         improvement that is based on the software capability evaluation results
                         contained in this report and specifies measurable goals and time frames,
                         prioritizes initiatives, estimates resource requirements (trained staff and
                         funding) and defines a process improvement management structure;
                     •   ensure that every new software development effort in Customs adopts
                         processes that satisfy at least SW-CMM level 2 requirements; and
                     •   ensure that process improvement activities are initiated for all ongoing
                         essential software maintenance projects.


                         In its written comments on a draft of this report, Customs acknowledged
Agency Comments          the importance of software process improvement and maturity. Also, it
and Our Evaluation

                         Page 66                             GAO/AIMD-99-35 Customs Service Modernization
Chapter 8
Overall Conclusions, Recommendations,
and Agency Comments




agreed with GAO’s overall findings, including that Customs’ software
development processes have not attained SW-CMM level 2 maturity.

To address these weaknesses, Customs stated that it has taken the first
step toward implementing our recommendations by assigning
responsibility and authority for software process improvement as part of a
reorganization of its Office of Information and Technology, which
Customs stated will be implemented in early 1999. Customs further stated
that once the reorganization is implemented, a formal software process
improvement program will be established, and that this program will
include definition of an action plan, commitment of resources, and
specification of goals for achieving CMM levels 2 and 3. According to
Customs, these improvement activities are in their early stages. When they
are successfully implemented, they should address many of our
recommendations.

Customs also stated that because its legacy systems are aging and need to
be enhanced and replaced, software process improvement must occur in
parallel with continued software development investments. History has
shown that attempting to modernize without first instituting disciplined
software processes has been a characteristic of failed modernization
programs.1 Until it implements disciplined software processes (i.e., at least
level 2 process maturity), Customs cannot prudently manage major system
investments, such as ACE with an estimated life cycle cost exceeding
$1 billion.

Customs’ comments also included a request to meet with us to discuss
system-specific KPA practice strength and weakness determinations. We
met prior to requesting comments on a draft of this report and then again
on January 12, 1999, to discuss SEI’s SW-CMM requirements and the basis for
our determinations. We are prepared to continue assisting Customs as it
improves its software processes.

Appendix I provides the full text of Customs’ comments and our responses
to additional Customs comments not discussed above.




1
 Tax Systems Modernization: Management and Technical Weaknesses Must Be Corrected If
Modernization Is To Succeed (GAO/AIMD-95-156, July 26, 1995), Tax Systems Modernization: Actions
Underway But IRS Has Not Yet Corrected Management and Technical Weaknesses (GAO/AIMD-96-106,
June 7, 1996), and Air Traffic Control: Immature Software Acquisition Processes Increase FAA System
Acquisition Risks (GAO/AIMD-97-47, March 21, 1997).



Page 67                                       GAO/AIMD-99-35 Customs Service Modernization
Appendix I

Comments From the Department of the
Treasury

Note: GAO comments
supplementing those in the
report text appear at the
end of this appendix.




                             Page 68   GAO/AIMD-99-35 Customs Service Modernization
                 Appendix I
                 Comments From the Department of the
                 Treasury




See comment 1.




See comment 2.




See comment 3.




See comment 4.




                 Page 69                               GAO/AIMD-99-35 Customs Service Modernization
               Appendix I
               Comments From the Department of the
               Treasury




               The following are responses to additional comments in Customs’ letter
               dated December 16, 1998.


               1. The report has been modified to reflect Customs’ comment.
GAO Comments
               2. Customs provided us a copy of its new Systems Development Life Cycle
               (SDLC) document after it provided us comments on a draft of this report,
               and we have not yet evaluated it. To develop systems effectively, Customs
               will have to ensure that its SDLC addresses the software development KPAs
               discussed in this report and that it effectively implements the SDLC.

               3. The statement that “Customs has a history of performing poorly when
               developing software-intensive systems” has been removed from the report.
               Regarding the statement in the report referring to “Customs’ limited
               success in delivering promised system capabilities on time and within
               budget,” there is clear evidence that Customs has not been successful in
               developing ACE software on time. Specifically, the first release of ACE,
               which is Customs’ largest software development effort, was delivered over
               8 months late. Also, because Customs does not track actual ACE costs by
               release against original estimates, it does not know the cost performance
               history of ACE. With the respect to the three systems that Customs cites as
               having been successfully delivered, the first (the Year 2000 program) is
               still incomplete and thus its success cannot yet be assessed. Because we
               have not previously reviewed the other two (AES and Tinman), we cannot
               comment on either’s success in delivering promised capabilities on time
               and within budget. The report has been clarified to reflect these points.

               4. The report has been clarified to reflect Customs’ comment.




               Page 70                               GAO/AIMD-99-35 Customs Service Modernization
Appendix II

Major Contributors to This Report


                       Dr. Rona B. Stillman, Chief Scientist for Computers and
Accounting and           Telecommunications
Information            Jack L. Brock, Director, Governmentwide and Defense Information
Management Division      Systems
                       Madhav S. Panwar, Technical Assistant Director
                       Dr. Nabajyoti Barkakati, Technical Assistant Director
                       Prithviraj Mukherji, Assistant Director
                       Carl M. Urie, Assistant Director
                       Bernard R. Anderson, Senior Information Systems Analyst
                       Suzanne M. Burns, Senior Information Systems Analyst


                       Teresa F. Tucker, Senior Information Systems Analyst
Atlanta Field Office




(511115)               Page 71                          GAO/AIMD-99-35 Customs Service Modernization
Ordering Information

The first copy of each GAO report and testimony is free.
Additional copies are $2 each. Orders should be sent to the
following address, accompanied by a check or money order
made out to the Superintendent of Documents, when
necessary. VISA and MasterCard credit cards are accepted, also.
Orders for 100 or more copies to be mailed to a single address
are discounted 25 percent.

Orders by mail:

U.S. General Accounting Office
P.O. Box 37050
Washington, DC 20013

or visit:

Room 1100
700 4th St. NW (corner of 4th and G Sts. NW)
U.S. General Accounting Office
Washington, DC

Orders may also be placed by calling (202) 512-6000
or by using fax number (202) 512-6061, or TDD (202) 512-2537.

Each day, GAO issues a list of newly available reports and
testimony. To receive facsimile copies of the daily list or any
list from the past 30 days, please call (202) 512-6000 using a
touchtone phone. A recorded menu will provide information on
how to obtain these lists.

For information on how to access GAO reports on the INTERNET,
send an e-mail message with "info" in the body to:

info@www.gao.gov

or visit GAO’s World Wide Web Home Page at:

http://www.gao.gov




PRINTED ON    RECYCLED PAPER
United States                       Bulk Rate
General Accounting Office      Postage & Fees Paid
Washington, D.C. 20548-0001           GAO
                                 Permit No. G100
Official Business
Penalty for Private Use $300

Address Correction Requested