oversight

Air Traffic Control: Immature Software Acquisition Processes Increase FAA System Acquisition Risks

Published by the Government Accountability Office on 1997-03-21.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                 United States General Accounting Office

GAO              Report to the Chairman, Subcommittee
                 on Transportation, Committee on
                 Appropriations, House of
                 Representatives

March 1997
                 AIR TRAFFIC
                 CONTROL
                 Immature Software
                 Acquisition Processes
                 Increase FAA System
                 Acquisition Risks




GAO/AIMD-97-47
      United States
GAO   General Accounting Office
      Washington, D.C. 20548

      Accounting and Information
      Management Division

      B-271531

      March 21, 1997

      The Honorable Frank R. Wolf
      Chairman, Subcommittee on
        Transportation
      Committee on Appropriations
      House of Representatives

      Dear Mr. Chairman:

      This report responds to your request that we review the Federal Aviation Administration’s (FAA)
      air traffic control modernization software acquisition maturity and improvement activities. FAA
      plans to spend billions of dollars replacing its existing air traffic control systems, but has a
      history of performing poorly when acquiring these software-intensive systems. We found that
      FAA’s software acquisition processes are immature, and are making recommendations to the
      Secretary of Transportation for strengthening them.

      We are sending copies of this report to the Secretary of Transportation, the Director of the
      Office of Management and Budget, the Administrator of the Federal Aviation Administration,
      and other congressional committees. We will also make copies available to other interested
      parties upon request. If you have questions or wish to discuss the issues in this report, please
      contact me at (202) 512-6412. Major contributors to this report are listed in appendix II.

      Sincerely yours,




      Dr. Rona B. Stillman
      Chief Scientist for Computers
        and Telecommunications
Executive Summary


             The Federal Aviation Administration (FAA) is modernizing the air traffic
Purpose      control (ATC) systems upon which it will rely to ensure safe, orderly, and
             efficient air travel well into the 21st century. Since software is the most
             expensive and complex component of these systems, FAA must use defined
             and disciplined processes when it acquires software.

             Recognizing software’s growing importance and prevalence in ATC
             systems, the Chairman, Subcommittee on Transportation and Related
             Agencies, House Committee on Appropriations, asked GAO to determine
             (1) the maturity of FAA’s ATC modernization software acquisition processes,
             and (2) the steps/actions FAA has underway or planned to improve these
             processes, including any obstacles that may impede FAA’s progress.


             To accommodate forecasted growth in air traffic and replace aging
Background   equipment, FAA embarked on an ambitious ATC modernization program in
             1981. FAA estimates that it will spend about $20 billion to replace and
             modernize software-intensive ATC systems between 1982 and 2003. Our
             work over the years has chronicled many FAA failures in meeting ATC
             projects’ cost, schedule, and performance goals, largely because of
             software-related problems. As a result of these failures as well as the
             tremendous cost, complexity, and mission criticality of FAA’s ATC
             modernization program, we designated the program as a high-risk
             information technology initiative in our 1995 and 1997 report series on
             high-risk programs.1

             Software quality is governed largely by the quality of the processes
             involved in developing or acquiring, and maintaining it. Carnegie Mellon
             University’s Software Engineering Institute (SEI), recognized for its
             expertise in software processes, has developed models and methods that
             define and determine organizations’ software process maturity. Together,
             they provide a logical framework for baselining an organization’s current
             process capabilities (i.e., strengths and weaknesses) and providing a
             structured plan for incremental process improvement.

             Using SEI’s software acquisition capability maturity model (SA-CMM),2 SEI’s
             software capability evaluation method, and SA-CMM authors as consultants,

             1
              High-Risk Series: An Overview (GAO/HR-95-1, Feb. 1995); High-Risk Series: Information Management
             and Technology (GAO/HR-97-9, Feb. 1997).
             2
              We used a draft version of the model for our evaluation (version 00.03, dated April 1996). The first
             published version of the model was released on October 1996, after we performed our evaluation.
             According to the model’s authors, the published version differed only editorially from the draft we
             used.



             Page 2                                                          GAO/AIMD-97-47 Air Traffic Control
                   Executive Summary




                   GAO  staff trained at SEI evaluated FAA’s ATC modernization software
                   acquisition maturity in the seven key process areas (KPA) necessary to
                   attain a “repeatable” level of process capability, and one KPA associated
                   with the “defined” level of process maturity.3 Repeatability ensures that an
                   organization has the necessary process discipline in place to repeat earlier
                   successes on projects in similar domains. Repeatable processes are at the
                   second level on SEI’s five-level scale of process maturity. Organizations
                   that do not satisfy the requirements for the “repeatable” level are by
                   default judged to be at the “initial” level of maturity, meaning that their
                   processes are ad hoc, sometimes even chaotic, with few of the processes
                   defined and success dependent mainly on the heroic efforts of individuals.
                   The one KPA associated with the third level of process maturity, which is
                   called the “defined” level, is acquisition risk management. It was included
                   because many software experts consider it to be a very important process
                   area.

                   As part of its evaluation, GAO examined five ongoing ATC modernization
                   projects selected by FAA.4 These were the Automated Radar Terminal
                   System, Display System Replacement, National Airspace System
                   Infrastructure Management System, Voice Switching and Control System,
                   and the Weather and Radar Processor. (See chapter 1 of this report for
                   more detailed information on GAO’s evaluation scope and methodology.)


                   Because of the number and severity of FAA ATC modernization software
Results in Brief   acquisition process weaknesses, FAA did not fully satisfy any of the seven
                   KPAs necessary to achieve the “repeatable” level of process maturity. As a
                   result, its processes for acquiring software, the most costly and complex
                   component of ATC systems, are ad hoc, sometimes chaotic, and not
                   repeatable across projects. In addition, serious process weaknesses
                   prevented FAA from satisfying the one KPA specified under SEI’s “defined”
                   maturity level. While FAA showed process strengths, primarily in the
                   solicitation and evaluation (i.e., testing) KPAs,5 GAO found extensive
                   weaknesses in these and the remaining six KPAs (i.e., software acquisition

                   3
                    The seven KPAs relating to the repeatable level are software acquisition planning, solicitation,
                   requirements development and management, project office management, contract tracking and
                   oversight, evaluation, and transition and support.
                   4
                    GAO asked FAA to choose five projects that are: (1) major efforts with large software acquisition
                   components, (2) managed by different FAA product teams, (3) at different life cycle stages, and
                   (4) among FAA’s best managed.
                   5
                     According to the SA-CMM, solicitation is the process of ensuring that award is made to the contractor
                   most capable of satisfying the specified requirements, and evaluation is the process of determining
                   that acquired software products and services satisfy contract requirements prior to acceptance.



                   Page 3                                                          GAO/AIMD-97-47 Air Traffic Control
                         Executive Summary




                         planning, requirements development and management, project office
                         management, contract tracking and oversight, transition and support, and
                         acquisition risk management).6 Some of these weaknesses were systemic,
                         recurring in each of the KPAs. For example, no software project teams
                         measured or reported to management on the status of activities
                         performed, and management never verified that critical activities were
                         being done. These types of problems are some of the reasons for FAA’s
                         frequent failures to deliver promised ATC system capabilities on time and
                         within budget.

                         FAA has stated its commitment to increasing ATC modernization process
                         maturity. However, despite 4 years of activity in this area, FAA lacks an
                         effective management approach for improving software acquisition
                         processes. Currently, the Software Engineering Process Group (SEPG) is
                         responsible for process improvement; but the SEPG has neither
                         organizational nor budgetary authority over the product teams that acquire
                         software, and, therefore, cannot effectively implement or enforce process
                         change. Instead, it can only recommend and encourage change.
                         Additionally, FAA does not have an effective plan to correctly target and
                         prioritize improvements and measure improvement progress. In the
                         absence of this plan, it has initiated a “hodge podge” of software
                         acquisition improvement efforts without any analytical justification. As a
                         result, FAA’s process improvement activities have yet to produce more
                         repeatable, better defined, more disciplined software acquisition
                         processes.



Principal Findings

ATC Modernization        To attain a given SEI-defined maturity level, an organization must satisfy
Software Acquisition     the key practices for the KPAs associated with that level. FAA’s ATC
Processes Are Immature   modernization organization had too many weaknesses to satisfy any of the
                         “repeatable” KPAs (i.e., software acquisition planning, solicitation,
                         requirements development and management, project office management,

                         6
                           According to the SA-CMM, software acquisition planning is the process for ensuring that reasonable
                         planning for all elements of the software acquisition occur; requirements development and
                         management is the process for establishing an unambiguous and agreed upon set of software
                         requirements; project office management is the process for effective and efficient management of
                         project office activities; contract tracking and oversight is the process of ensuring that contractor
                         activities, products, and services satisfy contract requirements; transition and support is the process of
                         transferring acquired software products to the eventual support organization; and acquisition risk
                         management is the process of identifying software risks early and adjusting the acquisition strategy to
                         mitigate those risks.



                         Page 4                                                          GAO/AIMD-97-47 Air Traffic Control
                                    Executive Summary




                                    contract tracking and oversight, evaluation, and transition and support),
                                    nor does it satisfy the acquisition risk management KPA from the “defined”
                                    or third maturity level.

                                    For FAA to satisfy any of these eight KPAs, it must eliminate the key practice
                                    weaknesses identified in this report.7 Each practice that is performed
                                    effectively constitutes a strength, and each practice not performed or
                                    performed poorly constitutes a weakness. While FAA’s ATC modernization
                                    has some strengths, it has more weaknesses. Table 1 tallies these strengths
                                    on the five projects that GAO evaluated. In summary, of the total number of
                                    KPA practices rated, 38 percent constituted strengths, 50 percent were
                                    weaknesses, and 12 percent were observations. An observation indicates
                                    that the evidence was inconclusive and did not clearly support a
                                    determination of either strength or weakness.

Table 1: Collective Number of KPA
Strengths, Weaknesses, and                                                          Number of           Number of           Number of
Observations on the Five Projects   Key Process Area                                 strengths         weaknesses         observations
                                    Software acquisition planning                             16                   37                    7
                                    Solicitation                                              36                   28                  14
                                    Requirements development and                              17                   35                    6
                                    management
                                    Project office management                                 26                   35                    6
                                    Contract tracking and oversight                           26                   32                    6
                                    Evaluation                                                43                   21                    8
                                    Transition and support                                    27                   32                    8
                                    Acquisition risk management                               16                   46                    7
                                    Totals                                                   207                 266                   62

                                    Additionally, GAO found that while the five projects varied as to practice
                                    strengths, weaknesses, and observations under three of the five “common
                                    features” or practice groupings (commitment to perform, ability to
                                    perform, and activities performed), the projects were consistently weak in
                                    all practices under the remaining two groupings (measurement and
                                    analysis and verifying implementation). As a result, software project teams
                                    and FAA management consistently lack reliable information on project
                                    team performance.




                                    7
                                     SEI groups each of its KPA practices into one of five “common features” or practice categories. These
                                    are “commitment to perform, ability to perform, activities performed, measurement and analysis, and
                                    verifying implementation.”



                                    Page 5                                                         GAO/AIMD-97-47 Air Traffic Control
                           Executive Summary




FAA’s Approach for         To be effective, the FAA organization responsible for software acquisition
Improving ATC              process improvement must have (1) organizational and/or budgetary
Modernization Software     authority over the ATC modernization units acquiring the software; and
                           (2) an effective plan to guide improvement efforts and measure progress
Acquisition Processes Is   on each. The FAA organizational entity currently responsible for ATC
Not Effective              modernization software acquisition process improvement, the SEPG, has
                           neither. As a result, little progress has been made over the last 4 years in
                           instituting definition and discipline into ATC modernization software
                           acquisition processes.

                           The SEPG is a multilevel committee structure chaired by a member of FAA’s
                           Chief Information Officer’s (CIO) staff. The SEPG is directed by the Software
                           Engineering Executive Committee, which is chaired by the head of the ATC
                           modernization program. The SEPG has no authority to implement and
                           enforce process change. Consequently, it can only attempt to encourage
                           and persuade software acquirers to establish and follow defined and
                           disciplined software acquisition processes.

                           The SEPG and its predecessors have advocated and initiated a collection of
                           efforts intended to strengthen ATC modernization software-related
                           processes, including software acquisition processes. However, there is no
                           analytical basis for the focus, content, timing, and interrelationships of
                           these efforts. Specifically, the efforts (1) are not based on any assessment
                           of current software acquisition process strengths and weaknesses; and
                           (2) are not detailed in a formal plan that specifies measurable goals,
                           objectives, milestones, and needed resources, prioritizes efforts, fixes
                           responsibility and accountability, and defines metrics for measuring
                           progress. Instead, these efforts were undertaken with no sound analytical
                           basis and, rather than being part of a comprehensive plan, are discussed in
                           general terms without detail and specificity in briefing documents, minutes
                           of meetings, and working group recommendations. While the SEPG is now
                           taking steps to establish the analytical basis needed to formulate a
                           comprehensive software process improvement plan, that plan does not yet
                           exist, and no schedule has been established for completing it.


                           Given the importance and the magnitude of information technology at FAA,
Recommendations            GAO reiterates its earlier recommendation that a CIO management structure
                           similar to the department-level CIOs as prescribed in the Clinger-Cohen Act
                           of 1996 be established for FAA.8

                           8
                            Air Traffic Control: Complete and Enforced Architecture Needed for FAA Systems Modernization
                           (GAO/AIMD-97-30, February 3, 1997).



                           Page 6                                                     GAO/AIMD-97-47 Air Traffic Control
                           Executive Summary




                           To improve FAA’s software acquisition capability for its ATC modernization,
                           GAO recommends that the Secretary of Transportation direct the FAA
                           Administrator to:

                       •   assign responsibility for software acquisition process improvement to
                           FAA’s CIO;
                       •   provide FAA’s CIO with the authority needed to implement and enforce ATC
                           modernization software acquisition process improvement;
                       •   require the CIO to develop and implement a formal plan for ATC
                           modernization software acquisition process improvement that is based on
                           the software capability evaluation results contained in this report and
                           specifies measurable goals and time frames, prioritizes initiatives,
                           estimates resource requirements, and assigns roles and responsibilities;
                       •   allocate adequate resources to ensure that planned initiatives are
                           implemented and enforced; and
                       •   require that, before being approved, every ATC modernization acquisition
                           project have software acquisition processes that satisfy at least SA-CMM
                           level 2 requirements.


                           In its written comments on a draft of this report, the Department of
Agency Comments            Transportation recognized the importance of mature software acquisition
and GAO’s Evaluation       processes, agreed that FAA’s processes are insufficiently mature,
                           acknowledged that FAA process improvement activities have yet to
                           produce greater software acquisition process discipline, and reaffirmed
                           FAA’s commitment to improving its software acquisition capabilities using
                           the SA-CMM. However, the Department did not state what, if any, specific
                           action it would take on GAO’s recommendations. Additionally, it took the
                           positions that (1) the SA-CMM by itself is inadequate to evaluate ATC system
                           acquisition capabilities, is too new to use as an authoritative source of
                           guidance, and “may” have been misapplied by GAO, (2) the report does not
                           sufficiently recognize FAA’s process improvement organization and
                           progress nor the difficulties and time required to affect process
                           improvement change, (3) the SEPG, which is FAA’s designated agent for
                           software acquisition process change, is organized as the Department
                           “understands” other SEPGs to be organized, and (4) the report “leads the
                           reader to erroneously conclude that the five programs reviewed are in
                           trouble” relative to attainment of cost and schedule goals.

                           None of these positions are valid. First, the SA-CMM, like the SW-CMM
                           (another SEI software-specific capability maturity model), focuses on
                           software because it is widely recognized as the most difficult and costly



                           Page 7                                        GAO/AIMD-97-47 Air Traffic Control
Executive Summary




component of modern computer systems; the one most frequently
associated with late deliveries, cost overruns, and performance shortfalls;
and the one in greatest need of special management attention. Further,
while the SA-CMM is relatively new, the processes it requires are well
established, experience-based tenets of effective software acquisition that
are widely supported throughout industry and government. Moreover, GAO
applied the SA-CMM at FAA properly and with extraordinary diligence: Every
member of the evaluation team was trained at SEI; the team leader was
certified by SEI as a lead evaluator; and three SEI professionals, including
two authors of the SA-CMM, participated in the evaluation and concurred
with every practice determination (e.g., strength, weakness).

Second, FAA’s many software acquisition process improvement activities
were undertaken without assessing current software acquisition process
strengths and weaknesses, and were not part of a comprehensive plan for
process improvement. Therefore, FAA had no analytical basis for deciding
what improvement activities to initiate, or what priorities to assign them.
Further, although FAA began drafting a plan during the course of GAO’s
evaluation, it has no schedule for completing it. In describing FAA’s
progress to date in improving its processes, the report delineates a wide
array of FAA process improvement activities, but distinguishes these
activities from actual progress. In fact, after 4 years of activity, FAA could
not point to a single case in which it had instituted a more disciplined
software acquisition process. Since SEI published statistics show that the
median time to improve from software development CMM level 1 to level 2
is 26 months, and from level 2 to level 3 is 17 months, it is entirely
reasonable to expect FAA to be able to demonstrate some improvement in
its processes after 4 years.

Third, the issue is not whether FAA’s SEPG is organized as the Department
“understands” other SEPGs to be organized, but whether the SEPG, or any
FAA organizational entity responsible for implementing and enforcing
software acquisition process change, has the authority needed to
accomplish the task. Currently, no organizational entity in FAA has the
requisite authority.

Last, the report addresses the maturity of FAA’s software acquisition
processes and concludes that these processes are ad hoc and
undisciplined, reducing the probability that software-intense ATC
modernization projects will consistently perform as intended and be
delivered on schedule and within budget. The report does not address the
overall status of the projects covered by GAO’s review, and, therefore,



Page 8                                          GAO/AIMD-97-47 Air Traffic Control
Executive Summary




provides no basis for drawing conclusions about the projects’ overall cost
or schedule performance.

The Department’s comments and GAO’s evaluation of them are presented in
greater detail in chapter 11 of this report.




Page 9                                       GAO/AIMD-97-47 Air Traffic Control
Contents



Executive Summary                                                                                   2


Chapter 1                                                                                          16
                        Overview of ATC                                                            16
Introduction            The ATC Modernization Program Is Complex, Costly, and                      19
                          Historically Problematic
                        ATC Modernization Now Proceeding Under a New Acquisition                   20
                          Management System
                        FAA Organizations Responsible for ATC Systems Acquisition and              21
                          Maintenance
                        Objectives, Scope, and Methodology                                         22

Chapter 2                                                                                          29
                        FAA’s Software Acquisition Planning Process Is Not Effective               29
Software Acquisition    Conclusions                                                                37
Planning
Chapter 3                                                                                          39
                        Product Teams Performing Many Solicitation Practices                       39
Solicitation            Conclusions                                                                51

Chapter 4                                                                                          52
                        Requirements Development and Management Process Is Not                     52
Requirements              Effective
Development and         Conclusions                                                                62
Management
Chapter 5                                                                                          63
                        FAA’s Project Office Management Process Area Is Not Effective              63
Project Office          Conclusions                                                                72
Management
Chapter 6                                                                                          73
                        FAA Lacks an Effective Contract Tracking and Oversight Process             73
Contract Tracking and   Conclusions                                                                85
Oversight




                        Page 10                                     GAO/AIMD-97-47 Air Traffic Control
                       Contents




Chapter 7                                                                                         87
                       FAA Is Strong in Most but Not All Evaluation KPA Practices                 87
Evaluation             Conclusions                                                                99

Chapter 8                                                                                        101
                       Transition and Support Not Being Performed Effectively                    101
Transition and         Conclusions                                                               113
Support
Chapter 9                                                                                        115
                       ATC Modernization Software Risk Management Is Ineffective                 115
Acquisition Risk       Conclusions                                                               126
Management
Chapter 10                                                                                       127
                       FAA Organization Responsible for ATC Software Acquisition                 127
FAA Lacks an             Process Improvement Lacks Authority to Affect Change
Effective Approach     FAA Planning for Software Process Improvement Has Not Been                129
                         Effective
for Improving ATC      Improvement Initiatives Have Thus Far Not Instilled Software              130
Modernization            Process Discipline
Software Acquisition   Conclusions                                                               131
Processes
Chapter 11                                                                                       132
                       Recommendations                                                           132
Overall Conclusions    Agency Comments and Our Evaluation                                        133
and
Recommendations
Appendixes             Appendix I: Comments From the Department of Transportation                138
                       Appendix II: Major Contributors to This Report                            144

Tables                 Table 1: Collective Number of KPA Strengths, Weaknesses, and                5
                         Observations on the Five Projects
                       Table 1.1: SA-CMM KPAs Used to Assess FAA Software                         25
                         Acquisition Maturity
                       Table 2.1: Software Acquisition Planning Findings for ARTSIIIE             31
                       Table 2.2: Software Acquisition Planning Findings for DSR                  32




                       Page 11                                     GAO/AIMD-97-47 Air Traffic Control
Contents




Table 2.3: Software Acquisition Planning Findings for NIMS                 33
Table 2.4: Software Acquisition Planning Findings for VSCS                 35
Table 2.5: Software Acquisition Planning Findings for WARP                 36
Table 3.1: Solicitation Findings for ARTSIIIE                              42
Table 3.2: Solicitation Findings for DSR                                   44
Table 3.3: Solicitation Findings for NIMS                                  46
Table 3.4: Solicitation Findings for VSCS                                  48
Table 3.5: Solicitation Findings for WARP                                  50
Table 4.1: Requirements Development and Management Findings                55
  for ARTSIIIE
Table 4.2: Requirements Development and Management Findings                57
  for DSR
Table 4.3: Requirements Development and Management Findings                58
  for NIMS
Table 4.4: Requirements Development and Management Findings                59
  for VSCS
Table 4.5: Requirements Development and Management Findings                61
  for WARP
Table 5.1: Project Office Management Findings for ARTSIIIE                 66
Table 5.2: Project Office Management Findings for DSR                      67
Table 5.3: Project Office Management Findings for NIMS                     68
Table 5.4: Project Office Management Findings for VSCS                     70
Table 5.5: Project Office Management Findings for WARP                     71
Table 6.1: Contract Tracking and Oversight Findings for ARTSIIIE           76
Table 6.2: Contract Tracking and Oversight Findings for DSR                78
Table 6.3: Contract Tracking and Oversight Findings for NIMS               80
Table 6.4: Contract Tracking and Oversight Findings for VSCS               82
Table 6.5: Contract Tracking and Oversight Findings for WARP               84
Table 7.1: Evaluation Findings for ARTSIIIE                                90
Table 7.2: Evaluation Findings for DSR                                     92
Table 7.3: Evaluation Findings for NIMS                                    94
Table 7.4: Evaluation Findings for VSCS                                    96
Table 7.5: Evaluation Findings for WARP                                    98
Table 8.1: Transition and Support Findings for ARTSIIIE                   104
Table 8.2: Transition and Support Findings for DSR                        106
Table 8.3: Transition and Support Findings for NIMS                       108
Table 8.4: Transition and Support Findings for VSCS                       110
Table 8.5: Transition and Support Findings for WARP                       112
Table 9.1: Acquisition Risk Management Findings for ARTSIIIE              118
Table 9.2: Acquisition Risk Management Findings for DSR                   119
Table 9.3: Acquisition Risk Management Findings for NIMS                  121




Page 12                                     GAO/AIMD-97-47 Air Traffic Control
          Contents




          Table 9.4: Acquisition Risk Management Findings for VSCS                 123
          Table 9.5: Acquisition Risk Management Findings for WARP                 125

Figures   Figure 1.1: Summary of ATC Over the Continental United States             18
            and Oceans
          Figure 1.2: ARA Organization Chart                                        22
          Figure 1.3: SA-CMM Levels and Descriptions                                24
          Figure 2.1: Software Acquisition Planning                                 30
          Figure 3.1: Solicitation Summary                                          40
          Figure 4.1: Requirements Development and Management                       54
            Summary
          Figure 5.1: Project Office Management Summary                             64
          Figure 6.1: Contract Tracking and Oversight Summary                       74
          Figure 7.1: Evaluation Summary                                            88
          Figure 8.1: Transition and Support Summary                               102
          Figure 9.1: Acquisition Risk Management Summary                          116




          Page 13                                    GAO/AIMD-97-47 Air Traffic Control
Contents




Abbreviations

AAR        Office of Aviation Research
AAS        Advanced Automation System
ACT        William J. Hughes Technical Center
AIMD       Accounting and Information Management Division
AIT        Office of Information Technology
AND        Office of Communication, Navigation, and Surveillance
                 Systems
ARA        Associate Administrator for Research and Acquisitions
ARINC      Aeronautical Radio Incorporated
ARTS       Automated Radar Terminal System
ASD        Office of System Architecture and Investment Analysis
ASU        Office of Acquisitions
ATC        air traffic control
ATCSCC     Air Traffic Control System Command Center
ATS        Associate Administrator for Air Traffic Services
AUA        Office of Air Traffic Systems Development
CIO        Chief Information Officer
CMM        Capability Maturity Model
COTS       commercial, off-the-shelf
DSR        Display System Replacement
FAA        Federal Aviation Administration
GAO        General Accounting Office
IPT        Integrated Product Team
ISSS       Initial Sector Suite System
KPA        key process area
NAS        National Airspace System
NDI        non-development item
NIMS       NAS Infrastructure Management System
SA-CMM     Software Acquisition Capability Maturity Model
SE-CMM     Systems Engineering Capability Maturity Model
SCE        Software Capability Evaluation
SEEC       Software Engineering Executive Committee
SEI        Software Engineering Institute
SEPG       Software Engineering Process Group
SW-CMM     Capability Maturity Model for Software
TRACON     Terminal Radar Approach Control
VSCS       Voice Switching and Control System
WARP       Weather and Radar Processor


Page 14                                    GAO/AIMD-97-47 Air Traffic Control
Page 15   GAO/AIMD-97-47 Air Traffic Control
Chapter 1

Introduction


                      The Federal Aviation Administration’s (FAA) primary mission is to ensure
                      safe, orderly, and efficient air travel in the national airspace. FAA’s ability
                      to fulfill this mission depends on the adequacy and reliability of the
                      nation’s air traffic control (ATC) system, a vast network of computer
                      hardware, software, and communications equipment.1 The ATC system,
                      however, is being strained by aging equipment, much of which is 1960’s
                      technology, and growing air traffic. This growth should continue as the
                      number of passengers traveling on U.S. airlines is expected to increase by
                      38 percent between 1995 and 2003, from about 580 million to nearly
                      800 million.

                      To accommodate the forecasted growth in air traffic and to relieve the
                      problems of the aging ATC system, FAA embarked on an ambitious ATC
                      modernization program in 1981. FAA estimates that it will spend about
                      $20 billion to replace and modernize software-intensive ATC systems
                      between 1982 and 2003. Our work over the years has chronicled many FAA
                      failures in meeting ATC projects’ cost, schedule, and performance goals.2
                      As a result of these failures as well as the tremendous cost, complexity,
                      and mission criticality of FAA’s ATC modernization program, we designated
                      the program as a high-risk information technology initiative in our 1995
                      and 1997 report series on high-risk programs.3


                      Automated information processing and display, communication,
Overview of ATC       navigation, surveillance, and weather resources permit air traffic
                      controllers to view key information, such as aircraft location, aircraft flight
                      plans, and prevailing weather conditions, and to communicate with pilots.
                      These resources reside at, or are associated with, several ATC
                      facilities—air traffic control towers, terminal radar approach control
                      (TRACON) facilities, air route traffic control centers (en route centers),
                      flight service stations, and the Air Traffic Control System Command
                      Center (ATCSCC). These facilities’ ATC functions are described below.

                  •   Airport towers control aircraft on the ground and before landing and after
                      take-off when they are within about 5 nautical miles of the airport, and up
                      to 3,000 feet above the airport. Air traffic controllers rely on a combination

                      1
                       The ATC system is a major component of the National Airspace System (NAS).
                      2
                       Air Traffic Control: Status of FAA’s Modernization Program (GAO/RCED-95-175FS, May 26, 1995); Air
                      Traffic Control: Status of FAA’s Modernization Program (GAO/RCED-94-167FS, Apr. 15, 1994); Air
                      Traffic Control: Status of FAA’s Modernization Program (GAO/RCED-93-121FS, Apr. 16, 1993).
                      3
                       High-Risk Series: An Overview (GAO/HR-95-1, Feb. 1995); High-Risk Series: Information Management
                      and Technology (GAO/HR-97-9, Feb. 1997).



                      Page 16                                                     GAO/AIMD-97-47 Air Traffic Control
    Chapter 1
    Introduction




    of technology and visual surveillance to direct aircraft departures and
    approaches, maintain safe distances between aircraft, and communicate
    weather-related information, clearances, and other instructions to pilots
    and other personnel.
•   Approximately 180 TRACONs sequence and separate aircraft as they
    approach and leave busy airports, beginning about 5 nautical miles and
    ending about 50 nautical miles from the airport, and generally up to 10,000
    feet above the airport, where en route centers’ control begins.
•   Twenty en route centers control planes over the continental United States
    in transit and during approaches to some airports. Each en route center
    handles a different region of airspace, passing control from one to another
    as respective borders are reached until the aircraft reaches TRACON
    airspace. En route center controlled airspace usually extends above 18,000
    feet for commercial aircraft. En route centers also handle lower altitudes
    when dealing directly with a tower, or when agreed upon with a TRACON.
•   Two en route centers—Oakland and New York—also control aircraft over
    the ocean. Controlling aircraft over oceans is radically different from
    controlling aircraft over land because radar surveillance only extends 175
    to 225 miles offshore. Beyond the radars’ sight, controllers must rely on
    periodic radio communications through a third party—Aeronautical Radio
    Incorporated (ARINC), a private organization funded by the airlines and FAA
    to operate radio stations—to determine aircraft locations.
•   About 90 flight service stations provide pre-flight and in-flight services,
    such as flight plan filing and weather report updates, primarily for general
    aviation aircraft.

    See figure 1.1 for a visual summary of air traffic control over the
    continental United States and oceans.




    Page 17                                        GAO/AIMD-97-47 Air Traffic Control
                                         Chapter 1
                                         Introduction




Figure 1.1: Summary of ATC Over the Continental United States and Oceans




                                                            Oceanic
                                                            En Route
                                                             Center



                                                                                ARINC




                                       En Route
                                        Center
                        TRACON




                                                   Flight Service                       En Route
                                                       Station                           Center




              Airport Tower




                                                                               TRACON

            Departure Control
                                                                    Airport Tower
            Approach Control
            Local and Ground Control

            Continental
            United States




                                         Page 18                                                   GAO/AIMD-97-47 Air Traffic Control
                      Chapter 1
                      Introduction




                      FAA faced two problems in continuing to fulfill its mission to ensure safe,
The ATC               orderly, and efficient air travel in the national airspace. First, the ATC
Modernization         system of the late 1970s was a blend of several generations of automated
Program Is Complex,   and manual equipment, much of it labor-intensive and obsolete. Second,
                      air traffic was projected to increase dramatically as a result of airline
Costly, and           deregulations of the late 1970s. FAA recognized that it could increase ATC
Historically          operating efficiency by increasing automation. It also anticipated that
                      meeting the demand safely and efficiently would require improved and
Problematic           expanded services, additional facilities and equipment, improved work
                      force productivity, and the orderly replacement of aging equipment.
                      Accordingly, in December 1981, FAA initiated its plan to modernize,
                      automate, and consolidate its enormous ATC system infrastructure by the
                      year 2000. In doing so, it chose to acquire new ATC systems by contracting
                      for systems development services from vendors rather than building new
                      ATC systems in-house.


                      This ambitious modernization program includes the acquisition of new
                      surveillance, data processing, navigation, and communication equipment
                      in addition to new facilities and support equipment. Totaling over 200
                      separate projects, the modernization is estimated to cost over $34 billion
                      through the year 2003. Software-intensive ATC systems make up a large
                      portion of this total, accounting for 169 projects costing $20.7 billion. The
                      Congress will have provided FAA with approximately $14.7 billion of the
                      $20.7 billion through fiscal year 1997. Many of these projects, for example
                      the Display System Replacement and the Voice Switching and Control
                      System, each involve the acquisition of over a million lines of code.
                      Moreover, because the software must operate in a real-time environment
                      in which human life is at stake, it must be fault tolerant, meaning that it
                      must be able to monitor its own execution and recover from failures
                      without losing any data.

                      Over the past 15 years, FAA’s modernization projects have experienced
                      substantial cost overruns, lengthy schedule delays, and significant
                      performance shortfalls. To illustrate, the long-time centerpiece of this
                      modernization program—the Advanced Automation System (AAS)—was
                      restructured in 1994 after estimated costs tripled from $2.5 billion to
                      $7.6 billion and delays in putting significantly less-than-promised system
                      capabilities into operation were expected to run 8 years or more over
                      original estimates. Similarly, increases in costs for three other ATC projects4
                       have ranged from 51 to 511 percent, and schedule delays have averaged

                      4
                      The three projects and their respective percentage increase in unit costs are the Voice Switching and
                      Control System (511 percent), the Integrated Terminal Weather System (129 percent), and the Aviation
                      Weather Observing System (51 percent).



                      Page 19                                                      GAO/AIMD-97-47 Air Traffic Control
                    Chapter 1
                    Introduction




                    almost 4 years. For example, the per-unit cost estimate for the Voice
                    Switching and Control System increased 511 percent, and the first site
                    implementation was delayed 6 years from the original estimate.

                    AAS  and other ATC projects have also experienced shortfalls in
                    performance. For example, the critical Initial Sector Suite System
                    component of AAS, which was intended to replace controllers’
                    workstations at en route centers, faced so many technical problems that
                    its functionality was severely scaled back. In addition, difficulties in
                    developing the Air Route Surveillance Radar-4 software and integrating it
                    with other ATC systems delayed its implementation for years.


                    Because of FAA’s contention that many of its modernization problems were
ATC Modernization   rooted in the Federal Acquisition System, the Congress enacted legislation
Now Proceeding      in October 1995 that exempted FAA from most federal procurement and
Under a New         personnel laws and regulations and directed FAA to develop and implement
                    a new acquisition system that would address the unique needs of the
Acquisition         agency.5 At a minimum, the system was to provide for more timely and
Management System   cost-effective acquisitions. On April 1, 1996, in response to the Act, the FAA
                    Administrator began implementation of a new acquisition management
                    system.

                    The new system is intended to reduce the time and cost to field new
                    products and services by introducing a new investment management
                    system that spans the investments’ entire life cycles, a new procurement
                    system that provides flexibility in selecting and managing contractors, and
                    organizational learning and cultural reform that supports the new
                    investment management and procurement systems.

                    This high-level policy promulgated by the new acquisition management
                    system is intended to be supplemented by guidelines in three areas:
                    software/systems acquisition, facilities acquisition, and services
                    acquisition. These guidelines will be available to FAA staff via the Internet
                    and were scheduled to be online by October 1, 1996. As of February 1,
                    1997, these guidelines were still in draft form and not available to FAA staff.




                    5
                     Department of Transportation and Related Agencies Appropriations Act 1996, P.L. No. 104-50, sec.
                    348, 109 Stat. 436, 460 (1995).



                    Page 20                                                      GAO/AIMD-97-47 Air Traffic Control
                      Chapter 1
                      Introduction




                      Two major FAA organizations play key roles in the modernization of ATC
FAA Organizations     systems—the Office of the Associate Administrator for Research and
Responsible for ATC   Acquisitions (ARA) and the Office of the Associate Administrator for Air
Systems Acquisition   Traffic Services (ATS). Briefly, ARA is responsible for acquiring ATC systems,
                      while ATS is responsible for operating and maintaining ATC systems.
and Maintenance       Cross-functional integrated product teams (IPT) residing in ARA are
                      responsible for specific ATC system acquisition projects.

                      ARA manages ATC modernization research and development and acquisition
                      activities. According to the Associate Administrator for ARA, only about
                      one-half of the total ATC systems development budget is spent by ARA,
                      while the other one-half is spent by ATS implementing system
                      enhancements. Within ARA, two groups are responsible for acquiring
                      systems, while other groups handle cross-cutting management functions,
                      such as budget formulation and program evaluation. These two groups are
                      the Office of Air Traffic Systems Development (AUA) and the Office of
                      Communication, Navigation, and Surveillance Systems (AND).

                      Five IPTs reside in AUA and are organized by ATC business areas (i.e., en
                      route, terminal, weather and flight service, air traffic management,
                      oceanic), and five IPTs reside in AND and are organized by ATC functional
                      areas (i.e., infrastructure, communications, surveillance, Global
                      Positioning System/navigation, aircraft/avionics). IPTs are responsible for
                      research, development, and acquisition as well as for ensuring that new
                      equipment is delivered, installed, and working properly. For example, the
                      en route IPT comprises product teams for the Display Channel Complex
                      Rehost, the Display System Replacement, the Voice Switching and Control
                      System, and several other en route systems. Each IPT includes systems and
                      specialty engineers, logistics personnel, testing personnel, contract
                      personnel, and lawyers as well as representatives from the organizations
                      responsible for operating and maintaining the ATC equipment. The
                      organization chart below shows the structure of the ARA organization.




                      Page 21                                        GAO/AIMD-97-47 Air Traffic Control
                                                                Chapter 1
                                                                Introduction




Figure 1.2: ARA Organization Chart


                                                                                                 Associate
                                                                                               Administrator
                                                                                             for Research and
                                                                                               Acquisitions
                                                                                                   ARA


                                       Office of Air Traffic        Office of Aviation            Office of                                  Office of System      William J. Hughes
              Office of Acquisitions                                                                               Office of Information
                                             Systems                    Research             Comm., Navigation,                              Architecture and
                       ASU                                                                                              Technology                                   Tech Center
                                          Development                     AAR                 and Surveillance                             Investment Analysis
                                               AUA                                             Systems, AND                 AIT                    ASD                    ACT

                                                                      Chief Scientist         Integrated Product        Corporate                Architecture           Resource
                 Acquisition Policy
                                         Integrated Product         for Human Factors              Team for            Information               and System            Management
                  and Procedures
                                         Team for En Route               Division                Infrastructure      Management Div.             Engineering             Division
                     Division
                                              AUA-200                    AAR-100                   AND-100               AIT-100                  ASD-100               ACT-100
                     ASU-100
                                                                                              Integrated Product     Integrated Product        Evaluation and
                                                                         Research                  Team for                                     Configuration        ATC Engineering
                 Quality Assurance       Integrated Product                                                               Team for
                                                                          Division             Communications                                   Management           and Test Division
                      Division           Team for Terminal                                                          Information Systems
                                                                         AAR-200                   AND-300                                       ASD-200                 ACT-200
                     ASU-200                  AUA-300                                                                     AIT-200
                                         Integrated Product             Research and          Integrated Product     Integrated Product       NAS Programming        CNS Engineering
                     Contracts           Team for Weather                Acquisitions              Team for              Team for IT            and Financial
                      Division                                                                                                                                       and Test Division
                                         and Flight Service         International Division        Surveillance            Services              Management              ACT-300
                     ASU-300                                              AAR-300                  AND-400                 AIT-300                ASD-300
                                         Systems, AUA-400
                                                                     Airport and Aircraft     Integrated Product     Integrated Product      Investment Analysis        Facilities
                                         Integrated Product
                                                                     Safety, Research           Team for GPS/            Team for IT           and Operations          Management
                                         Team for Air Traffic
                                                                     and Development              Navigation             Acquisitions             Research               Division
                                            Management
                                                                     Division, AAR-400             AND-500                AIT-400                 ASD-400               ACT-400
                                              AUA-500
                                                                     Aviation Security        Integrated Product                                                    Aviation Simulation
                                         Integrated Product            Research and                Team for                                                             and Human
                                          Team for Oceanic             Development             Aircraft/Avionics                                                     Factors Division
                                              AUA-600                Division, AAR-500             AND-600                                                               ACT-500
                                                                                                                                                                    Airport Management
                                                                                                                                                                      and Emergency
                                                                                                                                                                    Operations Division
                                                                                                                                                                          ACT-600




                                                                The Chairman, Subcommittee on Transportation and Related Agencies,
Objectives, Scope,                                              House Committee on Appropriations, requested that we review FAA’s
and Methodology                                                 ability to acquire software for ATC systems. Our objectives were to
                                                                determine (1) the maturity of FAA’s ATC modernization software acquisition
                                                                processes; and (2) the steps/actions FAA has underway or planned to
                                                                improve these processes, and any obstacles that may impede FAA’s
                                                                improvement actions.

                                                                To determine FAA’s software acquisition process maturity, we applied the
                                                                Software Engineering Institute’s Software Acquisition Capability Maturity
                                                                Model (SA-CMM)6 and its Software Capability Evaluation (SCE) method. SEI’s
                                                                expertise in software process assessment is accepted throughout the
                                                                industry. Our evaluators were all SEI-trained software specialists. In

                                                                6
                                                                 We used a draft version of the model for our evaluation (version 00.03, dated April 1996). The first
                                                                published version of the model was released in October 1996, after we performed our evaluation.
                                                                According to the model’s authors, the published version differed only editorially from the draft we
                                                                used.



                                                                Page 22                                                                          GAO/AIMD-97-47 Air Traffic Control
Chapter 1
Introduction




addition, we employed SEI consultants, two of whom are authors of the
model, as advisors to ensure proper application of the model.

The SA-CMM ranks organizational maturity according to five levels (see
figure 1.3). Maturity levels 2 through 5 require the verifiable existence and
use of certain software acquisition processes, known as key process areas
(KPA). According to the SEI, an agency that has these acquisition processes
in place is in a much better position to successfully acquire software than
an organization that does not have these processes in place. We evaluated
FAA’s software acquisition processes against all level 2 KPAs and one level 3
KPA (see table 1.1). We included one level 3 KPA—acquisition risk
management—because it is considered by software experts to be a very
important process area.




Page 23                                        GAO/AIMD-97-47 Air Traffic Control
                                             Chapter 1
                                             Introduction




Figure 1.3: SA-CMM Levels and Descriptions


                                                                                            Level 5 - Optimizing
                                                                           Continuous process improvement is empowered by
                                                                           quantitative feedback from the process and from piloting
                                             Continuously
                                                                           innovative ideas and technologies.
                                             improving
                                             process


                                                                             Level 4 - Managed
                                                            Detailed measures of quality of the software acquistion
                                                            processes, products, and services are collected. The
                               Predictable                  software processes, products, and services are
                               process                      quantitatively understood and controlled.



                                                                Level 3 - Defined
                                             The acquisition organization's software acquisition
                                             process is documented, standardized, and established
                                             as the standard software acquisition process. All
                Standard                     projects use an approved, tailored version of the
                consistant                   organization's standard software acquisition process
                process                      for acquiring their software products and services.



                                                Level 2 - Repeatable
                               Basic project management processes are established
                               to track performance, cost, and schedule. The
  Disciplined                  necessary process discipline is in place to repeat
  process                      earlier successes on projects in similar domains.




                                  Level 1 - Initial
                The software acquisiton process is characterized as
                ad hoc, and occasionally even chaotic. Few processes
                are defined and success depends on individual
                effort.




                                             Page 24                                               GAO/AIMD-97-47 Air Traffic Control
                                      Chapter 1
                                      Introduction




Table 1.1: SA-CMM KPAs Used to
Assess FAA Software Acquisition       SA-CMM Level 2 Key
Maturity                              process areas                  Description
                                      Software acquisition planning Ensuring that reasonable planning for the software
                                                                    acquisition is conducted and that all elements of the
                                                                    project are included.
                                      Solicitation                   Ensuring that award is made to the contractor most
                                                                     capable of satisfying the specified requirements.
                                      Requirements development       Establishing a common and unambiguous definition of
                                      and management                 software acquisition requirements understood by the
                                                                     acquisition team, system user, and the contractor.
                                      Project office management      Managing the activities of the project office and
                                                                     supporting contractor(s) to ensure a timely, efficient, and
                                                                     effective software acquisition.
                                      Contract tracking and          Ensuring that the software activities under contract are
                                      oversight                      being performed in accordance with contract
                                                                     requirements, and that products and services will satisfy
                                                                     contract requirements.
                                      Evaluation                     Determining that the acquired software products and
                                                                     services satisfy contract requirements prior to
                                                                     acceptance and transition to support.
                                      Transition and support         Providing for the transition of the software products being
                                                                     acquired to their eventual support organization.
                                      CMM Level 3                    Description
                                      Key process area
                                      Acquisition risk management    Identifying risks as early as possible, adjusting acquisition
                                                                     strategy to mitigate those risks, and developing and
                                                                     implementing a risk management process as an integral
                                                                     part of the acquisition process.

                                      As established by the model, each KPA contains five common attributes
                                      that indicate whether the implementation and institutionalization of a KPA
                                      can be effective, repeatable, and lasting. The five common features are:

                                  •   Commitment to perform. Commitment to perform describes the actions
                                      that the organization must take to establish the process and ensure that it
                                      can endure. Commitment to perform typically involves establishing
                                      organizational policies and sponsorship.
                                  •   Ability to perform. Ability to perform describes the preconditions that
                                      must exist in the project or organization to implement the software
                                      acquisition process competently. Ability to perform typically involves
                                      resources, organizational structures, and training.
                                  •   Activities performed. Activities performed describes the roles and
                                      procedures necessary to implement a KPA. Activities performed typically
                                      involve establishing plans and procedures, performing the work, tracking
                                      it, and taking appropriate management actions.



                                      Page 25                                                GAO/AIMD-97-47 Air Traffic Control
    Chapter 1
    Introduction




•   Measurement and analysis. Measurement and analysis describes activities
    performed to measure the process and analyze the measurements.
    Measurement and analysis typically includes defining the measurements to
    be taken and the analyses to be conducted to determine the status and
    effectiveness of the activities performed.
•   Verifying implementation. Verifying implementation describes the steps to
    ensure that the activities are performed in compliance with the process
    that has been established. Verification typically encompasses reviews by
    management.

    In accordance with SEI’s SCE method, for each KPA in level 2 and the one KPA
    in level 3 (risk management), we evaluated institutional FAA policies and
    practices and compared project-specific guidance and practices against
    the five common attributes. This project-specific comparison can result in
    one of four possible outcomes: (1) project strength—an effective
    implementation of the key practice; (2) project weakness—ineffective
    implementation of a key practice or failure to implement a key practice;
    (3) project observation—key practice evaluated but evidence inconclusive
    and cannot be characterized as either strength or weakness; and (4) not
    rated—key practice not currently relevant to project, therefore not
    evaluated.

    We performed the project-specific evaluations on five ongoing ATC
    modernization projects, each of which is described below. We asked FAA to
    choose these projects using the following criteria: (1) the projects are
    major efforts with large software acquisition components; (2) the projects
    are managed by different integrated product teams, (3) the projects are in
    different stages of their life cycles, and (4) the projects are among FAA’s
    best-managed acquisitions. The projects that FAA selected for our
    evaluation are:

•   Automated Radar Terminal System (ARTS) IIIE: ARTS gathers information
    from surveillance sensors, processes it, and sends it to air traffic
    controllers in terminal radar approach control facilities and control towers
    at airports. A series of improvements to ARTS have provided increased
    processor capacity and the ability to support a greater number of
    controller displays. The ARTS IIIE improvements provide for more
    controller positions and surveillance sensor inputs at selected large
    facilities. ARTS IIIE is operational at New York, Chicago, and Dallas/Fort
    Worth with additional systems planned for Southern California and
    Denver. FAA estimates that the enhancement will cost $383.8 million to
    develop and deploy.



    Page 26                                       GAO/AIMD-97-47 Air Traffic Control
    Chapter 1
    Introduction




•   Display System Replacement (DSR): DSR is intended to replace air traffic
    controllers’ display-related systems in each of the en route centers. DSR
    consists of controller work stations connected via a local area network to
    three interfacing systems (Host Computer System, Enhanced Direct
    Access Radar Channel, and Weather and Radar Processor). FAA plans to
    deploy DSR to all 20 en route centers in the continental United States, as
    well as ATC facilities in Anchorage and potentially in Honolulu. FAA is now
    conducting system acceptance testing. FAA estimates that DSR will cost
    $1,055.3 million to develop and deploy.
•   National Airspace System (NAS) Infrastructure Management System (NIMS):
    NIMS is intended to provide the system infrastructure, including data
    architecture and network communications, to permit remote ATC system
    operational monitoring and maintenance. This program will provide a
    three-tiered architecture consisting of a national control center, 4 to 10
    operational control centers, and over 300 local work centers. NIMS is in the
    pre-solicitation phase, and FAA estimates that it will cost about
    $500 million to develop and deploy.
•   Voice Switching and Control System (VSCS): VSCS is intended to provide
    air-to-ground and ground-to-ground communications between en route
    centers and aircraft. The VSCS is to replace the aging ground-to-ground
    switching equipment and the air-to-ground circuits with a single
    integrated, computer-controlled, digital voice switching system. The
    development of VSCS is completed and all systems are operational. FAA
    estimates that VSCS will cost about $1.5 billion to develop and deploy.
•   Weather and Radar Processor (WARP): WARP is a next generation weather
    and radar processing and display system that is intended to permit
    consolidation of weather data from several sources into a single,
    integrated display for controllers. Currently, the weather information
    provided to controllers in the en route centers comes from long-range
    aircraft surveillance radars, which are not well-suited for this purpose.
    Next generation weather radars are to replace the surveillance radars as
    the source of weather information. WARP is to collect, process, and
    disseminate this and other weather data to controllers, traffic management
    specialists, pilots, and meteorologists. WARP is currently under
    development, and FAA estimates that it will cost $124.6 million to develop
    and deploy.

    To address our second objective (what steps/actions FAA has underway or
    planned to improve its software acquisition processes and what obstacles,
    if any, may impede FAA’s progress), we interviewed FAA’s Chief Scientist for
    Software Engineering and his staff to determine: (1) process
    improvements that are planned and underway; (2) the rationale for each



    Page 27                                       GAO/AIMD-97-47 Air Traffic Control
Chapter 1
Introduction




initiative; (3) the relative priority of each; (4) progress made on each
initiative; and (5) obstacles, if any, impeding progress. We also analyzed
process improvement plans, meeting minutes, and related documentation
to further address these areas. Finally, we interviewed representatives
from the ATC modernization product teams and the SEPG to obtain their
perspectives in assessing process improvement support, activities,
progress, and obstacles.

The Department of Transportation provided written comments on a draft
of this report. These comments are presented and evaluated in chapter 11,
and are reprinted in appendix I. We performed our work at FAA
headquarters offices in Washington, D.C. between March 1996 and
February 1997, in accordance with generally accepted government
auditing standards.




Page 28                                      GAO/AIMD-97-47 Air Traffic Control
Chapter 2

Software Acquisition Planning


                       The purpose of software acquisition planning is to ensure that reasonable
                       planning for the software acquisition is conducted and that all aspects of
                       the total software acquisition effort are included in these plans. According
                       to the SA-CMM, a repeatable software acquisition planning process, among
                       other things, includes (1) addressing software life cycle support in
                       acquisition plans, (2) preparing life cycle software cost estimates,
                       (3) having a written software acquisition policy, (4) measuring and
                       reporting on the status of software acquisition planning activities, and
                       (5) having guidance on software training and experience requirements for
                       project personnel.


                       All five projects have some ability and/or activity strengths in this KPA. For
FAA’s Software         example, every project addresses software life cycle support in planning
Acquisition Planning   documents and software life cycle cost estimates were prepared for four
Process Is Not         of the projects. However, we found many more process weaknesses than
                       strengths. For example, FAA has a systems acquisition policy, but the
Effective              policy does not specifically address or provide guidance on software
                       acquisition. Therefore, FAA management has not formally recognized the
                       importance and uniqueness of software acquisition issues in the system
                       acquisition process, and has not formally committed to managing software
                       acquisition in a disciplined manner. Also, the product teams do not
                       measure or report on the status of software acquisition planning activities.
                       As a result, management is not always aware of problems in project
                       performance, and cannot always take corrective action expeditiously.
                       Additionally, none of the five projects has specific guidance on software
                       training or experience requirements for project participation. As a result,
                       software training is ad hoc, and decisions about project personnel
                       assignments are subjective.

                       Figure 2.1 provides a comprehensive listing of the five projects’ strengths,
                       weaknesses, and observations for the software acquisition planning KPA.
                       The specific findings supporting the practice ratings cited in figure 2.1 are
                       in tables 2.1 through 2.5.




                       Page 29                                        GAO/AIMD-97-47 Air Traffic Control
                                                       Chapter 2
                                                       Software Acquisition Planning




Figure 2.1: Software Acquisition Planning


                     Key Practice                                                  ARTSIIIE             DSR                NIMS               VSCS           WARP


   Commitment 1      The acquisition organization has a written policy for         Weakness          Weakness            Weakness           Weakness       Weakness
                     planning the software acquisition.


                     Personnel are assigned the responsibility for
      Ability 1                                                                    Weakness           Strength           Strength           Strength       Observation
                     performing software acquisition planning.


      Ability 2      The acquisition organization has experienced                 Observation       Observation         Observation       Observation      Observation
                     software acquisition management personnel.


      Ability 3      Software acquisition management personnel are                 Weakness          Weakness           Weakness            Weakness       Weakness
                     experienced in the domain of the project.


      Activity 1     Software acquisition planning personnel are                   Weakness           Strength           Strength           Strength       Weakness
                     involved in system acquisition planning.

                     The project's software acquisition planning is
      Activity 2     documented and the planning documentation is                  Weakness          Weakness           Observation         Weakness       Weakness
                     maintained over the life of the project.


      Activity 3     The software acquisition strategy for the project is          Weakness          Weakness            Strength           Weakness       Weakness
                     developed.

                     Software acquisition planning includes provisions
      Activity 4     for ensuring that the life cycle support of the               Strength           Strength           Strength           Strength        Strength
                     system is included in planning documentation.


      Activity 5     A life cycle cost estimate for the software activity is       Weakness           Strength           Strength            Strength       Strength
                     prepared and independently verified.

                     Measurements are made and used to determine the
   Measurement 1     status of the software acquisition planning                   Weakness          Weakness           Weakness            Weakness       Weakness
                     activities.

                     The activities for software acquisition planning are
    Verification 1   reviewed by acquisition organization management               Weakness          Weakness           Weakness            Weakness       Weakness
                     on a periodic basis.

                     The activities for software acquisition planning are
    Verification 2   reviewed by the project manager on both a periodic            Weakness          Weakness           Weakness            Weakness       Weakness
                     and event-driven basis.


                                                                 = Weakness      Key practice not implemented
                                                                 = Strength      Key practice effectively implemented
                                                                 = Observation Key practice evaluated but evidence inconclusive. Cannot characterize as either strength or
                                                                               weakness
                                                                 = Not rated     Key practice not currently relevant to project, therefore not evaluated




                                                       Page 30                                                                GAO/AIMD-97-47 Air Traffic Control
                                             Chapter 2
                                             Software Acquisition Planning




Table 2.1: Software Acquisition Planning Findings for ARTSIIIE
                                            Automated Radar Terminal System IIIE
                     Key Practice                                    Finding                                         Rating
Commitment 1         The acquisition organization has a written      The system acquisition policy does not          Weakness
                     policy for planning the software acquisition.   adequately address software, e.g., it does
                                                                     not address items that should be included
                                                                     in software planning such as contract
                                                                     tracking and oversight, requirements
                                                                     development, evaluation, and risk
                                                                     management.
Ability 1            Personnel are assigned the responsibility     No personnel are assigned the                     Weakness
                     for performing software acquisition planning. responsibility for software acquisition
                                                                   planning.
Ability 2            The acquisition organization has                The acquisition organization has no             Observation
                     experienced software acquisition                guidance regarding training or experience
                     management personnel.                           requirements for project participation.
Ability 3            Software acquisition management            There are no guidelines that define domain           Weakness
                     personnel are experienced in the domain of knowledge or experience.
                     the project.
Activity 1           Software acquisition planning personnel are No one on the product team is specifically          Weakness
                     involved in system acquisition planning.    assigned responsibility for software
                                                                 acquisition.
Activity 2           The project’s software acquisition planning There is no documented software                     Weakness
                     is documented and the planning               acquisition plan.
                     documentation is maintained over the life of
                     the project.
Activity 3           The software acquisition strategy for the       There is no software acquisition strategy.      Weakness
                     project is developed.
Activity 4           Software acquisition planning includes          The product team ensures that life cycle        Strength
                     provisions for ensuring that the life cycle     support is included in planning
                     support of the system is included in            documentation.
                     planning documentation.
Activity 5           A life cycle cost estimate for the software     The life cycle cost estimate was prepared       Weakness
                     activity is prepared and independently          but not independently verified.
                     verified.
Measurement 1        Measurements are made and used to               No internal process measurements are            Weakness
                     determine the status of the software            taken and used to determine the status of
                     acquisition planning activities.                activities for any of the key process areas.
Verification 1       The activities for software acquisition         While the Integrated Product Team leader     Weakness
                     planning are reviewed by acquisition            reviews the status of the contract and the
                     organization management on a periodic           contractor’s cost and schedule, he does not
                     basis.                                          review the status of the activities that are
                                                                     required to be performed for any of the key
                                                                     process areas.
Verification 2       The activities for software acquisition         While the product team leader reviews the       Weakness
                     planning are reviewed by the project            status of the contract and the contractor’s
                     manager on both a periodic and                  cost and schedule, he does not review the
                     event-driven basis.                             status of the activities that are required to
                                                                     be performed for any of the key process
                                                                     areas.


                                             Page 31                                                  GAO/AIMD-97-47 Air Traffic Control
                                            Chapter 2
                                            Software Acquisition Planning




Table 2.2: Software Acquisition Planning Findings for DSR
                                                Display System Replacement
                    Key Practice                                    Finding                                         Rating
Commitment 1        The acquisition organization has a written      The system acquisition policy does not          Weakness
                    policy for planning the software acquisition.   adequately address software, e.g., it does
                                                                    not address items that should be included
                                                                    in software planning such as contract
                                                                    tracking and oversight, requirements
                                                                    development, evaluation, and risk
                                                                    management.
Ability 1           Personnel are assigned the responsibility     Personnel are assigned the responsibility     Strength
                    for performing software acquisition planning. for performing software acquisition planning.
Ability 2           The acquisition organization has                The acquisition organization has no             Observation
                    experienced software acquisition                guidance regarding training or experience
                    management personnel.                           requirements for project participation.
Ability 3           Software acquisition management            There are no guidelines that define domain           Weakness
                    personnel are experienced in the domain of knowledge or experience.
                    the project.
Activity 1          Software acquisition planning personnel are Software acquisition personnel are involved Strength
                    involved in system acquisition planning.    in system acquisition planning.
Activity 2          The project’s software acquisition planning There is no software acquisition planning           Weakness
                    is documented and the planning               documentation.
                    documentation is maintained over the life of
                    the project.
Activity 3          The software acquisition strategy for the       Officials stated that the software acquisition Weakness
                    project is developed.                           strategy is developed, however, the
                                                                    documents provided did not address such
                                                                    things as objectives, technologies, and
                                                                    schedule.
Activity 4          Software acquisition planning includes          Software acquisition planning includes life     Strength
                    provisions for ensuring that the life cycle     cycle support planning.
                    support of the system is included in
                    planning documentation.
Activity 5          A life cycle cost estimate for the software     A life cycle cost estimate is prepared and      Strength
                    activity is prepared and independently          independently verified.
                    verified.
Measurement 1       Measurements are made and used to               No internal process measurements are            Weakness
                    determine the status of the software            taken and used to determine the status of
                    acquisition planning activities.                activities for any of the key process areas.
Verification 1      The activities for software acquisition         While the Integrated Product Team leader     Weakness
                    planning are reviewed by acquisition            reviews the status of the contract and the
                    organization management on a periodic           contractor’s cost and schedule, he does not
                    basis.                                          review the status of the activities that are
                                                                    required to be performed for any of the key
                                                                    process areas.
Verification 2      The activities for software acquisition         While the product team leader reviews the       Weakness
                    planning are reviewed by the project            status of the contract and the contractor’s
                    manager on both a periodic and                  cost and schedule, he does not review the
                    event-driven basis.                             status of the activities that are required to
                                                                    be performed for any of the key process
                                                                    areas.

                                            Page 32                                                  GAO/AIMD-97-47 Air Traffic Control
                                            Chapter 2
                                            Software Acquisition Planning




Table 2.3: Software Acquisition Planning Findings for NIMS
                                           NAS Infrastructure Management System
                    Key Practice                                    Finding                                       Rating
Commitment 1        The acquisition organization has a written      The system acquisition policy does not        Weakness
                    policy for planning the software acquisition.   adequately address software, e.g., it does
                                                                    not address items that should be included
                                                                    in software planning such as contract
                                                                    tracking and oversight, requirements
                                                                    development, evaluation, and risk
                                                                    management.
Ability 1           Personnel are assigned the responsibility     The team members are assigned the               Strength
                    for performing software acquisition planning. responsibility for software acquisition
                                                                  planning.
Ability 2           The acquisition organization has                The acquisition organization has no           Observation
                    experienced software acquisition                guidance regarding training or experience
                    management personnel.                           requirements for project participation.
Ability 3           Software acquisition management            There are no guidelines that define domain         Weakness
                    personnel are experienced in the domain of knowledge or experience.
                    the project.
Activity 1          Software acquisition planning personnel are The team members for software acquisition         Strength
                    involved in system acquisition planning.    are assigned collective responsibility and
                                                                are actively involved in system acquisition
                                                                planning.
Activity 2          The project’s software acquisition planning     At this early stage in the program, the       Observation
                    is documented and the planning                  software acquisition planning
                    documentation is maintained over the life of    documentation is being written but is not
                    the project.                                    complete.
Activity 3          The software acquisition strategy for the       Officials stated that the software acquisition Strength
                    project is developed.                           strategy will be contained within the
                                                                    acquisition plan.
Activity 4          Software acquisition planning includes          Software acquisition planning includes        Strength
                    provisions for ensuring that the life cycle     provisions for ensuring that life cycle
                    support of the system is included in            support is included in planning
                    planning documentation.                         documentation.
Activity 5          A life cycle cost estimate for the software     A life cycle cost estimate for software has   Strength
                    activity is prepared and independently          been prepared and independently verified.
                    verified.
Measurement 1       Measurements are made and used to               No internal process measurements are            Weakness
                    determine the status of the software            taken to determine the status of activities for
                    acquisition planning activities.                any of the key process areas.
Verification 1      The activities for software acquisition         While the Integrated Product Team leader     Weakness
                    planning are reviewed by acquisition            reviews the status of the contract and the
                    organization management on a periodic           contractor’s cost and schedule, he does not
                    basis.                                          review the status of the activities that are
                                                                    required to be performed for any of the key
                                                                    process areas.
                                                                                                                              (continued)




                                            Page 33                                                GAO/AIMD-97-47 Air Traffic Control
                                        Chapter 2
                                        Software Acquisition Planning




                                        NAS Infrastructure Management System
                 Key Practice                                 Finding                                         Rating
Verification 2   The activities for software acquisition      While the product team leader reviews the       Weakness
                 planning are reviewed by the project         status of the contract and the contractor’s
                 manager on both a periodic and               cost and schedule, he does not review the
                 event-driven basis.                          status of the activities that are required to
                                                              be performed for any of the key process
                                                              areas.




                                        Page 34                                                GAO/AIMD-97-47 Air Traffic Control
                                             Chapter 2
                                             Software Acquisition Planning




Table 2.4: Software Acquisition Planning Findings for VSCS
                                             Voice Switching and Control System
                     Key Practice                                    Finding                                         Rating
Commitment 1         The acquisition organization has a written      The system acquisition policy does not          Weakness
                     policy for planning the software acquisition.   adequately address software, e.g., it does
                                                                     not address items that should be included
                                                                     in software planning such as contract
                                                                     tracking and oversight, requirements
                                                                     development, evaluation, and risk
                                                                     management.
Ability 1            Personnel are assigned the responsibility     Personnel are assigned the responsibility     Strength
                     for performing software acquisition planning. for performing software acquisition planning.
Ability 2            The acquisition organization has                The acquisition organization has no             Observation
                     experienced software acquisition                guidance regarding training or experience
                     management personnel.                           requirements for project participation.
Ability 3            Software acquisition management            There are no guidelines that define domain           Weakness
                     personnel are experienced in the domain of knowledge or experience.
                     the project.
Activity 1           Software acquisition planning personnel are Software acquisition personnel are involved Strength
                     involved in system acquisition planning.    in system acquisition planning.
Activity 2           The project’s software acquisition planning The project’s software acquisition planning         Weakness
                     is documented and the planning               is not documented.
                     documentation is maintained over the life of
                     the project.
Activity 3           The software acquisition strategy for the       No software acquisition strategy exists. The    Weakness
                     project is developed.                           system acquisition strategy does not
                                                                     address software.
Activity 4           Software acquisition planning includes          The life cycle support of the system is         Strength
                     provisions for ensuring that the life cycle     included in the acquisition planning
                     support of the system is included in            documentation.
                     planning documentation.
Activity 5           A life cycle cost estimate for the software     The life cycle cost estimate has been           Strength
                     activity is prepared and independently          prepared and independently verified.
                     verified.
Measurement 1        Measurements are made and used to               No internal process measurements are            Weakness
                     determine the status of the software            taken or used to determine the status of
                     acquisition planning activities.                activities for any of the key process areas.
Verification 1       The activities for software acquisition         While the Integrated Product Team leader     Weakness
                     planning are reviewed by acquisition            reviews the status of the contract and the
                     organization management on a periodic           contractor’s cost and schedule, he does not
                     basis.                                          review the status of the activities that are
                                                                     required to be performed for any of the key
                                                                     process areas.
Verification 2       The activities for software acquisition         While the product team leader reviews the       Weakness
                     planning are reviewed by the project            status of the contract and the contractor’s
                     manager on both a periodic and                  cost and schedule, he does not review the
                     event-driven basis.                             status of the activities that are required to
                                                                     be performed for any of the key process
                                                                     areas.



                                             Page 35                                                  GAO/AIMD-97-47 Air Traffic Control
                                             Chapter 2
                                             Software Acquisition Planning




Table 2.5: Software Acquisition Planning Findings for WARP
                                                Weather and Radar Processor
                     Key Practice                                    Finding                                         Rating
Commitment 1         The acquisition organization has a written      The system acquisition policy does not          Weakness
                     policy for planning the software acquisition.   adequately address software, e.g., it does
                                                                     not address items that should be included
                                                                     in software planning such as contract
                                                                     tracking and oversight, requirements
                                                                     development, evaluation, and risk
                                                                     management.
Ability 1            Personnel are assigned the responsibility     Although the product team stated that they        Observation
                     for performing software acquisition planning. are assigned collective responsibility for
                                                                   systems acquisition, they could not provide
                                                                   documentation to show a specific
                                                                   assignment for software acquisition.
Ability 2            The acquisition organization has                The acquisition organization has no             Observation
                     experienced software acquisition                guidance regarding training or experience
                     management personnel.                           requirements for project participation.
Ability 3            Software acquisition management            There are no guidelines that define domain           Weakness
                     personnel are experienced in the domain of knowledge or experience.
                     the project.
Activity 1           Software acquisition planning personnel are Although the product team is responsible            Weakness
                     involved in system acquisition planning.    for systems acquisition, there is no one
                                                                 specifically assigned for software nor does
                                                                 any document expressly state that software
                                                                 is part of systems acquisition.
Activity 2           The project’s software acquisition planning There is no software acquisition plan.              Weakness
                     is documented and the planning
                     documentation is maintained over the life of
                     the project.
Activity 3           The software acquisition strategy for the       There is no software acquisition strategy for Weakness
                     project is developed.                           the project. The system acquisition strategy
                                                                     covers only software enhancements.
Activity 4           Software acquisition planning includes          The acquisition plan includes provisions for    Strength
                     provisions for ensuring that the life cycle     ensuring that life cycle support is included.
                     support of the system is included in
                     planning documentation.
Activity 5           A life cycle cost estimate for the software     The life cycle cost estimate is prepared and Strength
                     activity is prepared and independently          independently verified.
                     verified.
Measurement 1        Measurements are made and used to               No internal process measurements are            Weakness
                     determine the status of the software            taken and used to determine the status of
                     acquisition planning activities.                activities for any of the key process areas.
Verification 1       The activities for software acquisition         While the Integrated Product Team leader     Weakness
                     planning are reviewed by acquisition            reviews the status of the contract and the
                     organization management on a periodic           contractor’s cost and schedule, he does not
                     basis.                                          review the status of the activities that are
                                                                     required to be performed for any of the key
                                                                     process areas.
                                                                                                                                (continued)


                                             Page 36                                                 GAO/AIMD-97-47 Air Traffic Control
                                        Chapter 2
                                        Software Acquisition Planning




                                              Weather and Radar Processor
                 Key Practice                                 Finding                                         Rating
Verification 2   The activities for software acquisition      While the product team leader reviews the       Weakness
                 planning are reviewed by the project         status of the contract and the contractor’s
                 manager on both a periodic and               cost and schedule, he does not review the
                 event-driven basis.                          status of the activities that are required to
                                                              be performed for any of the key process
                                                              areas.


                                        Effective planning is the cornerstone of successful software acquisition.
Conclusions                             While FAA showed some strengths in this KPA, its many weaknesses render
                                        the software acquisition planning capability ad hoc and chaotic. Therefore,
                                        it is unlikely that projects are effectively measuring and monitoring
                                        software acquisition progress and taking corrective actions expeditiously.




                                        Page 37                                                GAO/AIMD-97-47 Air Traffic Control
Chapter 2
Software Acquisition Planning




Page 38                         GAO/AIMD-97-47 Air Traffic Control
Chapter 3

Solicitation


                         The purpose of solicitation is to prepare a request for proposal that
                         delineates a project’s software-related requirements, and select a
                         contractor that can most cost-effectively satisfy these requirements, while
                         complying with relevant solicitation laws and regulations. According to
                         the SA-CMM, specific requirements for a repeatable solicitation process
                         include, among other things, (1) having and following a solicitation plan,
                         (2) assigning responsibility and ensuring sufficient resources for
                         coordinating and conducting solicitation activities, (3) preparing and
                         reviewing cost and schedule estimates for the software products and
                         services being acquired, and (4) periodically measuring solicitation work
                         completed and effort and funds expended, comparing these measures to
                         plans, and reporting the results to management.


                         All five projects have strengths in many of the practices required by this
Product Teams            KPA. For example, most projects have written solicitation plans, assign
Performing Many          responsibility for coordinating and conducting the solicitation activities,
Solicitation Practices   and prepare and review contract-related software cost and schedule
                         estimates.

                         However, the projects are weak in several areas. For example, even
                         though most projects had a written solicitation plan, not all projects
                         followed their plans. Also, none of the projects adequately identified the
                         resources needed to conduct solicitation activities. While FAA personnel
                         stated that they had adequate solicitation resources, they provided no
                         evidence of either a mechanism for identifying required resources or for
                         ensuring that required resources are provided. These weaknesses increase
                         the risk of FAA not adequately evaluating the offerors’ proposals, and
                         making a suboptimal selection. Additionally, none of the five measured or
                         reported on the status of product team solicitation activities. As a result,
                         management cannot identify solicitation problems early and resolve them
                         expeditiously.

                         Figure 3.1 provides a comprehensive listing of the five projects’ strengths,
                         weaknesses, and observations for the solicitation KPA. The specific
                         findings supporting the practice ratings cited in figure 3.1 are in tables 3.1
                         through 3.5.




                         Page 39                                         GAO/AIMD-97-47 Air Traffic Control
                                                     Chapter 3
                                                     Solicitation




Figure 3.1: Solicitation Summary


                   Key Practice                                              ARTSIIIE         DSR          NIMS          VSCS          WARP

                   The acquisition organization has a written policy for
   Commitment 1                                                               Strength      Strength      Strength      Strength      Strength
                   the conduct of the solicitation.


   Commitment 2    Responsibility for the software portion of the            Weakness      Observation    Strength     Weakness       Strength
                   solicitation is designated.

                   A selection official has been designated to be
   Commitment 3    responsible for the selection process and the              Strength      Not rated     Strength      Strength      Not rated
                   decision.


      Ability 1    A group that is responsible for coordinating and           Strength      Strength      Strength      Not rated     Strength
                   conducting the solicitation activities exists.


      Ability 2    Adequate resources are provided for the solicitation      Weakness      Weakness      Weakness      Weakness      Weakness
                   activities.


      Ability 3    Individuals performing the solicitation activities have   Observation   Observation   Observation   Observation   Observation
                   experience or receive training.


     Activity 1    The project team documents its plans for solicitation      Strength      Not rated     Strength     Observation    Strength
                   activities.


                   The project's solicitation activities are performed in    Observation    Not rated     Strength     Weakness       Strength
     Activity 2
                   accordance with its plans.


     Activity 3    The project team documents its plans for proposal         Weakness       Strength      Strength     Observation    Strength
                   evaluation activities.


     Activity 4    The project team's proposal evaluation activities are     Weakness       Not rated     Strength     Weakness       Strength
                   performed in accordance with its plans.


     Activity 5    A cost estimate and schedule for the software              Strength      Strength      Strength      Strength      Strength
                   activity being sought are prepared.

                   The software cost estimate and schedule are
     Activity 6    independently reviewed for comprehensiveness and           Strength     Observation    Strength      Strength      Strength
                   realism.
                   The groups supporting the solicitation (e.g., end
                   user, systems engineering, support organization, and      Weakness       Not rated     Strength                   Observation
     Activity 7    application domain experts) receive orientation on                                                  Weakness
                   the solicitation's objectives and procedures.
                   The project team and the offeror review the project's
     Activity 8    software requirements during negotiations to ensure       Observation    Strength      Not rated    Observation    Strength
                   mutual understanding.

                   Measurements are made and used to determine the           Weakness       Weakness     Weakness      Weakness      Weakness
   Measurement 1
                   status of the solicitation activities.




                                                     Page 40                                                  GAO/AIMD-97-47 Air Traffic Control
                                                  Chapter 3
                                                  Solicitation




                 Key Practice                                                 ARTSIIIE             DSR                NIMS               VSCS          WARP
                 The activities for solicitation are reviewed by the
Verification 1   designated selection official or acquisition                 Weakness           Weakness           Weakness           Weakness       Weakness
                 organization management on a periodic basis.

                 The activities for solicitation are reviewed by the
Verification 2   project manager on both a periodic and event-driven          Weakness          Weakness            Weakness           Weakness       Weakness
                 basis.




                                                            = Weakness      Key practice not implemented
                                                            = Strength      Key practice effectively implemented
                                                            = Observation Key practice evaluated but evidence inconclusive. Cannot characterize as either strength or
                                                                          weakness

                                                            = Not rated     Key practice not currently relevant to project, therefore not evaluated




                                                  Page 41                                                                GAO/AIMD-97-47 Air Traffic Control
                                             Chapter 3
                                             Solicitation




Table 3.1: Solicitation Findings for ARTSIIIE
                                                Automated Radar Terminal System IIIE
                      Key Practice                                   Finding                                        Rating
Commitment 1          The acquisition organization has a written     FAA Order 1810.1F is the written policy.       Strength
                      policy for the conduct of the solicitation.
Commitment 2          Responsibility for the software portion of the Officials gave conflicting answers as to who Weakness
                      solicitation is designated.                    is responsible for the software portion of the
                                                                     solicitation, and could not provide a
                                                                     document that formally designates
                                                                     responsibility.
Commitment 3          A selection official has been designated to    The Administrator was the selection official   Strength
                      be responsible for the selection process       for the sole-source contract.
                      and the decision.
Ability 1             A group that is responsible for coordinating   A group (matrix team) is responsible for     Strength
                      and conducting the solicitation activities     coordinating and conducting the solicitation
                      exists.                                        activities.
Ability 2             Adequate resources are provided for the        No mechanism exists for identifying          Weakness
                      solicitation activities.                       resources required and for ensuring that the
                                                                     needed resources are provided to the
                                                                     project.
Ability 3             Individuals performing the solicitation        The acquisition organization has no            Observation
                      activities have experience or receive          guidance regarding training or experience
                      training.                                      requirements for project participation.
Activity 1            The project team documents its plans for       The team documents its plans for               Strength
                      solicitation activities.                       solicitation activities.
Activity 2            The project’s solicitation activities are      While officials stated that solicitation       Observation
                      performed in accordance with its plans.        activities are performed in accordance with
                                                                     its plans, they could not provide
                                                                     documentation to support this.
Activity 3            The project team documents its plans for       The team does not document its plans for       Weakness
                      proposal evaluation activities.                proposal evaluation activities.
Activity 4            The project team’s proposal evaluation         No evaluation plan exists, therefore, the      Weakness
                      activities are performed in accordance with    team could not perform in accordance with
                      its plans.                                     its plan.
Activity 5            A cost estimate and schedule for the         A cost estimate and schedule were                Strength
                      software activity being sought are prepared. prepared.
Activity 6            The software cost estimate and schedule        The cost estimate and schedule were            Strength
                      are independently reviewed for                 independently reviewed.
                      comprehensiveness and realism.
Activity 7            The groups supporting the solicitation (e.g., No orientation briefing occurred.               Weakness
                      end user, systems engineering, support
                      organization, and application domain
                      experts) receive orientation on the
                      solicitation’s objectives and procedures.
                                                                                                                               (continued)




                                             Page 42                                                 GAO/AIMD-97-47 Air Traffic Control
                                         Chapter 3
                                         Solicitation




                                          Automated Radar Terminal System IIIE
                 Key Practice                                      Finding                                        Rating
Activity 8       The project team and the offeror review the       Officials stated that meetings were held with Observation
                 project’s software requirements during            the contractor to ensure mutual
                 negotiations to ensure mutual                     understanding, however, they could not
                 understanding.                                    provide documents to support this.
Measurement 1    Measurements are made and used to                 No internal process measurements are           Weakness
                 determine the status of the solicitation          taken and used to determine the status of
                 activities.                                       activities for any of the key process areas.
Verification 1   The activities for solicitation are reviewed by   While the Integrated Product Team leader     Weakness
                 the designated selection official or              reviews the status of the contract and the
                 acquisition organization management on a          contractor’s cost and schedule, he does not
                 periodic basis.                                   review the status of the activities that are
                                                                   required to be performed for any of the
                                                                   various key process areas.
Verification 2   The activities for solicitation are reviewed by While the product team leader reviews the        Weakness
                 the project manager on both a periodic and status of the contract and the contractor’s
                 event-driven basis.                             cost and schedule, he does not review the
                                                                 status of the activities that are required to
                                                                 be performed for any of the key process
                                                                 areas.




                                         Page 43                                                   GAO/AIMD-97-47 Air Traffic Control
                                             Chapter 3
                                             Solicitation




Table 3.2: Solicitation Findings for DSR
                                                   Display System Replacement
                      Key Practice                                   Finding                                         Rating
Commitment 1          The acquisition organization has a written     There is an FAA policy that addresses           Strength
                      policy for the conduct of the solicitation.    solicitation conduct.
Commitment 2          Responsibility for the software portion of the Officials stated that responsibility for the    Observation
                      solicitation is designated.                    software portion of the solicitation has been
                                                                     assigned, however, they could not provide
                                                                     documents to support this.
Commitment 3          A selection official has been designated to    Not applicable. DSR was a change order          Not rated
                      be responsible for the selection process       from an existing larger contract that went
                      and the decision.                              through the acquisition phase in 1984.
                                                                     Current team members joined the team
                                                                     after the change order was negotiated and,
                                                                     therefore, could not address this key
                                                                     practice.
Ability 1             A group that is responsible for coordinating   A group responsible for the solicitation        Strength
                      and conducting the solicitation activities     exists.
                      exists.
Ability 2             Adequate resources are provided for the        No mechanism exists for identifying          Weakness
                      solicitation activities.                       resources required and for ensuring that the
                                                                     needed resources are provided to the
                                                                     project.
Ability 3             Individuals performing the solicitation        The acquisition organization has no             Observation
                      activities have experience or receive          guidance regarding training or experience
                      training.                                      requirements for project participation.
Activity 1            The project team documents its plans for       Not applicable. DSR was a change order          Not rated
                      solicitation activities.                       from an existing larger contract that went
                                                                     through the acquisition phase in 1984.
                                                                     Current team members joined the team
                                                                     after the change order was negotiated and,
                                                                     therefore, could not address this key
                                                                     practice.
Activity 2            The project’s solicitation activities are      Not applicable. DSR was a change order          Not rated
                      performed in accordance with its plans.        from an existing larger contract that went
                                                                     through the acquisition phase in 1984.
                                                                     Current team members joined the team
                                                                     after the change order was negotiated and,
                                                                     therefore, could not address this key
                                                                     practice.
Activity 3            The project team documents its plans for       The product team uses a change order            Strength
                      proposal evaluation activities.                evaluation plan to document plans for
                                                                     proposal evaluation activities.
                                                                                                                                (continued)




                                             Page 44                                                 GAO/AIMD-97-47 Air Traffic Control
                                         Chapter 3
                                         Solicitation




                                               Display System Replacement
                 Key Practice                                      Finding                                        Rating
Activity 4       The project team’s proposal evaluation            Not applicable. DSR was a change order         Not rated
                 activities are performed in accordance with       from an existing larger contract that went
                 its plans.                                        through the acquisition phase in 1984.
                                                                   Current team members joined the team
                                                                   after the change order was negotiated and,
                                                                   therefore, could not address this key
                                                                   practice.
Activity 5       A cost estimate and schedule for the         A cost estimate and schedule were                   Strength
                 software activity being sought are prepared. generated.
Activity 6       The software cost estimate and schedule           Officials could not produce documentation      Observation
                 are independently reviewed for                    that supported their statement that the
                 comprehensiveness and realism.                    software cost estimate and schedule were
                                                                   independently reviewed.
Activity 7       The groups supporting the solicitation (e.g.,     Not applicable. DSR was a change order         Not rated
                 end user, systems engineering, support            from an existing larger contract that went
                 organization, and application domain              through the acquisition phase in 1984.
                 experts) receive orientation on the               Current team members joined the team
                 solicitation’s objectives and procedures.         after the change order was negotiated and,
                                                                   therefore, could not address this key
                                                                   practice.
Activity 8       The project team and the offeror review the       A series of scheduled meetings were held       Strength
                 project’s software requirements during            to ensure mutual understanding of
                 negotiations to ensure mutual                     requirements.
                 understanding.
Measurement 1    Measurements are made and used to                 No internal process measurements are           Weakness
                 determine the status of the solicitation          taken and used to determine the status of
                 activities.                                       activities for any of the key process areas.
Verification 1   The activities for solicitation are reviewed by   While the Integrated Product Team leader     Weakness
                 the designated selection official or              reviews the status of the contract and the
                 acquisition organization management on a          contractor’s cost and schedule, he does not
                 periodic basis.                                   review the status of the activities that are
                                                                   required to be performed for any of the key
                                                                   process areas.
Verification 2   The activities for solicitation are reviewed by While the product team leader reviews the        Weakness
                 the project manager on both a periodic and status of the contract and the contractor’s
                 event-driven basis.                             cost and schedule, he does not review the
                                                                 status of the activities that are required to
                                                                 be performed for any of the key process
                                                                 areas.




                                         Page 45                                                   GAO/AIMD-97-47 Air Traffic Control
                                             Chapter 3
                                             Solicitation




Table 3.3: Solicitation Findings for NIMS
                                             NAS Infrastructure Management System
                      Key Practice                                   Finding                                          Rating
Commitment 1          The acquisition organization has a written     The Acquisition Management System is the         Strength
                      policy for the conduct of the solicitation.    written policy.
Commitment 2          Responsibility for the software portion of the Responsibility for the software portion of the Strength
                      solicitation is designated.                    solicitation has been designated to software
                                                                     experts on the team.
Commitment 3          A selection official has been designated to    A selection official has been designated to      Strength
                      be responsible for the selection process       be responsible for the selection process
                      and the decision.                              and decision.
Ability 1             A group that is responsible for coordinating   The contracting officer, support staff, and      Strength
                      and conducting the solicitation activities     the parent ASU organization are
                      exists.                                        responsible for coordinating and
                                                                     conducting the solicitation activities.
Ability 2             Adequate resources are provided for the        No mechanism exists for identifying          Weakness
                      solicitation activities.                       resources required and for ensuring that the
                                                                     needed resources are provided to the
                                                                     project.
Ability 3             Individuals performing the solicitation        The acquisition organization has no              Observation
                      activities have experience or receive          guidance regarding training or experience
                      training.                                      requirements for project participation.
Activity 1            The project team documents its plans for       The product team documents its plans for         Strength
                      solicitation activities.                       solicitation activities.
Activity 2            The project’s solicitation activities are      Officials stated that solicitation activities will Strength
                      performed in accordance with its plans.        be performed in accordance with its plans.
                                                                     NIMS is in the presolicitation phase.
Activity 3            The project team documents its plans for       The project team has documented its plans        Strength
                      proposal evaluation activities.                for proposal evaluation activities.
Activity 4            The project team’s proposal evaluation         In accordance with the plan,                     Strength
                      activities are performed in accordance with    prequalification was completed and
                      its plans.                                     vendors were down-selected from it.
Activity 5            A cost estimate and schedule for the         The Acquisition Program Baseline includes          Strength
                      software activity being sought are prepared. a cost estimate and schedule for the
                                                                   software acquisition.
Activity 6            The software cost estimate and schedule        An independent assessment was done.              Strength
                      are independently reviewed for
                      comprehensiveness and realism.
Activity 7            The groups supporting the solicitation (e.g., Solicitation activities orientation was           Strength
                      end user, systems engineering, support        conducted for NIMS personnel.
                      organization, and application domain
                      experts) receive orientation on the
                      solicitation’s objectives and procedures.
Activity 8            The project team and the offeror review the    The NIMS project has not yet reached this        Not rated
                      project’s software requirements during         stage, therefore, this activity was not rated.
                      negotiations to ensure mutual
                      understanding.
                                                                                                                                   (continued)



                                             Page 46                                                  GAO/AIMD-97-47 Air Traffic Control
                                         Chapter 3
                                         Solicitation




                                         NAS Infrastructure Management System
                 Key Practice                                      Finding                                       Rating
Measurement 1    Measurements are made and used to                 No internal process measurements are            Weakness
                 determine the status of the solicitation          taken to determine the status of activities for
                 activities.                                       any of the key process areas.
Verification 1   The activities for solicitation are reviewed by   While the Integrated Product Team leader     Weakness
                 the designated selection official or              reviews the status of the contract and the
                 acquisition organization management on a          contractor’s cost and schedule, he does not
                 periodic basis.                                   review the status of the activities that are
                                                                   required to be performed for any of the key
                                                                   process areas.
Verification 2   The activities for solicitation are reviewed by While the product team leader reviews the       Weakness
                 the project manager on both a periodic and status of the contract and the contractor’s
                 event-driven basis.                             cost and schedule, he does not review the
                                                                 status of the activities that are required to
                                                                 be performed for any of the key process
                                                                 areas.




                                         Page 47                                                  GAO/AIMD-97-47 Air Traffic Control
                                             Chapter 3
                                             Solicitation




Table 3.4: Solicitation Findings for VSCS
                                               Voice Switching and Control System
                      Key Practice                                    Finding                                            Rating
Commitment 1          The acquisition organization has a written      FAA Order 1810.1F is the written policy.           Strength
                      policy for the conduct of the solicitation.
Commitment 2          Responsibility for the software portion of the Officials gave conflicting answers as to who Weakness
                      solicitation is designated.                    is responsible for software acquisition, and
                                                                     could not provide documentation that
                                                                     formally designates responsibility.
Commitment 3          A selection official has been designated to     The Administrator was the selection official.      Strength
                      be responsible for the selection process
                      and the decision.
Ability 1             A group that is responsible for coordinating    Officials stated that a group was                  Observation
                      and conducting the solicitation activities      responsible for coordinating and
                      exists.                                         conducting solicitation activities; however,
                                                                      they could not provide documentation to
                                                                      support this claim.
Ability 2             Adequate resources are provided for the         No mechanism exists for identifying the      Weakness
                      solicitation activities.                        resources required and for ensuring that the
                                                                      needed resources are provided to the
                                                                      project.
Ability 3             Individuals performing the solicitation         The acquisition organization has no                Observation
                      activities have experience or receive           guidance regarding training or experience
                      training.                                       requirements for project participation.
Activity 1            The project team documents its plans for        Officials said that solicitation activities were   Observation
                      solicitation activities.                        documented in the Acquisition Plan and
                                                                      Source Evaluation Plan; however, they
                                                                      could not provide these documents.
Activity 2            The project’s solicitation activities are       Solicitation activities were not performed in      Weakness
                      performed in accordance with its plans.         accordance with plans.
Activity 3            The project team documents its plans for        Officials stated that proposal evaluation          Observation
                      proposal evaluation activities.                 activities were documented; however, they
                                                                      could not provide the documentation.
Activity 4            The project team’s proposal evaluation          Officials could not describe how or if             Weakness
                      activities are performed in accordance with     proposal evaluation activities were
                      its plans.                                      performed in accordance with plans;
                                                                      additionally, they could not provide
                                                                      documents to support this activity.
Activity 5            A cost estimate and schedule for the         A cost estimate and schedule for the                  Strength
                      software activity being sought are prepared. software activity were developed.
Activity 6            The software cost estimate and schedule         The software cost estimate and schedule            Strength
                      are independently reviewed for                  were independently reviewed for
                      comprehensiveness and realism.                  comprehensiveness and realism.
Activity 7            The groups supporting the solicitation (e.g.,   Officials stated that orientation was held,    Weakness
                      end user, systems engineering, support          however, the documentation provided did
                      organization, and application domain            not indicate that the product team received
                      experts) receive orientation on the             orientation on the solicitation objectives and
                      solicitation’s objectives and procedures.       procedures.
                                                                                                                                    (continued)


                                             Page 48                                                    GAO/AIMD-97-47 Air Traffic Control
                                         Chapter 3
                                         Solicitation




                                           Voice Switching and Control System
                 Key Practice                                      Finding                                        Rating
Activity 8       The project team and the offeror review the       Officials stated that the product team and     Observation
                 project’s software requirements during            the offeror reviewed project software
                 negotiations to ensure mutual                     requirements during pre-award
                 understanding.                                    negotiations, however, they could not
                                                                   provide documentation to support this.
Measurement 1    Measurements are made and used to                 No internal process measurements are           Weakness
                 determine the status of the solicitation          taken or used to determine the status of
                 activities.                                       activities for any of the key process areas.
Verification 1   The activities for solicitation are reviewed by   While the Integrated Product Team leader     Weakness
                 the designated selection official or              reviews the status of the contract and the
                 acquisition organization management on a          contractor’s cost and schedule, he does not
                 periodic basis.                                   review the status of the activities that are
                                                                   required to be performed for any of the key
                                                                   process areas.
Verification 2   The activities for solicitation are reviewed by While the product team leader reviews the        Weakness
                 the project manager on both a periodic and status of the contract and the contractor’s
                 event-driven basis.                             cost and schedule, he does not review the
                                                                 status of the activities that are required to
                                                                 be performed for any of the key process
                                                                 areas.




                                         Page 49                                                   GAO/AIMD-97-47 Air Traffic Control
                                             Chapter 3
                                             Solicitation




Table 3.5: Solicitation Findings for WARP
                                                   Weather and Radar Processor
                      Key Practice                                    Finding                                         Rating
Commitment 1          The acquisition organization has a written      There is a written policy for solicitation.     Strength
                      policy for the conduct of the solicitation.
Commitment 2          Responsibility for the software portion of the Responsibility for the solicitation has been     Strength
                      solicitation is designated.                    designated.
Commitment 3          A selection official has been designated to     Because there was only one offeror, this        Not rated
                      be responsible for the selection process        commitment was not rated.
                      and the decision.
Ability 1             A group that is responsible for coordinating    A group is responsible for coordinating and Strength
                      and conducting the solicitation activities      conducting the solicitation activities.
                      exists.
Ability 2             Adequate resources are provided for the         No mechanism exists for identifying and for     Weakness
                      solicitation activities.                        ensuring that the needed resources are
                                                                      provided to the project.
Ability 3             Individuals performing the solicitation         The acquisition organization has no             Observation
                      activities have experience or receive           guidance regarding training or experience
                      training.                                       requirements for project participation.
Activity 1            The project team documents its plans for        The project team documents its plans for        Strength
                      solicitation activities.                        solicitation activities.
Activity 2            The project’s solicitation activities are       The project’s solicitation activities were      Strength
                      performed in accordance with its plans.         performed in accordance with its plans.
Activity 3            The project team documents its plans for        The team documented its plans for               Strength
                      proposal evaluation activities.                 proposal evaluation activities.
Activity 4            The project team’s proposal evaluation          Proposal evaluation activities were             Strength
                      activities are performed in accordance with     performed in accordance with the plan.
                      its plans.
Activity 5            A cost estimate and schedule for the         An independent government cost estimate            Strength
                      software activity being sought are prepared. was prepared which included major
                                                                   milestone data.
Activity 6            The software cost estimate and schedule         The software cost estimate and schedule         Strength
                      are independently reviewed for                  were independently reviewed.
                      comprehensiveness and realism.
Activity 7            The groups supporting the solicitation (e.g.,   Officials stated that team members            Observation
                      end user, systems engineering, support          received orientation at the beginning of the
                      organization, and application domain            solicitation, however, they could not provide
                      experts) receive orientation on the             documentation to support this.
                      solicitation’s objectives and procedures.
Activity 8            The project team and the offeror review the     Numerous negotiation sessions were held.        Strength
                      project’s software requirements during
                      negotiations to ensure mutual
                      understanding.
Measurement 1         Measurements are made and used to               No internal process measurements are            Weakness
                      determine the status of the solicitation        taken and used to determine the status of
                      activities.                                     activities for any of the key process areas.
                                                                                                                                 (continued)



                                             Page 50                                                    GAO/AIMD-97-47 Air Traffic Control
                                         Chapter 3
                                         Solicitation




                                              Weather and Radar Processor
                 Key Practice                                      Finding                                       Rating
Verification 1   The activities for solicitation are reviewed by   While the Integrated Product Team leader     Weakness
                 the designated selection official or              reviews the status of the contract and the
                 acquisition organization management on a          contractor’s cost and schedule, he does not
                 periodic basis.                                   review the status of the activities that are
                                                                   required to be performed for any of the key
                                                                   process areas.
Verification 2   The activities for solicitation are reviewed by While the product team leader reviews the       Weakness
                 the project manager on both a periodic and status of the contract and the contractor’s
                 event-driven basis.                             cost and schedule, he does not review the
                                                                 status of the activities that are required to
                                                                 be performed for any of the key process
                                                                 areas.


                                         While FAA has many strengths in this KPA, systemic weaknesses in areas
Conclusions                              including measurement and analysis and management verification of
                                         practices, along with other project-specific weaknesses, render this KPA
                                         non-repeatable and dependent upon the capabilities and commitment of
                                         individual employees.




                                         Page 51                                                  GAO/AIMD-97-47 Air Traffic Control
Chapter 4

Requirements Development and
Management

                     The purpose of requirements development and management is to establish
                     and maintain a common and unambiguous definition of software
                     requirements among the acquisition team, the system users, and the
                     software development contractor. This KPA involves two subprocesses:
                     (1) developing a baseline set of software-related contractual requirements,
                     and (2) managing these requirements and changes to these requirements
                     for the duration of the acquisition.

                     The SA-CMM specifies a number of requirements development and
                     management practices necessary to achieve a repeatable maturity level.
                     These include (1) having a written organizational policy for establishing
                     and managing requirements allocated to software; (2) documenting plans
                     for the development and management of requirements; (3) having
                     documented processes for requirements development, including
                     elicitation, analysis, and verification; (4) measuring and reporting on the
                     status of requirements development and management activities to
                     management; (5) appraising the impact on software of system-level
                     requirements changes; and (6) having a mechanism to ensure that
                     contractor-delivered work products meet specified requirements.


                     In the past, we have attributed ATC modernization problems, in part, to
Requirements         FAA’s failure to effectively manage requirements. For example, we reported
Development and      in 1994 that FAA did not adequately specify or effectively control changes
Management Process   to the requirements of its Initial Sector Suite System (ISSS) component of
                     the Advanced Automation System.1
Is Not Effective
                     Our evaluation of FAA’s capability relative to this KPA’s requirements
                     reiterates our earlier reported concerns in this area and pinpoints specific
                     weaknesses. For example, while FAA has a policy on requirements
                     development and management, this policy does not address establishing
                     and managing software requirements. Further, product teams do not
                     always document their requirements development and management plans,
                     and while two had a defined process for controlling changes to existing,
                     baselined requirements, they did not have a documented process for
                     developing new software requirements, including requirements planning,
                     elicitation, analysis, or verification. Additionally, management does not
                     oversee or verify requirements development and management activities,
                     which means that management has no assurance that specified



                     1
                     Advanced Automation System: Implications of Problems and Recent Changes (GAO/T-RCED-94-188,
                     Apr. 13, 1994).



                     Page 52                                                 GAO/AIMD-97-47 Air Traffic Control
Chapter 4
Requirements Development and
Management




requirements are correct and complete, and does not know when
management action is warranted.

We also found some practice strengths. For example, most projects (1) are
assessing the impact on software requirements of system-level
requirements changes and (2) have a mechanism to ensure that
contractor-delivered work products and services satisfied specified
software requirements.

Figure 4.1 provides a comprehensive listing of the five projects’ strengths,
weaknesses, and observations for the requirements development and
management KPA. The specific findings supporting the practice ratings in
figure 4.1 are in tables 4.1 through 4.5.




Page 53                                       GAO/AIMD-97-47 Air Traffic Control
                                                     Chapter 4
                                                     Requirements Development and
                                                     Management




Figure 4.1: Requirements Development and Management Summary


                    Key Practice                                                ARTSIIIE              DSR                NIMS               VSCS           WARP

                    The acquisition organization has a written policy for
  Commitment 1      establishing and managing the system requirements            Weakness          Weakness            Weakness           Weakness       Weakness
                    allocated to software.


     Ability 1      A group that is responsible for performing requirements      Strength           Strength           Strength           Strength        Strength
                    development and management activities exists.


     Ability 2      Adequate resources are provided for requirements            Weakness            Strength          Weakness           Weakness        Weakness
                    development and management activities.

                    Individuals performing requirements development and
     Ability 3      management activities have experience or receive           Observation        Observation         Observation       Observation      Observation
                    training.

                    The project team documents its plans for
     Activity 1     requirements development and management activities.         Weakness           Weakness            Not rated          Weakness       Weakness


                    The project's requirements development and
     Activity 2     management activities are performed in accordance           Weakness           Weakness            Not rated          Weakness       Weakness
                    with its plans.

                    The project team baselines the software requirements
     Activity 3     and places them under change control early in the            Strength          Weakness            Not rated        Observation       Strength
                    project, but not later than release of the solicitation.

                    The project team appraises system requirements
     Activity 4     change requests for their impact on software.                Strength           Strength           Not rated          Strength        Strength

                    The project team appraises software requirements
                    changes for their impact on performance, schedule,
     Activity 5     cost, system capacities, supportability, and                 Strength          Weakness            Not rated          Weakness        Not rated
                    architecture.
                    The project team maintains a requirements
                    mechanism for traceability during the software effort
     Activity 6     to ensure requirements have been included in the             Strength           Strength           Not rated          Strength        Strength
                    implemented work products and services.

                    Measurements are made and used to determine the
  Measurement 1     status of the requirements development and                  Weakness           Weakness           Weakness            Weakness       Weakness
                    management activities.

                    The activities for requirements development and
   Verification 1   management are reviewed by acquisition organization         Weakness           Weakness           Weakness            Weakness       Weakness
                    management (and the contractor) on a periodic basis.

                    The activities for requirements development and
   Verification 2   management are reviewed by the project manager on           Weakness           Weakness           Weakness            Weakness       Weakness
                    both a periodic and event-driven basis.


                                                              = Weakness       Key practice not implemented
                                                              = Strength       Key practice effectively implemented
                                                              = Observation Key practice evaluated but evidence inconclusive. Cannot characterize as either strength
                                                                            or weakness

                                                              = Not rated      Key practice not currently relevant to project, therefore not evaluated




                                                     Page 54                                                                 GAO/AIMD-97-47 Air Traffic Control
                                           Chapter 4
                                           Requirements Development and
                                           Management




Table 4.1: Requirements Development and Management Findings for ARTSIIIE
                                         Automated Radar Terminal System IIIE
                    Key Practice                                    Finding                                        Rating
Commitment 1        The acquisition organization has a written      Officials stated that FAA Order 1810.1F is     Weakness
                    policy for establishing and managing the        the written policy for requirements
                    system requirements allocated to software.      development and management, however, it
                                                                    does not address software requirements.
Ability 1           A group that is responsible for performing      A group responsible for requirements           Strength
                    requirements development and                    development and management exists.
                    management activities exists.
Ability 2           Adequate resources are provided for             No mechanism exists for identifying and for    Weakness
                    requirements development and                    ensuring that the needed resources are
                    management activities.                          provided to the project.
Ability 3           Individuals performing requirements             The acquisition organization has no            Observation
                    development and management activities           guidance regarding training or experience
                    have experience or receive training.            requirements for project participation.
Activity 1          The project team documents its plans for        The project team does not document its         Weakness
                    requirements development and                    plans for requirements development,
                    management activities.                          planning, elicitation, analysis, and
                                                                    verification.
Activity 2          The project’s requirements development          There is no requirements development and       Weakness
                    and management activities are performed         management plan, therefore, the activities
                    in accordance with its plans.                   are not performed in accordance with it.
Activity 3          The project team baselines the software         A requirements baseline, which is under      Strength
                    requirements and places them under              change control, was established prior to the
                    change control early in the project, but not    release of the solicitation.
                    later than release of the solicitation.
Activity 4          The project team appraises system               The project team’s appraisals of the impact    Strength
                    requirements change requests for their          of system requirements changes on
                    impact on software.                             software are documented.
Activity 5          The project team appraises software             The project team’s appraisal of software       Strength
                    requirements changes for their impact on        requirements changes’ impact on
                    performance, schedule, cost, system             performance, schedule, cost, and system
                    capacities, supportability, and architecture.   capacities is documented, but not the
                                                                    impact on system supportability or
                                                                    architecture.
Activity 6          The project team maintains a requirements       There is a mechanism for traceability of       Strength
                    mechanism for traceability during the           software requirements implementation.
                    software effort to ensure requirements have
                    been included in the implemented work
                    products and services.
Measurement 1       Measurements are made and used to               No internal process measurements are           Weakness
                    determine the status of the requirements        taken and used to determine the status of
                    development and management activities.          activities for any of the key process areas.
                                                                                                                              (continued)




                                           Page 55                                                  GAO/AIMD-97-47 Air Traffic Control
                                       Chapter 4
                                       Requirements Development and
                                       Management




                                        Automated Radar Terminal System IIIE
                 Key Practice                                  Finding                                         Rating
Verification 1   The activities for requirements development   While the Integrated Product Team leader     Weakness
                 and management are reviewed by                reviews the status of the contract and the
                 acquisition organization management (and      contractor’s cost and schedule, he does not
                 the contractor) on a periodic basis.          review the status of the activities that are
                                                               required to be performed for any of the key
                                                               process areas.
Verification 2   The activities for requirements development   While the product team leader reviews the       Weakness
                 and management are reviewed by the            status of the contract and the contractor’s
                 project manager on both a periodic and        cost and schedule, he does not review the
                 event-driven basis.                           status of the activities that are required to
                                                               be performed for any of the key process
                                                               areas.




                                       Page 56                                                  GAO/AIMD-97-47 Air Traffic Control
                                               Chapter 4
                                               Requirements Development and
                                               Management




Table 4.2: Requirements Development and Management Findings for DSR
                                             Display System Replacement
                 Key Practice                                        Finding                                              Rating
Commitment 1     The acquisition organization has a written policy   Officials stated that FAA Order 1810.1F is the       Weakness
                 for establishing and managing the system            written policy for requirements development and
                 requirements allocated to software.                 management, however, it does not address
                                                                     software requirements.
Ability 1        A group that is responsible for performing          The group responsible for requirements               Strength
                 requirements development and management             development and management is the product
                 activities exists.                                  team.
Ability 2        Adequate resources are provided for                 Adequate resources are provided for                  Strength
                 requirements development and management             requirements development and management
                 activities.                                         activities.
Ability 3        Individuals performing requirements development     The acquisition organization has no guidance         Observation
                 and management activities have experience or        regarding training or experience requirements for
                 receive training.                                   project participation.
Activity 1       The project team documents its plans for            While processes exist for milestone review, these    Weakness
                 requirements development and management             documents do not cover the activities to be
                 activities.                                         performed such as user involvement, elicitation,
                                                                     and requirements development.
Activity 2       The project’s requirements development and          There is no requirements development and             Weakness
                 management activities are performed in              management plan, therefore, the activities are not
                 accordance with its plans.                          performed in accordance with it.
Activity 3       The project team baselines the software             Software requirements are baselined as part of the Weakness
                 requirements and places them under change           contract process, but not explicitly prior to
                 control early in the project, but not later than    solicitation.
                 release of the solicitation.
Activity 4       The project team appraises system requirements      The product team appraises system requirements       Strength
                 change requests for their impact on software.       changes for their impact on software.
Activity 5       The project team appraises software requirements Software requirements were not appraised for           Weakness
                 changes for their impact on performance,           their impact on performance, schedule, cost,
                 schedule, cost, system capacities, supportability, system capacities, supportability, and architecture.
                 and architecture.
Activity 6       The project team maintains a requirements           The product team maintains a mechanism for           Strength
                 mechanism for traceability during the software      requirements traceability.
                 effort to ensure requirements have been included
                 in the implemented work products and services.
Measurement 1    Measurements are made and used to determine         No internal measurements are taken and used to        Weakness
                 the status of the requirements development and      determine the status of activities for any of the key
                 management activities.                              process areas.
Verification 1   The activities for requirements development and     While the Integrated Product Team leader reviews Weakness
                 management are reviewed by acquisition              the status of the contract and the contractor’s cost
                 organization management (and the contractor) on     and schedule, he does not review the status of the
                 a periodic basis.                                   activities that are required to be performed for any
                                                                     of the key process areas.
Verification 2   The activities for requirements development and     While the product team leader reviews the status     Weakness
                 management are reviewed by the project              of the contract and the contractor’s cost and
                 manager on both a periodic and event-driven         schedule, he does not review the status of the
                 basis.                                              activities that are required to be performed for any
                                                                     of the key process areas.

                                               Page 57                                             GAO/AIMD-97-47 Air Traffic Control
                                            Chapter 4
                                            Requirements Development and
                                            Management




Table 4.3: Requirements Development and Management Findings for NIMS
                                        NAS Infrastructure Management System
                   Key Practice                                       Finding                                              Rating
Commitment 1       The acquisition organization has a written policy Officials cited the Acquisition Management            Weakness
                   for establishing and managing the system          System and FAA Order 1810.1F, but these do
                   requirements allocated to software.               not address software requirements.
Ability 1          A group that is responsible for performing         The team members are assigned collective             Strength
                   requirements development and management            responsibility for the requirements process.
                   activities exists.
Ability 2          Adequate resources are provided for                No mechanism exists for identifying resources        Weakness
                   requirements development and management            required and for ensuring that the needed
                   activities.                                        resources are provided to the project.
Ability 3          Individuals performing requirements                The acquisition organization has no guidance         Observation
                   development and management activities have         regarding training or experience requirements
                   experience or receive training.                    for project participation.
Activity 1         The project team documents its plans for           The project has not reached the point where          Not rated
                   requirements development and management            these activities are performed.
                   activities.
Activity 2         The project’s requirements development and         The project has not reached the point where          Not rated
                   management activities are performed in             these activities are performed.
                   accordance with its plans.
Activity 3         The project team baselines the software            It is too early in the project life to assess: the   Not rated
                   requirements and places them under change          software requirements have not been
                   control early in the project, but not later than   developed.
                   release of the solicitation.
Activity 4         The project team appraises system                  It is too early in the project life to assess: no    Not rated
                   requirements change requests for their impact      software has been developed or specified.
                   on software.
Activity 5         The project team appraises software                It is too early in the project life to assess: no    Not rated
                   requirements changes for their impact on           software requirements have been developed or
                   performance, schedule, cost, system                specified.
                   capacities, supportability, and architecture.
Activity 6         The project team maintains a requirements      No traceability matrix of software requirements          Not rated
                   mechanism for traceability during the software has been developed at this point in the project.
                   effort to ensure requirements have been
                   included in the implemented work products and
                   services.
Measurement 1      Measurements are made and used to determine No internal process measurements are taken to               Weakness
                   the status of the requirements development and determine the status of activities for any of the
                   management activities.                         key process areas.
Verification 1     The activities for requirements development and    While the Integrated Product Team leader             Weakness
                   management are reviewed by acquisition             reviews the status of the contract and the
                   organization management (and the contractor)       contractor’s cost and schedule, he does not
                   on a periodic basis.                               review the status of the activities that are
                                                                      required to be performed for any of the key
                                                                      process areas.
Verification 2     The activities for requirements development and    While the product team leader reviews the            Weakness
                   management are reviewed by the project             status of the contract and the contractor’s cost
                   manager on both a periodic and event-driven        and schedule, he does not review the status of
                   basis.                                             the activities that are required to be performed
                                                                      for any of the key process areas.
                                            Page 58                                                  GAO/AIMD-97-47 Air Traffic Control
                                           Chapter 4
                                           Requirements Development and
                                           Management




Table 4.4: Requirements Development and Management Findings for VSCS
                                         Voice Switching and Control System
                    Key Practice                                    Finding                                        Rating
Commitment 1        The acquisition organization has a written      Officials stated that FAA Order 1810.1F is     Weakness
                    policy for establishing and managing the        the written policy for requirements
                    system requirements allocated to software.      development and management, however, it
                                                                    does not address software requirements.
Ability 1           A group that is responsible for performing      The product team is responsible for            Strength
                    requirements development and                    requirements development and
                    management activities exists.                   management planning.
Ability 2           Adequate resources are provided for             No mechanism exists for identifying the      Weakness
                    requirements development and                    resources required and for ensuring that the
                    management activities.                          needed resources are provided to the
                                                                    project.
Ability 3           Individuals performing requirements             The acquisition organization has no            Observation
                    development and management activities           guidance regarding training or experience
                    have experience or receive training.            requirements for project participation.
Activity 1          The project team documents its plans for        There are no documented plans for              Weakness
                    requirements development and                    requirements development and
                    management activities.                          management activities.
Activity 2          The project’s requirements development          There is no requirements development and       Weakness
                    and management activities are performed         management plan, therefore, the activities
                    in accordance with its plans.                   cannot be performed in accordance with it.
Activity 3          The project team baselines the software         Officials stated that requirements are     Observation
                    requirements and places them under              baselined at contract award, but no
                    change control early in the project, but not    documentation was provided to support this
                    later than release of the solicitation.         statement.
Activity 4          The project team appraises system               The project team appraises system              Strength
                    requirements change requests for their          requirements change requests for their
                    impact on software.                             impact on software.
Activity 5          The project team appraises software             The project team appraises software          Weakness
                    requirements changes for their impact on        requirements changes for their impact on
                    performance, schedule, cost, system             performance, schedule, cost, and system
                    capacities, supportability, and architecture.   capacities, but not on system supportability
                                                                    and architecture.
Activity 6          The project team maintains a requirements       A mechanism for traceability during the        Strength
                    mechanism for traceability during the           software effort is maintained.
                    software effort to ensure requirements have
                    been included in the implemented work
                    products and services.
Measurement 1       Measurements are made and used to               No internal process measurements are           Weakness
                    determine the status of the requirements        taken or used to determine the status of
                    development and management activities.          activities for any of the key process areas.
                                                                                                                              (continued)




                                           Page 59                                                  GAO/AIMD-97-47 Air Traffic Control
                                       Chapter 4
                                       Requirements Development and
                                       Management




                                         Voice Switching and Control System
                 Key Practice                                  Finding                                         Rating
Verification 1   The activities for requirements development   While the Integrated Product Team leader     Weakness
                 and management are reviewed by                reviews the status of the contract and the
                 acquisition organization management (and      contractor’s cost and schedule, he does not
                 the contractor) on a periodic basis.          review the status of the activities that are
                                                               required to be performed for any of the key
                                                               process areas.
Verification 2   The activities for requirements development   While the product team leader reviews the       Weakness
                 and management are reviewed by the            status of the contract and the contractor’s
                 project manager on both a periodic and        cost and schedule, he does not review the
                 event-driven basis.                           status of the activities that are required to
                                                               be performed for any of the key process
                                                               areas.




                                       Page 60                                                  GAO/AIMD-97-47 Air Traffic Control
                                           Chapter 4
                                           Requirements Development and
                                           Management




Table 4.5: Requirements Development and Management Findings for WARP
                                            Weather and Radar Processor
                    Key Practice                                    Finding                                        Rating
Commitment 1        The acquisition organization has a written      Officials stated that FAA Order 1810.1F is     Weakness
                    policy for establishing and managing the        the written policy for requirements
                    system requirements allocated to software.      development and management, however, it
                                                                    does not address software requirements.
Ability 1           A group that is responsible for performing      The team is responsible for requirements       Strength
                    requirements development and                    development and measurement activities.
                    management activities exists.
Ability 2           Adequate resources are provided for             No mechanism exists for identifying and for    Weakness
                    requirements development and                    ensuring that the needed resources are
                    management activities.                          provided to the project.
Ability 3           Individuals performing requirements             The acquisition organization has no            Observation
                    development and management activities           guidance regarding training or experience
                    have experience or receive training.            requirements for project participation.
Activity 1          The project team documents its plans for        While a process for managing requirements Weakness
                    requirements development and                    changes exists, there is no documented
                    management activities.                          process for requirements development and
                                                                    management.
Activity 2          The project’s requirements development          There is no plan, thus, it cannot be followed. Weakness
                    and management activities are performed
                    in accordance with its plans.
Activity 3          The project team baselines the software         The product team baselined the software        Strength
                    requirements and places them under              requirements and placed them under
                    change control early in the project, but not    change control before the release of the
                    later than release of the solicitation.         solicitation.
Activity 4          The project team appraises system               The product team appraises system              Strength
                    requirements change requests for their          requirements change requests for their
                    impact on software.                             impact on software.
Activity 5          The project team appraises software             WARP has not had a software requirement        Not rated
                    requirements changes for their impact on        change yet; therefore, this activity was not
                    performance, schedule, cost, system             rated.
                    capacities, supportability, and architecture.
Activity 6          The project team maintains a requirements       There is a traceability matrix for tracking    Strength
                    mechanism for traceability during the           software requirements implementation in
                    software effort to ensure requirements have     the system specification.
                    been included in the implemented work
                    products and services.
Measurement 1       Measurements are made and used to               No internal process measurements are           Weakness
                    determine the status of the requirements        taken and used to determine the status of
                    development and management activities.          activities for any of the key process areas.
Verification 1      The activities for requirements development     While the Integrated Product Team leader     Weakness
                    and management are reviewed by                  reviews the status of the contract and the
                    acquisition organization management (and        contractor’s cost and schedule, he does not
                    the contractor) on a periodic basis.            review the status of the activities that are
                                                                    required to be performed for any of the key
                                                                    process areas.
                                                                                                                              (continued)


                                           Page 61                                                   GAO/AIMD-97-47 Air Traffic Control
                                       Chapter 4
                                       Requirements Development and
                                       Management




                                            Weather and Radar Processor
                 Key Practice                                  Finding                                         Rating
Verification 2   The activities for requirements development   While the product team leader reviews the       Weakness
                 and management are reviewed by the            status of the contract and the contractor’s
                 project manager on both a periodic and        cost and schedule, he does not review the
                 event-driven basis.                           status of the activities that are required to
                                                               be performed for any of the key process
                                                               areas.


                                       Requirements management has been a pervasive and longstanding
Conclusions                            problem with FAA’s ATC modernization, and the results of our evaluation
                                       point to many software-specific weaknesses in this area. Because of these
                                       weaknesses, it is likely that requirements management problems will
                                       continue to jeopardize projects’ cost, schedule, and performance goals.




                                       Page 62                                                  GAO/AIMD-97-47 Air Traffic Control
Chapter 5

Project Office Management


                        The purpose of project office management is to manage the activities of
                        the project office and supporting contractors to ensure a timely, efficient,
                        and effective software acquisition. According to the SA-CMM, effective
                        project office management requires, among other things, that project
                        teams (1) be organized to accomplish the project’s objective, (2) have a
                        written policy for the management of the software project, (3) document
                        its plans for the activities of the project team, (4) have the authority to
                        alter either the project’s performance, cost, or schedule baseline while
                        maintaining the other two, and (5) periodically brief management on the
                        status of project office management activities.


                        ATC modernization teams are organized to accomplish project objectives,
FAA’s Project Office    with each team including representatives from key functional areas (e.g.,
Management Process      software engineering, contracting, test and evaluation, operations and
Area Is Not Effective   maintenance). However, serious weaknesses in other KPA requirements
                        undermine FAA’s project office management capability. For example, most
                        teams lack a written policy for software project management, do not
                        document its plans for software acquisition management activities, and
                        could not identify which team member(s) is responsible for different team
                        activities (e.g., software, support, requirements, testing, and/or reviews).
                        As a result, lines of accountability and decision-making are blurred,
                        increasing the chances of delays and mistakes. Additionally, the product
                        lead cannot adjust either software performance, cost, or schedule baseline
                        while holding the other two constant. This inflexibility limits the teams’
                        ability to effectively and efficiently respond to such events as valid
                        requirements changes and funding changes. Also, project teams do not
                        periodically brief management on the status of project office activities,
                        which means that management may not be able take corrective action
                        when warranted.

                        Figure 5.1 provides a comprehensive listing of the five projects’ strengths,
                        weaknesses, and observations for the project office management KPA. The
                        specific findings supporting the practice ratings cited in figure 5.1 are in
                        tables 5.1 through 5.5.




                        Page 63                                        GAO/AIMD-97-47 Air Traffic Control
                                                      Chapter 5
                                                      Project Office Management




Figure 5.1: Project Office Management Summary


                     Key Practice                                              ARTSIIIE         DSR          NIMS          VSCS          WARP


   Commitment 1      The acquisition organization has a written policy for     Weakness       Weakness      Strength     Weakness      Weakness
                     execution of the software project.


   Commitment 2      Performance, cost, and schedule baselines are              Strength      Strength      Strength     Weakness       Strength
                     supported.

                     A project team that is responsible for performing the
      Ability 1      project's software acquisition management activities       Strength      Strength     Weakness       Strength      Strength
                     exists.

                     Adequate resources for the project team and matrix
      Ability 2      support organization(s) are provided for the duration     Weakness       Weakness     Weakness      Weakness      Weakness
                     of the software acquisition project.

                     The project manager is permitted to alter either the
      Ability 3      performance, cost, or schedule software acquisition       Weakness       Weakness     Observation   Weakness      Weakness
                     baseline while maintaining the other two constant.

                     The project team and matrix support individual(s)
      Ability 4      have experience or receive training in project office     Observation   Observation   Observation   Observation   Observation
                     software acquisition management activities.


      Activity 1     The project team documents its plans for software         Weakness       Weakness     Weakness       Strength      Strength
                     acquisition management activities.


      Activity 2     The project team is organized to accomplish the            Strength      Strength      Strength      Strength      Strength
                     project's objectives.


      Activity 3     The software acquisition activities of the project team    Strength      Strength      Not rated    Weakness       Strength
                     are directed to accomplish the project's objectives.


      Activity 4     The software acquisition activities of the project team    Strength      Strength      Not rated     Strength     Weakness
                     are controlled.


      Activity 5     Measurements are used to track project status,             Strength      Strength      Not rated     Strength      Strength
                     execution, and funding expenditures.


   Measurement 1     Measurements are made and used to determine the           Weakness       Weakness     Weakness      Weakness      Weakness
                     status of the project office management activities.

                     The activities for project office management are
    Verification 1   reviewed by acquisition organization management on        Weakness       Weakness     Weakness      Weakness      Weakness
                     a periodic basis.




                                                      Page 64                                                   GAO/AIMD-97-47 Air Traffic Control
                                               Chapter 5
                                               Project Office Management




                 Key Practice                                             ARTSIIIE             DSR                NIMS               VSCS          WARP

                 The activities for project office management are
Verification 2   reviewed by the project manager on both a periodic       Weakness           Weakness           Weakness           Weakness       Weakness
                 and event-driven basis.


                                                        = Weakness      Key practice not implemented
                                                        = Strength      Key practice effectively implemented
                                                        = Observation Key practice evaluated but evidence inconclusive. Cannot characterize as either strength or
                                                                      weakness

                                                        = Not rated     Key practice not currently relevant to project, therefore not evaluated




                                               Page 65                                                               GAO/AIMD-97-47 Air Traffic Control
                                            Chapter 5
                                            Project Office Management




Table 5.1: Project Office Management Findings for ARTSIIIE
                                           Automated Radar Terminal System IIIE
                     Key Practice                                    Finding                                        Rating
Commitment 1         The acquisition organization has a written      FAA Order 1810.1F was cited as the written Weakness
                     policy for execution of the software project.   policy, but it does not contain policy for
                                                                     software project execution.
Commitment 2         Performance, cost, and schedule baselines       Performance, cost, and schedule baselines      Strength
                     are supported.                                  are supported.
Ability 1            A project team that is responsible for          The product team is responsible for            Strength
                     performing the project’s software               performing software acquisition
                     acquisition management activities exists.       management activities.
Ability 2            Adequate resources for the project team         No mechanism exists for identifying          Weakness
                     and matrix support organization(s) are          resources required and for ensuring that the
                     provided for the duration of the software       needed resources are provided to the
                     acquisition project.                            project.
Ability 3            The project manager is permitted to alter       The acquisition baseline process does not      Weakness
                     either the performance, cost, or schedule       allow the project manager to alter the
                     software acquisition baseline while             baseline.
                     maintaining the other two constant.
Ability 4            The project team and matrix support             The acquisition organization has no            Observation
                     individual(s) have experience or receive        guidance regarding training or experience
                     training in project office software acquisition requirements for project participation.
                     management activities.
Activity 1           The project team documents its plans for    The project plans do not address software          Weakness
                     software acquisition management activities. acquisition project management.
Activity 2           The project team is organized to                The product team is organized to               Strength
                     accomplish the project’s objectives.            accomplish the project’s objectives.
Activity 3           The software acquisition activities of the  The activities of the product team are             Strength
                     project team are directed to accomplish the directed and controlled to accomplish the
                     project’s objectives.                       project’s objectives.
Activity 4           The software acquisition activities of the      The software activities of the product team    Strength
                     project team are controlled.                    are controlled.
Activity 5           Measurements are used to track project       Measurements are used to track project       Strength
                     status, execution, and funding expenditures. status, execution, and funding expenditures.
Measurement 1        Measurements are made and used to               No internal process measurements are           Weakness
                     determine the status of the project office      taken and used to determine the status of
                     management activities.                          activities for any of the key process areas.
Verification 1       The activities for project office management While the Integrated Product Team leader     Weakness
                     are reviewed by acquisition organization     reviews the status of the contract and the
                     management on a periodic basis.              contractor’s cost and schedule, he does not
                                                                  review the status of the activities that are
                                                                  required to be performed for any of the key
                                                                  process areas.
Verification 2       The activities for project office management While the product team leader reviews the         Weakness
                     are reviewed by the project manager on       status of the contract and the contractor’s
                     both a periodic and event-driven basis.      cost and schedule, he does not review the
                                                                  status of the activities that are required to
                                                                  be performed for any of the key process
                                                                  areas.


                                            Page 66                                                  GAO/AIMD-97-47 Air Traffic Control
                                            Chapter 5
                                            Project Office Management




Table 5.2: Project Office Management Findings for DSR
                                                Display System Replacement
                   Key Practice                                         Finding                                            Rating
Commitment 1       The acquisition organization has a written policy FAA Order 1810.1F was cited as the written          Weakness
                   for execution of the software project.            policy, but it does not contain policy for software
                                                                     project execution.
Commitment 2       Performance, cost, and schedule baselines are        Performance, cost, and schedule baselines are      Strength
                   supported.                                           supported.
Ability 1          A project team that is responsible for performing The product team is responsible for managing          Strength
                   the project’s software acquisition management the software acquisition management activities.
                   activities exists.
Ability 2          Adequate resources for the project team and          No mechanism exists for identifying resources      Weakness
                   matrix support organization(s) are provided for      required and for ensuring that the needed
                   the duration of the software acquisition project.    resources are provided to the project.
Ability 3          The project manager is permitted to alter either     The product team leader cannot alter the           Weakness
                   the performance, cost, or schedule software          performance, cost, or schedule software
                   acquisition baseline while maintaining the other     acquisition baseline.
                   two constant.
Ability 4          The project team and matrix support               The acquisition organization has no guidance          Observation
                   individual(s) have experience or receive training regarding training or experience requirements
                   in project office software acquisition            for project participation.
                   management activities.
Activity 1         The project team documents its plans for             Plans for software acquisition management          Weakness
                   software acquisition management activities.          activities are not documented.
Activity 2         The project team is organized to accomplish the The product team is organized to achieve the            Strength
                   project’s objectives.                           project’s objectives.
Activity 3         The software acquisition activities of the project   The software acquisition activities of the product Strength
                   team are directed to accomplish the project’s        team are directed to accomplish the project’s
                   objectives.                                          objectives.
Activity 4         The software acquisition activities of the project   The software acquisition activities of the product Strength
                   team are controlled.                                 team are controlled by the Integrated Product
                                                                        Team leader.
Activity 5         Measurements are used to track project status,       The product team is using measurements to          Strength
                   execution, and funding expenditures.                 track product status, execution, and funding
                                                                        expenditures.
Measurement 1      Measurements are made and used to determine No internal process measurements are taken         Weakness
                   the status of the project office management and used to determine the status of activities for
                   activities.                                 any of the key process areas.
Verification 1     The activities for project office management are     While the Integrated Product Team leader           Weakness
                   reviewed by acquisition organization                 reviews the status of the contract and the
                   management on a periodic basis.                      contractor’s cost and schedule, he does not
                                                                        review the status of the activities that are
                                                                        required to be performed for any of the key
                                                                        process areas.
Verification 2     The activities for project office management are     While the product team leader reviews the          Weakness
                   reviewed by the project manager on both a            status of the contract and the contractor’s cost
                   periodic and event-driven basis.                     and schedule, he does not review the status of
                                                                        the activities that are required to be performed
                                                                        for any of the key process areas.


                                            Page 67                                                  GAO/AIMD-97-47 Air Traffic Control
                                           Chapter 5
                                           Project Office Management




Table 5.3: Project Office Management Findings for NIMS
                                          NAS Infrastructure Management System
                    Key Practice                                    Finding                                       Rating
Commitment 1        The acquisition organization has a written      The policy for project office management is   Strength
                    policy for execution of the software project.   contained in the Acquisition Management
                                                                    System.
Commitment 2        Performance, cost, and schedule baselines       Performance, cost, and schedule baselines     Strength
                    are supported.                                  were developed and are being reviewed
                                                                    through the Acquisition Program Baseline
                                                                    process.
Ability 1           A project team that is responsible for          The product team is assigned the              Weakness
                    performing the project’s software               responsibility for acquisition management
                    acquisition management activities exists.       activities. However, no one is assigned
                                                                    responsibility for software acquisition
                                                                    management activities.
Ability 2           Adequate resources for the project team         No mechanism exists for identifying          Weakness
                    and matrix support organization(s) are          resources required and for ensuring that the
                    provided for the duration of the software       needed resources are provided to the
                    acquisition project.                            project.
Ability 3           The project manager is permitted to alter       Officials said they could change the cost, Observation
                    either the performance, cost, or schedule       performance, or schedule baseline, but
                    software acquisition baseline while             could not provide documentation to support
                    maintaining the other two constant.             this.
Ability 4           The project team and matrix support             The acquisition organization has no           Observation
                    individual(s) have experience or receive        guidance regarding training or experience
                    training in project office software acquisition requirements for project participation.
                    management activities.
Activity 1          The project team documents its plans for    The Integrated Program Plan and the               Weakness
                    software acquisition management activities. Product Team Plan provide plans for
                                                                acquisition management, but these do not
                                                                address software acquisition management
                                                                activities.
Activity 2          The project team is organized to                The product team is being organized to        Strength
                    accomplish the project’s objectives.            accomplish the project’s objectives.
Activity 3          The software acquisition activities of the  Too early in project life to assess: no           Not rated
                    project team are directed to accomplish the software acquisition activities performed.
                    project’s objectives.
Activity 4          The software acquisition activities of the      Too early in project life to assess: no       Not rated
                    project team are controlled.                    software acquisition activities performed.
Activity 5          Measurements are used to track project       The project has not reached a stage where        Not rated
                    status, execution, and funding expenditures. this activity applies.
Measurement 1       Measurements are made and used to               No internal process measurements are            Weakness
                    determine the status of the project office      taken to determine the status of activities for
                    management activities.                          any of the key process areas.
                                                                                                                             (continued)




                                           Page 68                                                  GAO/AIMD-97-47 Air Traffic Control
                                        Chapter 5
                                        Project Office Management




                                        NAS Infrastructure Management System
                 Key Practice                                  Finding                                        Rating
Verification 1   The activities for project office management While the Integrated Product Team leader     Weakness
                 are reviewed by acquisition organization     reviews the status of the contract and the
                 management on a periodic basis.              contractor’s cost and schedule, he does not
                                                              review the status of the activities that are
                                                              required to be performed for any of the key
                                                              process areas.
Verification 2   The activities for project office management While the product team leader reviews the       Weakness
                 are reviewed by the project manager on       status of the contract and the contractor’s
                 both a periodic and event-driven basis.      cost and schedule, he does not review the
                                                              status of the activities that are required to
                                                              be performed for any of the key process
                                                              areas.




                                        Page 69                                                GAO/AIMD-97-47 Air Traffic Control
                                             Chapter 5
                                             Project Office Management




Table 5.4: Project Office Management Findings for VSCS
                                            Voice Switching and Control System
                    Key Practice                                         Finding                                            Rating
Commitment 1        The acquisition organization has a written policy FAA Order 1810.1F was cited as the written          Weakness
                    for execution of the software project.            policy, but it does not contain policy for software
                                                                      project execution.
Commitment 2        Performance, cost, and schedule baselines are        Performance, cost, and schedule baselines are      Weakness
                    supported.                                           not supported.
Ability 1           A project team that is responsible for performing The product team is responsible for performing        Strength
                    the project’s software acquisition management software acquisition management activities.
                    activities exists.
Ability 2           Adequate resources for the project team and          No mechanism exists for identifying the            Weakness
                    matrix support organization(s) are provided for      resources required and for ensuring that the
                    the duration of the software acquisition project.    needed resources are provided to the project.
Ability 3           The project manager is permitted to alter either     The product team leader does not have the           Weakness
                    the performance, cost, or schedule software          flexibility to alter cost, performance, or schedule
                    acquisition baseline while maintaining the other     while maintaining the other two.
                    two constant.
Ability 4           The project team and matrix support               The acquisition organization has no guidance          Observation
                    individual(s) have experience or receive training regarding training or experience requirements
                    in project office software acquisition            for project participation.
                    management activities.
Activity 1          The project team documents its plans for             The Product Team Plan and the Contract             Strength
                    software acquisition management activities.          Management Plan document acquisition
                                                                         activities.
Activity 2          The project team is organized to accomplish the The product team is organized to accomplish             Strength
                    project’s objectives.                           the project’s objectives.
Activity 3          The software acquisition activities of the project   While officials stated that software activities of Weakness
                    team are directed to accomplish the project’s        the team are directed to accomplish the
                    objectives.                                          project’s objectives, documents provided do not
                                                                         specify the activities that the team members
                                                                         must accomplish.
Activity 4          The software acquisition activities of the project   The software acquisition activities are controlled. Strength
                    team are controlled.
Activity 5          Measurements are used to track project status,       Measurements are used to track project status,     Strength
                    execution, and funding expenditures.                 execution, and funding expenditures.
Measurement 1       Measurements are made and used to determine No internal process measurements are taken or Weakness
                    the status of the project office management used to determine the status of activities for any
                    activities.                                 of the key process areas.
Verification 1      The activities for project office management are     While the Integrated Product Team leader           Weakness
                    reviewed by acquisition organization                 reviews the status of the contract and the
                    management on a periodic basis.                      contractor’s cost and schedule, he does not
                                                                         review the status of the activities that are
                                                                         required to be performed for any of the key
                                                                         process areas.
Verification 2      The activities for project office management are     While the product team leader reviews the          Weakness
                    reviewed by the project manager on both a            status of the contract and the contractor’s cost
                    periodic and event-driven basis.                     and schedule, he does not review the status of
                                                                         the activities that are required to be performed
                                                                         for any of the key process areas.

                                             Page 70                                                  GAO/AIMD-97-47 Air Traffic Control
                                            Chapter 5
                                            Project Office Management




Table 5.5: Project Office Management Findings for WARP
                                               Weather and Radar Processor
                   Key Practice                                         Finding                                              Rating
Commitment 1       The acquisition organization has a written policy FAA Order 1810.1F was cited as the written          Weakness
                   for execution of the software project.            policy, but it does not contain policy for software
                                                                     project execution.
Commitment 2       Performance, cost, and schedule baselines are        Performance, cost, and schedule baselines are        Strength
                   supported.                                           generated and supported.
Ability 1          A project team that is responsible for performing The product team is responsible for performing          Strength
                   the project’s software acquisition management software acquisition management activities.
                   activities exists.
Ability 2          Adequate resources for the project team and          No mechanism exists for identifying and for          Weakness
                   matrix support organization(s) are provided for      ensuring that the needed resources are
                   the duration of the software acquisition project.    provided to the project.
Ability 3          The project manager is permitted to alter either     The acquisition baseline process does not allow Weakness
                   the performance, cost, or schedule software          the project manager to alter the baseline.
                   acquisition baseline while maintaining the other
                   two constant.
Ability 4          The project team and matrix support               The acquisition organization has no guidance            Observation
                   individual(s) have experience or receive training regarding training or experience requirements
                   in project office software acquisition            for project participation.
                   management activities.
Activity 1         The project team documents its plans for             The product team documents its plans for             Strength
                   software acquisition management activities.          software acquisition management activities.
Activity 2         The project team is organized to accomplish the The product team is organized to accomplish               Strength
                   project’s objectives.                           the project’s objectives.
Activity 3         The software acquisition activities of the project   The software acquisition activities of the project   Strength
                   team are directed to accomplish the project’s        team are directed to accomplish the project’s
                   objectives.                                          objectives.
Activity 4         The software acquisition activities of the project   No individual is controlling the software            Weakness
                   team are controlled.                                 acquisition activities of the product team.
Activity 5         Measurements are used to track project status,       Measurements are used to track project status,       Strength
                   execution, and funding expenditures.                 execution, and funding status.
Measurement 1      Measurements are made and used to determine No internal process measurements are taken         Weakness
                   the status of the project office management and used to determine the status of activities for
                   activities.                                 any of the key process areas.
Verification 1     The activities for project office management are     While the Integrated Product Team leader             Weakness
                   reviewed by acquisition organization                 reviews the status of the contract and the
                   management on a periodic basis.                      contractor’s cost and schedule, he does not
                                                                        review the status of the activities that are
                                                                        required to be performed for any of the key
                                                                        process areas.
Verification 2     The activities for project office management are     While the product team leader reviews the            Weakness
                   reviewed by the project manager on both a            status of the contract and the contractor’s cost
                   periodic and event-driven basis.                     and schedule, he does not review the status of
                                                                        the activities that are required to be performed
                                                                        for any of the key process areas.




                                            Page 71                                                  GAO/AIMD-97-47 Air Traffic Control
              Chapter 5
              Project Office Management




              Numerous ad hoc project office management practices, including a
Conclusions   pervasive lack of measurement, analysis, and verification of project status
              and progress, are limiting FAA’s ability to meet ATC modernization project
              commitments. More discipline and definition in this KPA is needed before
              ATC modernization teams can consistently repeat project successes.




              Page 72                                       GAO/AIMD-97-47 Air Traffic Control
Chapter 6

Contract Tracking and Oversight


                     The purpose of contract tracking and oversight is to ensure that (1) the
                     software development contractor performs according to the terms of the
                     contract; (2) needed contract changes are identified, negotiated, and
                     incorporated into the contract; and (3) contractor performance issues are
                     identified early, when they are easier to address. According to the SA-CMM,
                     a repeatable contract tracking and oversight process, among other things,
                     includes (1) having a written organizational policy for contract tracking
                     and oversight, (2) having a documented plan for contract tracking and
                     oversight, (3) conducting tracking and oversight activities in accordance
                     with the plan, and (4) ensuring that individuals performing contract
                     tracking and oversight are suitably experienced or trained.


                     Our past work on ATC modernization projects has raised concerns about
FAA Lacks an         contract tracking and oversight. For example, in 1994 we reported that FAA
Effective Contract   did not provide adequate oversight of its contractor during the initial
Tracking and         development of the ISSS.1 As a result, development problems and lack of
                     progress were not always recognized in a timely manner. The results of
Oversight Process    this software capability evaluation indicate that these problems persist
                     and pinpoint the underlying contract tracking and oversight weaknesses.
                     For example, FAA does not have a written organizational policy for
                     contract tracking and oversight, and most teams have no documented plan
                     for contract tracking and oversight activities. Furthermore, the team that
                     has a plan does not always follow the plan, and none of the teams ensure
                     that persons responsible for managing software contracts have suitable
                     experience or training. As a result, the product teams cannot formulate an
                     independent assessment of contract progress and are forced to rely on
                     data provided by the contractor. Since contractor reports do not always
                     identify problems expeditiously, FAA is not always positioned to correct
                     them promptly.

                     Figure 6.1 provides a comprehensive listing of the five projects’ strengths,
                     weaknesses, and observations for the contractor tracking and oversight
                     KPA. The specific findings supporting the practice ratings cited in figure 6.1
                     are in tables 6.1 through 6.5.




                     1
                     Advanced Automation System: Implications of Problems and Recent Changes (GAO/T-RCED-94-188,
                     Apr. 13, 1994).



                     Page 73                                                 GAO/AIMD-97-47 Air Traffic Control
                                                    Chapter 6
                                                    Contract Tracking and Oversight




Figure 6.1: Contract Tracking and Oversight Summary


                   Key Practice                                               ARTSIIIE         DSR         NIMS          VSCS         WARP

                   The acquisition organization has a written policy for
   Commitment 1    the contract tracking and oversight of the contracted      Weakness       Weakness     Not rated    Weakness      Weakness
                   software effort.


   Commitment 2    Responsibility for the contract tracking and oversight      Strength      Weakness     Not rated     Strength      Strength
                   activities is designated.


                   The project team is supported by contracting
   Commitment 3                                                                Strength      Strength     Not rated     Strength      Strength
                   specialists in the execution of the contract.


                    A group that is responsible for managing contract
     Ability 1                                                                Weakness       Strength     Not rated     Strength      Strength
                   tracking and oversight activities exists.


                   Adequate resources are provided for contract
     Ability 2     tracking and oversight activities.                         Weakness      Weakness      Not rated    Weakness      Weakness


                   Individuals performing contract tracking and oversight
     Ability 3                                                                Observation   Observation   Not rated   Observation   Observation
                   activities have experience or receive training.


                   The project team documents its plans for contract
     Activity 1                                                               Weakness       Weakness     Not rated     Strength     Weakness
                   tracking and oversight activities.


                   The project's contract tracking and oversight activities    Weakness      Weakness     Not rated    Weakness      Weakness
     Activity 2
                   are performed in accordance with its plans.

                   The project team reviews required contractor software
     Activity 3    planning documents which, when satisfactory, are           Observation    Strength     Not rated    Weakness       Strength
                   made part of the contractor's baseline.

                   The project team, with end user input, conducts
     Activity 4                                                                Strength     Observation   Not rated     Strength      Strength
                   periodic reviews and interchanges with the contractor.

                   The project team tracks the contractor's development
     Activity 5    of the software engineering environment required to         Strength      Strength     Not rated     Strength      Strength
                   support the software.
                   Any problems or issues found by the project team
     Activity 6    during contract tracking and oversight are recorded         Strength      Strength     Not rated     Strength      Strength
                   in the appropriate corrective action system and tracked
                   to closure.

     Activity 7    The project team maintains the integrity of the contract
                   throughout the contract performance period.                Weakness       Strength     Not rated    Weakness       Strength




                                                    Page 74                                                   GAO/AIMD-97-47 Air Traffic Control
                                                    Chapter 6
                                                    Contract Tracking and Oversight




                  Key Practice                                                  ARTSIIIE             DSR                NIMS               VSCS          WARP

                  Measurements are made and used to determine the
Measurement 1                                                                   Weakness          Weakness            Not rated          Weakness       Weakness
                  status of the contract tracking and oversight activities.

                  The activities for contract tracking and oversight are
 Verification 1   reviewed by acquisition organization management               Weakness          Weakness            Not rated          Weakness       Weakness
                  on a periodic basis.

                  The activities for contract tracking and oversight are
 Verification 2   reviewed by the project manager on both a periodic            Weakness          Weakness            Not rated         Weakness        Weakness
                  and event-driven basis.


                                                              = Weakness      Key practice not implemented
                                                              = Strength      Key practice effectively implemented
                                                              = Observation Key practice evaluated but evidence inconclusive. Cannot characterize as either strength or
                                                                            weakness
                                                              = Not rated     Key practice not currently relevant to project, therefore not evaluated




                                                    Page 75                                                                GAO/AIMD-97-47 Air Traffic Control
                                            Chapter 6
                                            Contract Tracking and Oversight




Table 6.1: Contract Tracking and Oversight Findings for ARTSIIIE
                                            Automated Radar Terminal System IIIE
                     Key Practice                                   Finding                                        Rating
Commitment 1         The acquisition organization has a written     FAA Order 1810.1F was cited as the written Weakness
                     policy for the contract tracking and           policy, but it does not provide the policy for
                     oversight of the contracted software effort.   contract tracking and oversight.
Commitment 2         Responsibility for the contract tracking and   Responsibility for contract tracking and       Strength
                     oversight activities is designated.            oversight is designated.
Commitment 3         The project team is supported by               Contract specialists are assigned to the       Strength
                     contracting specialists in the execution of    product team.
                     the contract.
Ability 1            A group that is responsible for managing       Officials gave conflicting statements as to    Weakness
                     contract tracking and oversight activities     who has the responsibility for contract
                     exists.                                        tracking and oversight activities, and could
                                                                    not provide documentation that formally
                                                                    delegates responsibility.
Ability 2            Adequate resources are provided for            No mechanism exists for identifying          Weakness
                     contract tracking and oversight activities.    resources required and for ensuring that the
                                                                    needed resources are provided to the
                                                                    project.
Ability 3            Individuals performing contract tracking       The acquisition organization has no            Observation
                     and oversight activities have experience or    guidance regarding training or experience
                     receive training.                              requirements for project participation.
Activity 1           The project team documents its plans for       Contract tracking and oversight plans do       Weakness
                     contract tracking and oversight activities.    not exist.
Activity 2           The project’s contract tracking and            No plan exists, therefore, activities cannot   Weakness
                     oversight activities are performed in          be performed in accordance with it.
                     accordance with its plans.
Activity 3           The project team reviews required              While officials stated that the product team   Observation
                     contractor software planning documents         reviews contractor software planning
                     which, when satisfactory, are made part of     documents, they could not provide
                     the contractor’s baseline.                     documentation to support this.
Activity 4           The project team, with end user input,         Periodic reviews are held.                     Strength
                     conducts periodic reviews and
                     interchanges with the contractor.
Activity 5           The project team tracks the contractor’s       The product team tracks the development        Strength
                     development of the software engineering        of the software engineering environment.
                     environment required to support the
                     software.
Activity 6           Any problems or issues found by the            Issues found during meetings and reviews       Strength
                     project team during contract tracking and      are documented in minutes and tracked.
                     oversight are recorded in the appropriate
                     corrective action system and tracked to
                     closure.
                                                                                                                              (continued)




                                            Page 76                                                 GAO/AIMD-97-47 Air Traffic Control
                                        Chapter 6
                                        Contract Tracking and Oversight




                                         Automated Radar Terminal System IIIE
                 Key Practice                                  Finding                                         Rating
Activity 7       The project team maintains the integrity of   While product team members (including the Weakness
                 the contract throughout the contract          contracting officer) stated that the
                 performance period.                           contracting officer is responsible for
                                                               maintaining the integrity of the contract,
                                                               they could not provide any documents that
                                                               support this.
Measurement 1    Measurements are made and used to             No internal process measurements are            Weakness
                 determine the status of the contract tracking taken and used to determine the status of
                 and oversight activities.                     activities for any of the key process areas.
Verification 1   The activities for contract tracking and      While the Integrated Product Team leader     Weakness
                 oversight are reviewed by acquisition         reviews the status of the contract and the
                 organization management on a periodic         contractor’s cost and schedule, he does not
                 basis.                                        review the status of the activities that are
                                                               required to be performed for any of the key
                                                               process areas.
Verification 2   The activities for contract tracking and      While the product team leader reviews the       Weakness
                 oversight are reviewed by the project         status of the contract and the contractor’s
                 manager on both a periodic and                cost and schedule, he does not review the
                 event-driven basis.                           status of the activities that are required to
                                                               be performed for any of the key process
                                                               areas.




                                        Page 77                                                 GAO/AIMD-97-47 Air Traffic Control
                                           Chapter 6
                                           Contract Tracking and Oversight




Table 6.2: Contract Tracking and Oversight Findings for DSR
                                                Display System Replacement
                    Key Practice                                   Finding                                       Rating
Commitment 1        The acquisition organization has a written     FAA Order 1810.1F was cited as the written Weakness
                    policy for the contract tracking and           policy, but it does not provide the policy for
                    oversight of the contracted software effort.   contract tracking and oversight.
Commitment 2        Responsibility for the contract tracking and   Officials gave conflicting answers on who is Weakness
                    oversight activities is designated.            responsible for contract tracking and
                                                                   oversight, and they could not provide
                                                                   documentation that formally delegates
                                                                   responsibility.
Commitment 3        The project team is supported by               The product team is supported by a            Strength
                    contracting specialists in the execution of    contracting specialist in execution of the
                    the contract.                                  contract.
Ability 1           A group that is responsible for managing       The product team is responsible for           Strength
                    contract tracking and oversight activities     managing contract tracking and oversight
                    exists.                                        activities.
Ability 2           Adequate resources are provided for            No mechanism exists for identifying          Weakness
                    contract tracking and oversight activities.    resources required and for ensuring that the
                                                                   needed resources are provided to the
                                                                   project.
Ability 3           Individuals performing contract tracking       The acquisition organization has no           Observation
                    and oversight activities have experience or    guidance regarding training or experience
                    receive training.                              requirements for project participation.
Activity 1          The project team documents its plans for       There are no written plans for contract       Weakness
                    contract tracking and oversight activities.    tracking and oversight activities.
Activity 2          The project’s contract tracking and            No plan exists, therefore, the product team   Weakness
                    oversight activities are performed in          cannot perform in accordance with it.
                    accordance with its plans.
Activity 3          The project team reviews required              Reviews are used to approve contract          Strength
                    contractor software planning documents         planning documents, which, when
                    which, when satisfactory, are made part of     satisfactory, are made part of the
                    the contractor’s baseline.                     contractor’s baseline.
Activity 4          The project team, with end user input,         While it was stated that continuous           Observation
                    conducts periodic reviews and                  interactions with the contractor are held,
                    interchanges with the contractor.              officials could provide no documents to
                                                                   support this.
Activity 5          The project team tracks the contractor’s       The product team tracks the contractor’s      Strength
                    development of the software engineering        development of the software engineering
                    environment required to support the            environment required to support the
                    software.                                      software.
Activity 6          Any problems or issues found by the            The product team records problems and         Strength
                    project team during contract tracking and      issues found during contract tracking and
                    oversight are recorded in the appropriate      oversight and tracks them to closure.
                    corrective action system and tracked to
                    closure.
                                                                                                                            (continued)




                                           Page 78                                                 GAO/AIMD-97-47 Air Traffic Control
                                        Chapter 6
                                        Contract Tracking and Oversight




                                              Display System Replacement
                 Key Practice                                  Finding                                         Rating
Activity 7       The project team maintains the integrity of   The product team maintains the integrity of     Strength
                 the contract throughout the contract          the contract throughout the contract
                 performance period.                           performance period.
Measurement 1    Measurements are made and used to             No internal process measurements are            Weakness
                 determine the status of the contract tracking taken and used to determine the status of
                 and oversight activities.                     activities for any of the key process areas.
Verification 1   The activities for contract tracking and      While the Integrated Product Team leader     Weakness
                 oversight are reviewed by acquisition         reviews the status of the contract and the
                 organization management on a periodic         contractor’s cost and schedule, he does not
                 basis.                                        review the status of the activities that are
                                                               required to be performed for any of the key
                                                               process areas.
Verification 2   The activities for contract tracking and      While the product team leader reviews the       Weakness
                 oversight are reviewed by the project         status of the contract and the contractor’s
                 manager on both a periodic and                cost and schedule, he does not review the
                 event-driven basis.                           status of the activities that are required to
                                                               be performed for any of the key process
                                                               areas.




                                        Page 79                                                 GAO/AIMD-97-47 Air Traffic Control
                                           Chapter 6
                                           Contract Tracking and Oversight




Table 6.3: Contract Tracking and Oversight Findings for NIMS
                                           NAS Infrastructure Management System
                    Key Practice                                   Findings                                     Rating
Commitment 1        The acquisition organization has a written     Since NIMS is not under contract yet, it was Not rated
                    policy for the contract tracking and           not evaluated against this KPA.
                    oversight of the contracted software effort.
Commitment 2        Responsibility for the contract tracking and   Too early to assess: the project has not     Not rated
                    oversight activities is designated.            reached a stage where this applies.
Commitment 3        The project team is supported by               Too early to assess: the project has not     Not rated
                    contracting specialists in the execution of    reached a stage where this applies.
                    the contract.
Ability 1           A group that is responsible for managing       Too early to assess: the project has not     Not rated
                    contract tracking and oversight activities     reached a stage where this applies.
                    exists.
Ability 2           Adequate resources are provided for            Too early to assess: the project has not     Not rated
                    contract tracking and oversight activities.    reached a stage where this applies.
Ability 3           Individuals performing contract tracking       Too early to assess: the project has not     Not rated
                    and oversight activities have experience or    reached a stage where this applies.
                    receive training.
Activity 1          The project team documents its plans for       Too early to assess: the project has not     Not rated
                    contract tracking and oversight activities.    reached a stage where this applies.
Activity 2          The project’s contract tracking and            Too early to assess: the project has not     Not rated
                    oversight activities are performed in          reached a stage where this applies.
                    accordance with its plans.
Activity 3          The project team reviews required              Too early to assess: the project has not     Not rated
                    contractor software planning documents         reached a stage where this applies.
                    which, when satisfactory, are made part of
                    the contractor’s baseline.
Activity 4          The project team, with end user input,         Too early to assess: the project has not     Not rated
                    conducts periodic reviews and                  reached a stage where this applies.
                    interchanges with the contractor.
Activity 5          The project team tracks the contractor’s       Too early to assess: the project has not     Not rated
                    development of the software engineering        reached a stage where this applies.
                    environment required to support the
                    software.
Activity 6          Any problems or issues found by the            Too early to assess: the project has not     Not rated
                    project team during contract tracking and      reached a stage where this applies.
                    oversight are recorded in the appropriate
                    corrective action system and tracked to
                    closure.
Activity 7          The project team maintains the integrity of    Too early to assess: the project has not     Not rated
                    the contract throughout the contract           reached a stage where this applies.
                    performance period.
Measurement 1       Measurements are made and used to             Too early to assess: the project has not      Not rated
                    determine the status of the contract tracking reached a stage where this applies.
                    and oversight activities.
                                                                                                                         (continued)




                                           Page 80                                                GAO/AIMD-97-47 Air Traffic Control
                                        Chapter 6
                                        Contract Tracking and Oversight




                                        NAS Infrastructure Management System
                 Key Practice                                 Findings                                     Rating
Verification 1   The activities for contract tracking and     Too early to assess: the project has not     Not rated
                 oversight are reviewed by acquisition        reached a stage where this applies.
                 organization management on a periodic
                 basis.
Verification 2   The activities for contract tracking and     Too early to assess: the project has not     Not rated
                 oversight are reviewed by the project        reached a stage where this applies.
                 manager on both a periodic and
                 event-driven basis.




                                        Page 81                                              GAO/AIMD-97-47 Air Traffic Control
                                            Chapter 6
                                            Contract Tracking and Oversight




Table 6.4: Contract Tracking and Oversight Findings for VSCS
                                             Voice Switching and Control System
                     Key Practice                                   Finding                                        Rating
Commitment 1         The acquisition organization has a written     There is no policy on contract tracking and    Weakness
                     policy for the contract tracking and           oversight.
                     oversight of the contracted software effort.
Commitment 2         Responsibility for the contract tracking and   Responsibility for contract tracking and       Strength
                     oversight activities is designated.            oversight is designated.
Commitment 3         The project team is supported by               The product team is supported by a             Strength
                     contracting specialists in the execution of    contracting specialist.
                     the contract.
Ability 1            A group that is responsible for managing       A group exists that is responsible for         Strength
                     contract tracking and oversight activities     managing contract tracking and oversight.
                     exists.
Ability 2            Adequate resources are provided for            No mechanism exists for identifying the      Weakness
                     contract tracking and oversight activities.    resources required and for ensuring that the
                                                                    needed resources are provided to the
                                                                    project.
Ability 3            Individuals performing contract tracking       The acquisition organization has no            Observation
                     and oversight activities have experience or    guidance regarding training or experience
                     receive training.                              requirements for project participation.
Activity 1           The project team documents its plans for       The product team has documented its            Strength
                     contract tracking and oversight activities.    plans.
Activity 2           The project’s contract tracking and            The product team could not provide           Weakness
                     oversight activities are performed in          evidence that shows its contracting tracking
                     accordance with its plans.                     and oversight activities are performed in
                                                                    accordance with its plans.
Activity 3           The project team reviews required              While reviews are conducted, officials could Weakness
                     contractor software planning documents         not provide documentation that shows the
                     which, when satisfactory, are made part of     results are made part of the contractor’s
                     the contractor’s baseline.                     baseline.
Activity 4           The project team, with end user input,         The product team conducts periodic             Strength
                     conducts periodic reviews and                  reviews and interchanges with the
                     interchanges with the contractor.              contractor.
Activity 5           The project team tracks the contractor’s       The contractor is required to provide a list   Strength
                     development of the software engineering        of common tools and support equipment,
                     environment required to support the            which the product team uses to track the
                     software.                                      software engineering environment
                                                                    development.
Activity 6           Any problems or issues found by the            The contractor has an extensive software       Strength
                     project team during contract tracking and      development environment and problems or
                     oversight are recorded in the appropriate      issues are tracked to closure.
                     corrective action system and tracked to
                     closure.
Activity 7           The project team maintains the integrity of    There is no evidence that either the           Weakness
                     the contract throughout the contract           contracting officer or the product team are
                     performance period.                            following the process and maintaining the
                                                                    integrity of the contract.
                                                                                                                              (continued)


                                            Page 82                                                 GAO/AIMD-97-47 Air Traffic Control
                                        Chapter 6
                                        Contract Tracking and Oversight




                                          Voice Switching and Control System
                 Key Practice                                  Finding                                         Rating
Measurement 1    Measurements are made and used to             No internal process measurements are            Weakness
                 determine the status of the contract tracking taken or used to determine the status of
                 and oversight activities.                     activities for any of the key process areas.
Verification 1   The activities for contract tracking and      While the Integrated Product Team leader     Weakness
                 oversight are reviewed by acquisition         reviews the status of the contract and the
                 organization management on a periodic         contractor’s cost and schedule, he does not
                 basis.                                        review the status of the activities that are
                                                               required to be performed for any of the key
                                                               process areas.
Verification 2   The activities for contract tracking and      While the product team leader reviews the       Weakness
                 oversight are reviewed by the project         status of the contract and the contractor’s
                 manager on both a periodic and                cost and schedule, he does not review the
                 event-driven basis.                           status of the activities that are required to
                                                               be performed for any of the key process
                                                               areas.




                                        Page 83                                                 GAO/AIMD-97-47 Air Traffic Control
                                            Chapter 6
                                            Contract Tracking and Oversight




Table 6.5: Contract Tracking and Oversight Findings for WARP
                                                Weather and Radar Processor
                     Key Practice                                   Finding                                        Rating
Commitment 1         The acquisition organization has a written     There is no written policy for contract        Weakness
                     policy for the contract tracking and           tracking and oversight activities.
                     oversight of the contracted software effort.
Commitment 2         Responsibility for the contract tracking and   The product team is responsible for            Strength
                     oversight activities is designated.            contract tracking and oversight activities.
Commitment 3         The project team is supported by               The product team is supported by               Strength
                     contracting specialists in the execution of    contracting specialists.
                     the contract.
Ability 1            A group that is responsible for managing       The product team is collectively responsible Strength
                     contract tracking and oversight activities     for contract tracking and oversight activities.
                     exists.
Ability 2            Adequate resources are provided for            No mechanism exists for identifying and for    Weakness
                     contract tracking and oversight activities.    ensuring that the needed resources are
                                                                    provided to the project.
Ability 3            Individuals performing contract tracking       The acquisition organization has no            Observation
                     and oversight activities have experience or    guidance regarding training or experience
                     receive training.                              requirements for project participation.
Activity 1           The project team documents its plans for       Plans for contract tracking and oversight      Weakness
                     contract tracking and oversight activities.    activities are not documented.
Activity 2           The project’s contract tracking and            Since there is no contract tracking and        Weakness
                     oversight activities are performed in          oversight plan, there is no way to assess
                     accordance with its plans.                     whether activities are being performed in
                                                                    accordance with the plan.
Activity 3           The project team reviews required              The product team reviews required              Strength
                     contractor software planning documents         contractor software planning documents
                     which, when satisfactory, are made part of     which, when satisfactory, are made part of
                     the contractor’s baseline.                     the contractor’s baseline.
Activity 4           The project team, with end user input,         The product team conducts periodic             Strength
                     conducts periodic reviews and                  reviews and interchanges with the
                     interchanges with the contractor.              contractor.
Activity 5           The project team tracks the contractor’s       As a deliverable, the status of the software   Strength
                     development of the software engineering        support environment development is
                     environment required to support the            reviewed and tracked.
                     software.
Activity 6           Any problems or issues found by the            The contracting officer’s technical            Strength
                     project team during contract tracking and      representative is responsible for managing
                     oversight are recorded in the appropriate      and tracking action items, and these are
                     corrective action system and tracked to        recorded in an appropriate correction
                     closure.                                       system.
Activity 7           The project team maintains the integrity of    The contracting officer maintains the          Strength
                     the contract throughout the contract           integrity of the contract and is responsible
                     performance period.                            for doing so throughout the contract
                                                                    performance period.
                                                                                                                              (continued)




                                            Page 84                                                 GAO/AIMD-97-47 Air Traffic Control
                                        Chapter 6
                                        Contract Tracking and Oversight




                                              Weather and Radar Processor
                 Key Practice                                  Finding                                         Rating
Measurement 1    Measurements are made and used to             No internal process measurements are            Weakness
                 determine the status of the contract tracking taken and used to determine the status of
                 and oversight activities.                     activities for any of the key process areas.
Verification 1   The activities for contract tracking and      While the Integrated Product Team leader     Weakness
                 oversight are reviewed by acquisition         reviews the status of the contract and the
                 organization management on a periodic         contractor’s cost and schedule, he does not
                 basis.                                        review the status of the activities that are
                                                               required to be performed for any of the key
                                                               process areas.
Verification 2   The activities for contract tracking and      While the product team leader reviews the       Weakness
                 oversight are reviewed by the project         status of the contract and the contractor’s
                 manager on both a periodic and                cost and schedule, he does not review the
                 event-driven basis.                           status of the activities that are required to
                                                               be performed for any of the key process
                                                               areas.


                                        To effectively and efficiently acquire software, FAA must have a
Conclusions                             well-defined and enforced process that provides for proactive tracking and
                                        oversight of its software development contractors. FAA’s current process
                                        for ATC modernization contractor tracking and oversight is ad hoc and
                                        reactive, thereby increasing the chances of its ATC software acquisitions
                                        being late, costing more than expected, and not performing as intended.




                                        Page 85                                                 GAO/AIMD-97-47 Air Traffic Control
Chapter 6
Contract Tracking and Oversight




Page 86                           GAO/AIMD-97-47 Air Traffic Control
Chapter 7

Evaluation


                         The purpose of evaluation, or testing, is to determine that the acquired
                         software products and services satisfy contract requirements prior to
                         acceptance. According to the SA-CMM, a repeatable evaluation process
                         includes (1) documenting evaluation plans and conducting evaluation
                         activities in accordance with the plan, (2) developing and managing
                         evaluation requirements in conjunction with developing software technical
                         requirements, (3) incorporating evaluation requirements into the
                         solicitation and the resulting contract, (4) tracking contractor
                         performance of evaluation activities for compliance with the contract,
                         (5) ensuring that adequate resources are provided for evaluation activities,
                         and (6) measuring and reporting on the status of evaluation activities to
                         management.


                         All of the projects were strong in many evaluation practice areas. For
FAA Is Strong in Most    example, all rated projects have documented test and evaluation plans and
but Not All Evaluation   conduct test and evaluation activities in accordance with the plans. In
KPA Practices            addition, most teams develop evaluation requirements for
                         contractor-conducted software tests concurrent with developing software
                         technical requirements, and all teams incorporate evaluation requirements
                         into the solicitation and resulting contract. Also, most teams track
                         contractor performance of test activities for compliance with the contract.

                         Despite these many strengths, several weaknesses prevented FAA from
                         meeting this KPA. For example, only one of the teams ensures that
                         adequate resources are provided for evaluation activities. Additionally,
                         none of the teams measure and report on the status of all evaluation
                         activities to management. As a result, management does not have a
                         complete and accurate picture of evaluation status and progress, which
                         could impair decision-making.

                         Figure 7.1 provides a comprehensive listing of the five projects’ strengths,
                         weaknesses, and observations for the evaluation KPA. The specific findings
                         supporting the practice ratings cited in figure 7.1 are in tables 7.1 through
                         7.5.




                         Page 87                                        GAO/AIMD-97-47 Air Traffic Control
                                                       Chapter 7
                                                       Evaluation




Figure 7.1: Evaluation Summary


                     Key Practice                                              ARTSIIIE        DSR          NIMS          VSCS          WARP
                     The acquisition organization has a written policy for
   Commitment 1      managing the evaluation of the acquired software          Strength      Strength      Strength      Strength      Strength
                     products and services.


   Commitment 2      Responsibility for evaluation activities is clearly       Strength      Strength     Observation    Strength      Strength
                     defined.

                     A group that is responsible for planning, managing,
      Ability 1      and performing evaluation activities for the project      Strength      Strength      Strength      Strength      Strength
                     exists.


      Ability 2      Adequate resources are provided for evaluation            Weakness      Strength     Weakness      Weakness      Weakness
                     activities.


      Ability 3      Individuals performing evaluation activities have        Observation   Observation   Observation   Observation   Observation
                     experience or receive training.

                     Members of the project team and groups supporting
      Ability 4      the software acquisition receive orientation on the       Weakness      Strength     Weakness       Strength     Observation
                     objectives of the evaluation approach.


      Activity 1     The project team documents its plans for evaluation       Strength      Strength      Strength      Strength      Strength
                     activities.


      Activity 2     The project's evaluation activities are performed in      Strength      Strength      Not rated     Strength      Not rated
                     accordance with its plans.

                     Evaluation requirements are developed and
      Activity 3     managed in conjunction with development of the            Strength      Strength     Observation    Strength      Strength
                     system or software technical requirements.


      Activity 4     The evaluation requirements are incorporated into the     Strength      Strength      Strength      Strength      Strength
                     solicitation and resulting contract.

                     The project team tracks contractor's performance in
      Activity 5     terms of evaluation requirements for compliance with      Strength      Strength      Not rated     Strength      Not rated
                     the contract.

                     Planned evaluations are performed on the acquired
      Activity 6     software products and services prior to acceptance for    Strength      Strength      Not rated     Strength      Not rated
                     operational use.

                     Results of the evaluations are analyzed and
      Activity 7                                                               Strength      Strength      Not rated     Strength      Not rated
                     compared to the contract's requirements to establish
                     a basis for acceptance.


   Measurement 1     Measurements are made and used to determine the          Weakness      Weakness      Weakness      Weakness      Weakness
                     status of the evaluation activities.

                     The activities for evaluation are reviewed with
    Verification 1   acquisition organization management on a periodic        Weakness      Weakness      Weakness      Weakness      Weakness
                     basis.




                                                       Page 88                                                 GAO/AIMD-97-47 Air Traffic Control
                                               Chapter 7
                                               Evaluation




                 Key Practice                                             ARTSIIIE             DSR                NIMS               VSCS          WARP

                 The activities for evaluation are reviewed with the
Verification 2   project manager on both a periodic and event-driven      Weakness           Weakness           Weakness           Weakness       Weakness
                 basis.


                                                        = Weakness      Key practice not implemented
                                                        = Strength      Key practice effectively implemented
                                                        = Observation Key practice evaluated but evidence inconclusive. Cannot characterize as either strength or
                                                                      weakness

                                                        = Not rated     Key practice not currently relevant to project, therefore not evaluated




                                               Page 89                                                               GAO/AIMD-97-47 Air Traffic Control
                                              Chapter 7
                                              Evaluation




Table 7.1: Evaluation Findings for ARTSIIIE
                                               Automated Radar Terminal System IIIE
                      Key Practice                                   Finding                                         Rating
Commitment 1          The acquisition organization has a written     FAA Order 1810.4B is the written policy.        Strength
                      policy for managing the evaluation of the
                      acquired software products and services.
Commitment 2          Responsibility for evaluation activities is    Evaluation responsibility is clearly defined.   Strength
                      clearly defined.
Ability 1             A group that is responsible for planning,      The product team is responsible for             Strength
                      managing, and performing evaluation            evaluation activities.
                      activities for the project exists.
Ability 2             Adequate resources are provided for            No mechanism exists for identifying          Weakness
                      evaluation activities.                         resources required and for ensuring that the
                                                                     needed resources are provided to the
                                                                     project.
Ability 3             Individuals performing evaluation activities   The acquisition organization has no             Observation
                      have experience or receive training.           guidance regarding training or experience
                                                                     requirements for project participation.
Ability 4             Members of the project team and groups         The product team did not receive                Weakness
                      supporting the software acquisition receive    orientation on the evaluation approach.
                      orientation on the objectives of the
                      evaluation approach.
Activity 1            The project team documents its plans for       The Test and Evaluation Master Plan             Strength
                      evaluation activities.                         delineates all evaluation activities.
Activity 2            The project’s evaluation activities are        Evaluation activities are performed in          Strength
                      performed in accordance with its plans.        accordance with plans.
Activity 3            Evaluation requirements are developed and      Evaluation requirements are developed and Strength
                      managed in conjunction with development        managed in conjunction with development
                      of the system or software technical            of the system or software technical
                      requirements.                                  requirements.
Activity 4            The evaluation requirements are                Evaluation requirements are part of the         Strength
                      incorporated into the solicitation and         contract.
                      resulting contract.
Activity 5            The project team tracks contractor’s           The evaluation activities performed by the      Strength
                      performance in terms of evaluation             contractor are tracked.
                      requirements for compliance with the
                      contract.
Activity 6            Planned evaluations are performed on the       Planned evaluations are performed to            Strength
                      acquired software products and services        ensure that technical and contract
                      prior to acceptance for operational use.       requirements are met prior to acceptance.
Activity 7            Results of the evaluations are analyzed and Technical and contract requirements are            Strength
                      compared to the contract’s requirements to met prior to acceptance.
                      establish a basis for acceptance.
Measurement 1         Measurements are made and used to              No internal process measurements are            Weakness
                      determine the status of the evaluation         taken and used to determine the status of
                      activities.                                    activities for any of the key process areas.
                                                                                                                                (continued)




                                              Page 90                                                 GAO/AIMD-97-47 Air Traffic Control
                                       Chapter 7
                                       Evaluation




                                        Automated Radar Terminal System IIIE
                 Key Practice                                  Finding                                       Rating
Verification 1   The activities for evaluation are reviewed    While the Integrated Product Team leader     Weakness
                 with acquisition organization management      reviews the status of the contract and the
                 on a periodic basis.                          contractor’s cost and schedule, he does not
                                                               review the status of the activities that are
                                                               required to be performed for any of the key
                                                               process areas.
Verification 2   The activities for evaluation are reviewed  While the product team leader reviews the       Weakness
                 with the project manager on both a periodic status of the contract and the contractor’s
                 and event-driven basis.                     cost and schedule, he does not review the
                                                             status of the activities that are required to
                                                             be performed for any of the key process
                                                             areas.




                                       Page 91                                                GAO/AIMD-97-47 Air Traffic Control
                                             Chapter 7
                                             Evaluation




Table 7.2: Evaluation Findings for DSR
                                                    Display System Replacement
                     Key Practice                                   Finding                                        Rating
Commitment 1         The acquisition organization has a written     FAA Order 1810.4B is the written policy.       Strength
                     policy for managing the evaluation of the
                     acquired software products and services.
Commitment 2         Responsibility for evaluation activities is    The product team leader is responsible for     Strength
                     clearly defined.                               evaluation activities.
Ability 1            A group that is responsible for planning,      The test and maintenance leader, along         Strength
                     managing, and performing evaluation            with the product team, are responsible for
                     activities for the project exists.             planning, managing, and performing
                                                                    evaluation activities.
Ability 2            Adequate resources are provided for            Adequate resources are provided for            Strength
                     evaluation activities.                         evaluation activities.
Ability 3            Individuals performing evaluation activities   The acquisition organization has no            Observation
                     have experience or receive training.           guidance regarding training or experience
                                                                    requirements for project participation.
Ability 4            Members of the project team and groups         Members of the project team received           Strength
                     supporting the software acquisition receive    orientation on the evaluation approach.
                     orientation on the objectives of the
                     evaluation approach.
Activity 1           The project team documents its plans for       The product team documents its plans for       Strength
                     evaluation activities.                         evaluation activities.
Activity 2           The project’s evaluation activities are        The project’s evaluation activities are        Strength
                     performed in accordance with its plans.        performed in accordance with its plans.
Activity 3           Evaluation requirements are developed and Evaluation requirements are developed in            Strength
                     managed in conjunction with development conjunction with the system requirements.
                     of the system or software technical
                     requirements.
Activity 4           The evaluation requirements are                The evaluation requirements are                Strength
                     incorporated into the solicitation and         incorporated into the contract.
                     resulting contract.
Activity 5           The project team tracks contractor’s           The product team tracks the contractor’s       Strength
                     performance in terms of evaluation             performance, in terms of evaluation
                     requirements for compliance with the           requirements, using the A-level and B-level
                     contract.                                      specifications in the contract as criteria.
Activity 6           Planned evaluations are performed on the       Planned evaluations are performed on the       Strength
                     acquired software products and services        acquired software products and services
                     prior to acceptance for operational use.       prior to acceptance for operational use.
Activity 7           Results of the evaluations are analyzed and Results of evaluations are analyzed and           Strength
                     compared to the contract’s requirements to compared to the contract’s requirements to
                     establish a basis for acceptance.           establish a basis for acceptance.
Measurement 1        Measurements are made and used to              No internal process measurements are           Weakness
                     determine the status of the evaluation         taken and used to determine the status of
                     activities.                                    activities for any of the key process areas.
                                                                                                                              (continued)




                                             Page 92                                                GAO/AIMD-97-47 Air Traffic Control
                                       Chapter 7
                                       Evaluation




                                             Display System Replacement
                 Key Practice                                  Finding                                       Rating
Verification 1   The activities for evaluation are reviewed    While the Integrated Product Team leader     Weakness
                 with acquisition organization management      reviews the status of the contract and the
                 on a periodic basis.                          contractor’s cost and schedule, he does not
                                                               review the status of the activities that are
                                                               required to be performed for any of the key
                                                               process areas.
Verification 2   The activities for evaluation are reviewed  While the product team leader reviews the       Weakness
                 with the project manager on both a periodic status of the contract and the contractor’s
                 and event-driven basis.                     cost and schedule, he does not review the
                                                             status of the activities that are required to
                                                             be performed for any of the key process
                                                             areas.




                                       Page 93                                                GAO/AIMD-97-47 Air Traffic Control
                                             Chapter 7
                                             Evaluation




Table 7.3: Evaluation Findings for NIMS
                                             NAS Infrastructure Management System
                     Key Practice                                   Finding                                         Rating
Commitment 1         The acquisition organization has a written     The Acquisition Management System is the        Strength
                     policy for managing the evaluation of the      written policy.
                     acquired software products and services.
Commitment 2         Responsibility for evaluation activities is    Officials stated that the test leader is        Observation
                     clearly defined.                               responsible for evaluation activities;
                                                                    however, they could not provide
                                                                    documentation to support this.
Ability 1            A group that is responsible for planning,      The product team has responsibility for         Strength
                     managing, and performing evaluation            planning, managing, and performing
                     activities for the project exists.             evaluation activities.
Ability 2            Adequate resources are provided for            No mechanism exists for identifying          Weakness
                     evaluation activities.                         resources required and for ensuring that the
                                                                    needed resources are provided to the
                                                                    project.
Ability 3            Individuals performing evaluation activities   The acquisition organization has no             Observation
                     have experience or receive training.           guidance regarding training or experience
                                                                    requirements for project participation.
Ability 4            Members of the project team and groups         Orientation on the objectives of the            Weakness
                     supporting the software acquisition receive    evaluation approach was not received.
                     orientation on the objectives of the
                     evaluation approach.
Activity 1           The project team documents its plans for       The project team documents its plans for        Strength
                     evaluation activities.                         evaluation activities.
Activity 2           The project’s evaluation activities are        NIMS has not yet reached the stage where        Not rated
                     performed in accordance with its plans.        evaluation activities are being performed.
Activity 3           Evaluation requirements are developed and Too early to assess: system software                 Observation
                     managed in conjunction with development requirements are not developed sufficiently
                     of the system or software technical       to develop evaluation requirements.
                     requirements.
Activity 4           The evaluation requirements are                The evaluation requirements are                Strength
                     incorporated into the solicitation and         incorporated into the solicitation through the
                     resulting contract.                            statement of work, which will be part of the
                                                                    contract.
Activity 5           The project team tracks contractor’s           NIMS has not yet reached the stage where        Not rated
                     performance in terms of evaluation             evaluation activities are being performed.
                     requirements for compliance with the
                     contract.
Activity 6           Planned evaluations are performed on the       NIMS has not yet reached the stage where        Not rated
                     acquired software products and services        operational evaluation activities are being
                     prior to acceptance for operational use.       performed.
Activity 7           Results of the evaluations are analyzed and NIMS has not yet reached the stage where           Not rated
                     compared to the contract’s requirements to evaluation activities are analyzed and
                     establish a basis for acceptance.           compared to the contract.
                                                                                                                               (continued)




                                             Page 94                                                  GAO/AIMD-97-47 Air Traffic Control
                                       Chapter 7
                                       Evaluation




                                       NAS Infrastructure Management System
                 Key Practice                                  Finding                                       Rating
Measurement 1    Measurements are made and used to             No internal process measurements are            Weakness
                 determine the status of the evaluation        taken to determine the status of activities for
                 activities.                                   any of the key process areas.
Verification 1   The activities for evaluation are reviewed    While the Integrated Product Team leader     Weakness
                 with acquisition organization management      reviews the status of the contract and the
                 on a periodic basis.                          contractor’s cost and schedule, he does not
                                                               review the status of the activities that are
                                                               required to be performed for any of the key
                                                               process areas.
Verification 2   The activities for evaluation are reviewed  While the product team leader reviews the       Weakness
                 with the project manager on both a periodic status of the contract and the contractor’s
                 and event-driven basis.                     cost and schedule, he does not review the
                                                             status of the activities that are required to
                                                             be performed for any of the key process
                                                             areas.




                                       Page 95                                                GAO/AIMD-97-47 Air Traffic Control
                                             Chapter 7
                                             Evaluation




Table 7.4: Evaluation Findings for VSCS
                                               Voice Switching and Control System
                     Key Practice                                   Finding                                        Rating
Commitment 1         The acquisition organization has a written     FAA Order 1810.4B provides policy              Strength
                     policy for managing the evaluation of the      guidance.
                     acquired software products and services.
Commitment 2         Responsibility for evaluation activities is    The Test and Evaluation Master Plan            Strength
                     clearly defined.                               defines the responsibility for evaluation.
Ability 1            A group that is responsible for planning,      A group is assigned responsibility for         Strength
                     managing, and performing evaluation            planning, managing, and performing
                     activities for the project exists.             evaluation activities.
Ability 2            Adequate resources are provided for            No mechanism exists for identifying the      Weakness
                     evaluation activities.                         resources required and for ensuring that the
                                                                    needed resources are provided to the
                                                                    project.
Ability 3            Individuals performing evaluation activities   The acquisition organization has no            Observation
                     have experience or receive training.           guidance regarding training or experience
                                                                    requirements for project participation.
Ability 4            Members of the project team and groups         An orientation was conducted.                  Strength
                     supporting the software acquisition receive
                     orientation on the objectives of the
                     evaluation approach.
Activity 1           The project team documents its plans for       Test and evaluation activities are             Strength
                     evaluation activities.                         documented.
Activity 2           The project’s evaluation activities are        Evaluation activities are performed in         Strength
                     performed in accordance with its plans.        accordance with plans.
Activity 3           Evaluation requirements are developed and Test and evaluation requirements are                Strength
                     managed in conjunction with development developed and managed in conjunction
                     of the system or software technical       with system requirements.
                     requirements.
Activity 4           The evaluation requirements are                Evaluation requirements are part of the        Strength
                     incorporated into the solicitation and         contract.
                     resulting contract.
Activity 5           The project team tracks contractor’s           The contractor’s performance in terms of       Strength
                     performance in terms of evaluation             evaluation requirements is tracked through
                     requirements for compliance with the           weekly meetings.
                     contract.
Activity 6           Planned evaluations are performed on the       Evaluation procedures are documented in     Strength
                     acquired software products and services        the Test and Evaluation Master Plan and are
                     prior to acceptance for operational use.       performed prior to acceptance.
Activity 7           Results of the evaluations are analyzed and FAA conducts factory acceptance tests and Strength
                     compared to the contract’s requirements to compares results to the contract’s
                     establish a basis for acceptance.           requirements.
Measurement 1        Measurements are made and used to              No internal process measurements are           Weakness
                     determine the status of the evaluation         taken or used to determine the status of
                     activities.                                    activities for any of the key process areas.
                                                                                                                              (continued)




                                             Page 96                                                 GAO/AIMD-97-47 Air Traffic Control
                                       Chapter 7
                                       Evaluation




                                         Voice Switching and Control System
                 Key Practice                                  Finding                                       Rating
Verification 1   The activities for evaluation are reviewed    While the Integrated Product Team leader     Weakness
                 with acquisition organization management      reviews the status of the contract and the
                 on a periodic basis.                          contractor’s cost and schedule, he does not
                                                               review the status of the activities that are
                                                               required to be performed for any of the key
                                                               process areas.
Verification 2   The activities for evaluation are reviewed  While the product team leader reviews the       Weakness
                 with the project manager on both a periodic status of the contract and the contractor’s
                 and event-driven basis.                     cost and schedule, he does not review the
                                                             status of the activities that are required to
                                                             be performed for any of the key process
                                                             areas.




                                       Page 97                                                GAO/AIMD-97-47 Air Traffic Control
                                             Chapter 7
                                             Evaluation




Table 7.5: Evaluation Findings for WARP
                                                   Weather and Radar Processor
                     Key Practice                                   Finding                                        Rating
Commitment 1         The acquisition organization has a written     FAA Order 1810.4B is the written policy.       Strength
                     policy for managing the evaluation of the
                     acquired software products and services.
Commitment 2         Responsibility for evaluation activities is    Responsibility for evaluation activities has   Strength
                     clearly defined.                               been clearly defined.
Ability 1            A group that is responsible for planning,      A group is responsible for evaluation          Strength
                     managing, and performing evaluation            activities for the project.
                     activities for the project exists.
Ability 2            Adequate resources are provided for            No mechanism exists for identifying and for    Weakness
                     evaluation activities.                         ensuring that the needed resources are
                                                                    provided to the project.
Ability 3            Individuals performing evaluation activities   The acquisition organization has no            Observation
                     have experience or receive training.           guidance regarding training or experience
                                                                    requirements for project participation.
Ability 4            Members of the project team and groups         Orientation was not needed because the         Observation
                     supporting the software acquisition receive    test leader and his team were familiar with
                     orientation on the objectives of the           the evaluation approach.
                     evaluation approach.
Activity 1           The project team documents its plans for       The Test and Evaluation Master Plan            Strength
                     evaluation activities.                         documents the team’s plan for evaluation
                                                                    activities.
Activity 2           The project’s evaluation activities are        WARP has not reached the stage where           Not rated
                     performed in accordance with its plans.        evaluation activities are being performed
                                                                    and can be compared to a plan.
Activity 3           Evaluation requirements are developed and      Evaluation requirements are developed and Strength
                     managed in conjunction with development        managed in conjunction with development
                     of the system or software technical            of system or software technical
                     requirements.                                  requirements.
Activity 4           The evaluation requirements are                The evaluation requirements are                Strength
                     incorporated into the solicitation and         incorporated into the solicitation and
                     resulting contract.                            resulting contract.
Activity 5           The project team tracks contractor’s           WARP has not reached the stage where the Not rated
                     performance in terms of evaluation             contractor is performing evaluation activities
                     requirements for compliance with the           that can be tracked for compliance with
                     contract.                                      contract.
Activity 6           Planned evaluations are performed on the       WARP has not yet reached the stage where Not rated
                     acquired software products and services        evaluations are being performed.
                     prior to acceptance for operational use.
Activity 7           Results of the evaluations are analyzed and WARP has not yet reached the stage where Not rated
                     compared to the contract’s requirements to evaluations are being performed.
                     establish a basis for acceptance.
Measurement 1        Measurements are made and used to              No internal process measurements are           Weakness
                     determine the status of the evaluation         taken and used to determine the status of
                     activities.                                    activities for any of the key process areas.
                                                                                                                              (continued)



                                             Page 98                                                 GAO/AIMD-97-47 Air Traffic Control
                                       Chapter 7
                                       Evaluation




                                             Weather and Radar Processor
                 Key Practice                                  Finding                                       Rating
Verification 1   The activities for evaluation are reviewed    While the Integrated Product Team leader     Weakness
                 with acquisition organization management      reviews the status of the contract and the
                 on a periodic basis.                          contractor’s cost and schedule, he does not
                                                               review the status of the activities that are
                                                               required to be performed for any of the key
                                                               process areas.
Verification 2   The activities for evaluation are reviewed  While the product team leader reviews the       Weakness
                 with the project manager on both a periodic status of the contract and the contractor’s
                 and event-driven basis.                     cost and schedule, he does not review the
                                                             status of the activities that are required to
                                                             be performed for any of the key process
                                                             areas.


                                       FAA performs most but not all evaluation KPA practices. To satisfy all
Conclusions                            evaluation practices and thereby have reasonable assurance that its
                                       software acquisition projects will be effectively evaluated and tested on a
                                       repeatable basis, FAA must ensure that its product teams identify
                                       evaluation resource, training, and experience needs, and that they
                                       measure and brief management on the status of all evaluation activities.




                                       Page 99                                                GAO/AIMD-97-47 Air Traffic Control
Chapter 7
Evaluation




Page 100     GAO/AIMD-97-47 Air Traffic Control
Chapter 8

Transition and Support


                        The purpose of transition and support is to provide for the effective and
                        efficient “hand-off” of the acquired software products to the support
                        organization responsible for software maintenance. According to the
                        SA-CMM, repeatable transition and support processes, among other things,
                        include (1) having a written policy for transitioning software products to
                        the support organization, (2) designating a group that is responsible for
                        coordinating transition and support activities, (3) having a complete
                        inventory of all software and related items that are to be transitioned,
                        (4) including members of the support organization in transition and
                        support planning, (5) requiring the support organization to demonstrate its
                        capability to modify and support the software, and (6) measuring and
                        reporting to management on the status of transition and support activities.


                        All of the projects were strong in several transition and support practice
Transition and          areas. For example, FAA has a written policy for transitioning software
Support Not Being       products to the support organization. Additionally, all five projects have
Performed Effectively   designated a group responsible for transition and support. However,
                        various weaknesses in other practices prevented FAA from satisfying this
                        KPA. In particular, some of the teams lack a complete inventory of all the
                        software and related products to be transitioned, thus jeopardizing the
                        efforts of the support organization to effectively maintain the full software
                        configuration. Additionally, one team did not include the software support
                        organization in planning for transition and support, and some teams do not
                        have plans to require the support organization to demonstrate its
                        capability to maintain the software after transition. As a result, support
                        problems, such as the inability to perform required maintenance, may
                        result. Further, none of the projects are measuring and reporting to
                        management on the status of transition and support activities, precluding
                        management from addressing transition and support problems
                        expeditiously.

                        Figure 8.1 provides a comprehensive listing of the five projects’ strengths,
                        weaknesses, and observations for the transition and support KPA. The
                        specific findings supporting the practice ratings cited in figure 8.1 are in
                        tables 8.1 through 8.5.




                        Page 101                                      GAO/AIMD-97-47 Air Traffic Control
                                                      Chapter 8
                                                      Transition and Support




Figure 8.1: Transition and Support Summary


                    Key Practice                                               ARTSIIIE        DSR         NIMS          VSCS         WARP

                    The acquisition organization has a written policy for
   Commitment 1     the transitioning of software products to the support      Strength      Strength     Strength    Observation    Strength
                    organization.

                    The acquisition organization ensures that the
   Commitment 2     software support organization is involved in planning      Weakness      Strength     Strength      Strength     Strength
                    for transition and support.

                    A group that is responsible for coordinating the           Strength      Strength     Weakness      Strength      Strength
     Ability 1
                    transition and support activities exists.


     Ability 2      Responsibility for transition and support activities is    Strength      Strength     Strength      Strength     Strength
                    designated.


     Ability 3      Adequate resources are provided for transition and         Weakness     Weakness      Not rated    Weakness      Weakness
                    support activities.

                    The organization responsible for providing support
     Ability 4      of the software products is identified no later than       Strength     Weakness      Not rated     Strength     Weakness
                    release of the solicitation.

                    The software support organization has a complete
     Ability 5      inventory of all software and related items that are       Strength     Weakness      Not rated    Weakness      Not rated
                    to be transitioned.


     Ability 6      Individuals performing transition and support             Observation   Observation   Not rated   Observation   Observation
                    activities have experience or receive training.

                    The members of organizations interfacing with the
     Ability 7      transition and support activities receive orientation     Weakness      Observation   Not rated   Observation    Not rated
                    on the salient aspects of the transition and support
                    activities.

                    The project team documents its plans for transition        Strength      Strength     Not rated    Weakness      Strength
     Activity 1
                    and support activities.


     Activity 2     The project team's transition and support activities       Strength     Weakness      Not rated   Observation    Not rated
                    are performed in accordance with its plans.

                    The project team transfers responsibility for the
     Activity 3     software products only after the support organization     Weakness       Strength     Not rated    Weakness     Observation
                    demonstrates its capability to modify and support the
                    software products.

                    The project team oversees the configuration control        Strength      Strength     Not rated    Weakness      Not rated
     Activity 4
                    of the software products throughout the transition.


  Measurement 1     Measurements are made and used to determine the           Weakness      Weakness      Weakness     Weakness      Weakness
                    status of the transition and support activities.

                    The activities for transition and support are reviewed
   Verification 1   with acquisition and software support organizations'      Weakness      Weakness      Weakness     Weakness      Weakness
                    management on a periodic basis.




                                                      Page 102                                                GAO/AIMD-97-47 Air Traffic Control
                                                 Chapter 8
                                                 Transition and Support




                 Key Practice                                                ARTSIIIE             DSR                NIMS               VSCS          WARP

                 The activities for transition and support are reviewed
Verification 2   with the project manager on both a periodic and             Weakness           Weakness           Weakness           Weakness       Weakness
                 event-driven basis.


                                                           = Weakness      Key practice not implemented
                                                           = Strength      Key practice effectively implemented
                                                           = Observation Key practice evaluated but evidence inconclusive. Cannot characterize as either strength or
                                                                         weakness

                                                           = Not rated     Key practice not currently relevant to project, therefore not evaluated




                                                 Page 103                                                               GAO/AIMD-97-47 Air Traffic Control
                                            Chapter 8
                                            Transition and Support




Table 8.1: Transition and Support Findings for ARTSIIIE
                                            Automated Radar Terminal System IIIE
                     Key Practice                                    Finding                                        Rating
Commitment 1         The acquisition organization has a written      FAA Order 1800.58 is the written policy.       Strength
                     policy for the transitioning of software
                     products to the support organization.
Commitment 2         The acquisition organization ensures that       The Product Team Plan identifies the           Weakness
                     the software support organization is            organization responsible for software
                     involved in planning for transition and         support, but transition is not mentioned.
                     support.
Ability 1            A group that is responsible for coordinating    A group responsible for coordinating the       Strength
                     the transition and support activities exists.   transition to support activities exists.
Ability 2            Responsibility for transition and support       Responsibility for transition and support      Strength
                     activities is designated.                       activities is designated.
Ability 3            Adequate resources are provided for             No mechanism exists for identifying          Weakness
                     transition and support activities.              resources required and for ensuring that the
                                                                     needed resources are provided to the
                                                                     project.
Ability 4            The organization responsible for providing      The Product Team Plan designates               Strength
                     support of the software products is             responsibility for providing support of the
                     identified no later than release of the         software products before release of the
                     solicitation.                                   solicitation.
Ability 5            The software support organization has a         The software support organization has a        Strength
                     complete inventory of all software and          complete inventory of all software and
                     related items that are to be transitioned.      related items that are to be transitioned.
Ability 6            Individuals performing transition and           The acquisition organization has no            Observation
                     support activities have experience or           guidance regarding training or experience
                     receive training.                               requirements for project participation.
Ability 7            The members of organizations interfacing        Orientation was not given.                     Weakness
                     with the transition and support activities
                     receive orientation on the salient aspects of
                     the transition and support activities.
Activity 1           The project team documents its plans for        The product team documents its plans for       Strength
                     transition and support activities.              transition and support activities.
Activity 2           The project team’s transition and support       The product team’s activities are being        Strength
                     activities are performed in accordance with     conducted in accordance with its plans.
                     its plans.
Activity 3           The project team transfers responsibility for No certification procedures to test the          Weakness
                     the software products only after the support capability of the support organization was
                     organization demonstrates its capability to   provided.
                     modify and support the software products.
Activity 4           The project team oversees the configuration The product team oversees the                      Strength
                     control of the software products throughout configuration management of the software
                     the transition.                             throughout transition.
Measurement 1        Measurements are made and used to               No internal process measurements are           Weakness
                     determine the status of the transition and      taken and used to determine the status of
                     support activities.                             activities for any of the key process areas.
                                                                                                                               (continued)



                                            Page 104                                                 GAO/AIMD-97-47 Air Traffic Control
                                        Chapter 8
                                        Transition and Support




                                         Automated Radar Terminal System IIIE
                 Key Practice                                    Finding                                         Rating
Verification 1   The activities for transition and support are   While the Integrated Product Team leader     Weakness
                 reviewed by acquisition and software            reviews the status of the contract and the
                 support organizations’ management on a          contractor’s cost and schedule, he does not
                 periodic basis.                                 review the status of the activities that are
                                                                 required to be performed for any of the key
                                                                 process areas.
Verification 2   The activities for transition and support are   While the product team leader reviews the       Weakness
                 reviewed by the project manager on both a       status of the contract and the contractor’s
                 periodic and event-driven basis.                cost and schedule, he does not review the
                                                                 status of the activities that are required to
                                                                 be performed for any of the key process
                                                                 areas.




                                        Page 105                                                  GAO/AIMD-97-47 Air Traffic Control
                                            Chapter 8
                                            Transition and Support




Table 8.2: Transition and Support Findings for DSR
                                                 Display System Replacement
                     Key Practice                                    Finding                                          Rating
Commitment 1         The acquisition organization has a written      FAA has a written policy for transition and      Strength
                     policy for the transitioning of software        support.
                     products to the support organization.
Commitment 2         The acquisition organization ensures that       The Product Team Plan identifies the             Strength
                     the software support organization is            individual responsible for support planning.
                     involved in planning for transition and
                     support.
Ability 1            A group that is responsible for coordinating    The product team is responsible for              Strength
                     the transition and support activities exists.   transition and support activities.
Ability 2            Responsibility for transition and support       Responsibility is assigned to the product        Strength
                     activities is designated.                       team for transition and support activities.
Ability 3            Adequate resources are provided for             No mechanism exists for identifying          Weakness
                     transition and support activities.              resources required and for ensuring that the
                                                                     needed resources are provided to the
                                                                     project.
Ability 4            The organization responsible for providing      Although DSR is in the development stage,        Weakness
                     support of the software products is             the support organization has not been
                     identified no later than release of the         identified.
                     solicitation.
Ability 5            The software support organization has a         While documents are planned for                  Weakness
                     complete inventory of all software and          configuration audits as part of the transition
                     related items that are to be transitioned.      process, a complete inventory does not
                                                                     exist.
Ability 6            Individuals performing transition and           The acquisition organization has no              Observation
                     support activities have experience or           guidance regarding training or experience
                     receive training.                               requirements for project participation.
Ability 7            The members of organizations interfacing        While officials stated that orientation had      Observation
                     with the transition and support activities      occurred, they could not provide
                     receive orientation on the salient aspects of   documents to support this.
                     the transition and support activities.
Activity 1           The project team documents its plans for        The Program Implementation Plan contains         Strength
                     transition and support activities.              the plans for transition and support.
Activity 2           The project team’s transition and support       Transition planning is being delayed             Weakness
                     activities are performed in accordance with     pending a decision on who will provide
                     its plans.                                      maintenance.
Activity 3           The project team transfers responsibility for There is a plan for the support organization       Strength
                     the software products only after the support to demonstrate its capabilities prior to
                     organization demonstrates its capability to   transition of responsibilities.
                     modify and support the software products.
Activity 4           The project team oversees the configuration The product team oversees the contractor,            Strength
                     control of the software products throughout who will maintain configuration
                     the transition.                             management throughout the transition.
Measurement 1        Measurements are made and used to               No internal process measurements are             Weakness
                     determine the status of the transition and      taken and used to determine the status of
                     support activities.                             activities for any of the key process areas.
                                                                                                                                 (continued)


                                            Page 106                                                  GAO/AIMD-97-47 Air Traffic Control
                                        Chapter 8
                                        Transition and Support




                                               Display System Replacement
                 Key Practice                                    Finding                                         Rating
Verification 1   The activities for transition and support are   While the Integrated Product Team leader     Weakness
                 reviewed by acquisition and software            reviews the status of the contract and the
                 support organizations’ management on a          contractor’s cost and schedule, he does not
                 periodic basis.                                 review the status of the activities that are
                                                                 required to be performed for any of the key
                                                                 process areas.
Verification 2   The activities for transition and support are   While the product team leader reviews the       Weakness
                 reviewed by the project manager on both a       status of the contract and the contractor’s
                 periodic and event-driven basis.                cost and schedule, he does not review the
                                                                 status of the activities that are required to
                                                                 be performed for any of the key process
                                                                 areas.




                                        Page 107                                                  GAO/AIMD-97-47 Air Traffic Control
                                           Chapter 8
                                           Transition and Support




Table 8.3: Transition and Support Findings for NIMS
                                           NAS Infrastructure Management System
                    Key Practice                                    Finding                                          Rating
Commitment 1        The acquisition organization has a written      The Acquisition Management System is the         Strength
                    policy for the transitioning of software        written policy.
                    products to the support organization.
Commitment 2        The acquisition organization ensures that       The software support organization is             Strength
                    the software support organization is            represented in the product team.
                    involved in planning for transition and
                    support.
Ability 1           A group that is responsible for coordinating    Officials gave conflicting answers as to who Weakness
                    the transition and support activities exists.   is responsible for coordinating the transition
                                                                    and support activities, and they could not
                                                                    provide documentation that formally
                                                                    designates responsibility.
Ability 2           Responsibility for transition and support       Responsibility for transition and support        Strength
                    activities is designated.                       activities is designated.
Ability 3           Adequate resources are provided for             It is too early in the project’s cycle to        Not rated
                    transition and support activities.              evaluate this ability.
Ability 4           The organization responsible for providing      It is too early in the project’s cycle to        Not rated
                    support of the software products is             evaluate this ability.
                    identified no later than release of the
                    solicitation.
Ability 5           The software support organization has a         It is too early in the project’s cycle to        Not rated
                    complete inventory of all software and          evaluate this ability.
                    related items that are to be transitioned.
Ability 6           Individuals performing transition and           It is too early in the project’s cycle to        Not rated
                    support activities have experience or           evaluate this ability.
                    receive training.
Ability 7           The members of organizations interfacing        It is too early in the project’s cycle to        Not rated
                    with the transition and support activities      evaluate this ability.
                    receive orientation on the salient aspects of
                    the transition and support activities.
Activity 1          The project team documents its plans for        It is too early in the project’s cycle to        Not rated
                    transition and support activities.              evaluate this activity.
Activity 2          The project team’s transition and support       It is too early in the project’s cycle to        Not rated
                    activities are performed in accordance with     evaluate this activity.
                    its plans.
Activity 3          The project team transfers responsibility for It is too early in the project’s cycle to          Not rated
                    the software products only after the support evaluate this activity.
                    organization demonstrates its capability to
                    modify and support the software products.
Activity 4          The project team oversees the configuration It is too early in the project’s cycle to            Not rated
                    control of the software products throughout evaluate this activity.
                    the transition.
Measurement 1       Measurements are made and used to               No internal process measurements are            Weakness
                    determine the status of the transition and      taken to determine the status of activities for
                    support activities.                             any of the key process areas.
                                                                                                                                (continued)


                                           Page 108                                                    GAO/AIMD-97-47 Air Traffic Control
                                        Chapter 8
                                        Transition and Support




                                        NAS Infrastructure Management System
                 Key Practice                                    Finding                                         Rating
Verification 1   The activities for transition and support are   While the Integrated Product Team leader     Weakness
                 reviewed by acquisition and software            reviews the status of the contract and the
                 support organizations’ management on a          contractor’s cost and schedule, he does not
                 periodic basis.                                 review the status of the activities that are
                                                                 required to be performed for any of the key
                                                                 process areas.
Verification 2   The activities for transition and support are   While the product team leader reviews the       Weakness
                 reviewed by the project manager on both a       status of the contract and the contractor’s
                 periodic and event-driven basis.                cost and schedule, he does not review the
                                                                 status of the activities that are required to
                                                                 be performed for any of the key process
                                                                 areas.




                                        Page 109                                                  GAO/AIMD-97-47 Air Traffic Control
                                            Chapter 8
                                            Transition and Support




Table 8.4: Transition and Support Findings for VSCS
                                             Voice Switching and Control System
                     Key Practice                                    Finding                                         Rating
Commitment 1         The acquisition organization has a written      A written policy for transition exists;         Observation
                     policy for the transitioning of software        however, the officials interviewed did not
                     products to the support organization.           identify it.
Commitment 2         The acquisition organization ensures that       The acquisition organization ensured that       Strength
                     the software support organization is            the software support organization was
                     involved in planning for transition and         involved in planning for transition and
                     support.                                        support.
Ability 1            A group that is responsible for coordinating    The product team is responsible for             Strength
                     the transition and support activities exists.   coordinating transition and support
                                                                     activities.
Ability 2            Responsibility for transition and support       Responsibility for transition and support       Strength
                     activities is designated.                       activities is designated.
Ability 3            Adequate resources are provided for             No mechanism exists for identifying the      Weakness
                     transition and support activities.              resources required and for ensuring that the
                                                                     needed resources are provided to the
                                                                     project.
Ability 4            The organization responsible for providing      The organization responsible was identified     Strength
                     support of the software products is             before the release of the solicitation.
                     identified no later than release of the
                     solicitation.
Ability 5            The software support organization has a         The software support organization does not Weakness
                     complete inventory of all software and          have a complete inventory of all software
                     related items that are to be transitioned.      and related items that are to be transitioned.
Ability 6            Individuals performing transition and           The acquisition organization has no             Observation
                     support activities have experience or           guidance regarding training or experience
                     receive training.                               requirements for project participation.
Ability 7            The members of organizations interfacing        While officials said that members of            Observation
                     with the transition and support activities      organizations interfacing with the transition
                     receive orientation on the salient aspects of   and support activities received orientation,
                     the transition and support activities.          they could not provide documentation to
                                                                     support this.
Activity 1           The project team documents its plans for        Only a draft plan exists.                       Weakness
                     transition and support activities.
Activity 2           The project team’s transition and support       There is no plan for transition and support,    Weakness
                     activities are performed in accordance with     therefore, the activity cannot be performed
                     its plans.                                      in accordance with it.
Activity 3           The project team transfers responsibility for No demonstrations were held for the          Weakness
                     the software products only after the support support organization to demonstrate its
                     organization demonstrates its capability to   capability to support the software products.
                     modify and support the software products.
Activity 4           The project team oversees the configuration Since there is no transition and support         Weakness
                     control of the software products throughout plan, there is no evidence that this activity is
                     the transition.                             being performed.
                                                                                                                                (continued)




                                            Page 110                                                 GAO/AIMD-97-47 Air Traffic Control
                                        Chapter 8
                                        Transition and Support




                                          Voice Switching and Control System
                 Key Practice                                    Finding                                         Rating
Measurement 1    Measurements are made and used to               No internal process measurements are            Weakness
                 determine the status of the transition and      taken or used to determine the status of
                 support activities.                             activities for any of the key process areas.
Verification 1   The activities for transition and support are   While the Integrated Product Team leader     Weakness
                 reviewed by acquisition and software            reviews the status of the contract and the
                 support organizations’ management on a          contractor’s cost and schedule, he does not
                 periodic basis.                                 review the status of the activities that are
                                                                 required to be performed for any of the key
                                                                 processes.
Verification 2   The activities for transition and support are   While the product team leader reviews the       Weakness
                 reviewed by the project manager on both a       status of the contract and the contractor’s
                 periodic and event-driven basis.                cost and schedule, he does not review the
                                                                 status of the activities that are required to
                                                                 be performed for any of the key process
                                                                 areas.




                                        Page 111                                                  GAO/AIMD-97-47 Air Traffic Control
                                            Chapter 8
                                            Transition and Support




Table 8.5: Transition and Support Findings for WARP
                                                Weather and Radar Processor
                     Key Practice                                    Finding                                        Rating
Commitment 1         The acquisition organization has a written      There is a written policy for transition and   Strength
                     policy for the transitioning of software        support.
                     products to the support organization.
Commitment 2         The acquisition organization ensures that       The support organization is part of the      Strength
                     the software support organization is            product team and is involved in planning for
                     involved in planning for transition and         transition.
                     support.
Ability 1            A group that is responsible for coordinating    The product team has the responsibility for    Strength
                     the transition and support activities exists.   coordinating the transition and support
                                                                     activities.
Ability 2            Responsibility for transition and support       Responsibility for transition and support     Strength
                     activities is designated.                       activities is designated to the product team.
Ability 3            Adequate resources are provided for             No mechanism exists for identifying and for    Weakness
                     transition and support activities.              ensuring that the needed resources are
                                                                     provided to the project.
Ability 4            The organization responsible for providing      The organization responsible for providing     Weakness
                     support of the software products is             support of the software products was not
                     identified no later than release of the         chosen before the release of the solicitation.
                     solicitation.
Ability 5            The software support organization has a         It is too early in the project’s development   Not rated
                     complete inventory of all software and          for it to have developed an inventory.
                     related items that are to be transitioned.
Ability 6            Individuals performing transition and           The acquisition organization has no            Observation
                     support activities have experience or           guidance regarding training or experience
                     receive training.                               requirements for project participation.
Ability 7            The members of organizations interfacing        It is too early in the project’s development   Not rated
                     with the transition and support activities      for it to define support.
                     receive orientation on the salient aspects of
                     the transition and support activities.
Activity 1           The project team documents its plans for        The Product Team Plan documents the            Strength
                     transition and support activities.              initial plans for transition and support
                                                                     activities.
Activity 2           The project team’s transition and support       It is too early in the project’s development   Not rated
                     activities are performed in accordance with     for it to be evaluated against this key
                     its plans.                                      practice.
Activity 3           The project team transfers responsibility for Officials gave conflicting answers as to         Observation
                     the software products only after the support whether or not the demonstration has been
                     organization demonstrates its capability to   planned.
                     modify and support the software products.
Activity 4           The project team oversees the configuration It is too early in the project’s development       Not rated
                     control of the software products throughout for it to be evaluated against this key
                     the transition.                             practice.
Measurement 1        Measurements are made and used to               No internal process measurements are           Weakness
                     determine the status of the transition and      taken and used to determine the status of
                     support activities.                             activities for any of the key process areas.
                                                                                                                               (continued)


                                            Page 112                                                  GAO/AIMD-97-47 Air Traffic Control
                                        Chapter 8
                                        Transition and Support




                                              Weather and Radar Processor
                 Key Practice                                    Finding                                         Rating
Verification 1   The activities for transition and support are   While the Integrated Product Team leader     Weakness
                 reviewed by acquisition and software            reviews the status of the contract and the
                 support organizations’ management on a          contractor’s cost and schedule, he does not
                 periodic basis.                                 review the status of the activities that are
                                                                 required to be performed for any of the key
                                                                 process areas.
Verification 2   The activities for transition and support are   While the product team leader reviews the       Weakness
                 reviewed by the project manager on both a       status of the contract and the contractor’s
                 periodic and event-driven basis.                cost and schedule, he does not review the
                                                                 status of the activities that are required to
                                                                 be performed for any of the key process
                                                                 areas.


                                        FAA’stransition and support process for its ATC modernization suffers from
Conclusions                             weaknesses which render it undefined and undisciplined. In light of FAA’s
                                        enormous investment in ATC-related software and the fact that over
                                        65 percent of the life cycle cost of software is incurred during
                                        maintenance, it is essential that these weaknesses be corrected.




                                        Page 113                                                  GAO/AIMD-97-47 Air Traffic Control
Chapter 8
Transition and Support




Page 114                 GAO/AIMD-97-47 Air Traffic Control
Chapter 9

Acquisition Risk Management


                    SEI defines risk as the possibility of suffering a loss. The purpose of
                    acquisition risk management is to formally identify risks as early as
                    possible and adjust the acquisition to mitigate those risks. According to
                    the SA-CMM, an effective risk management process, among other things,
                    includes (1) having a written policy on acquisition risk management,
                    (2) developing a software acquisition risk management plan,
                    (3) conducting software risk management activities in accordance with the
                    plan (e.g., identifying risks, taking mitigation actions, and tracking risk
                    mitigation actions to completion), and (4) measuring and reporting on the
                    status of acquisition risk management activities to management.


                    FAA is not effectively performing ATC modernization software acquisition
ATC Modernization   risk management. Although FAA has a written policy on risk management,
Software Risk       it has many weaknesses in this KPA. For example, most teams have no risk
Management Is       management plan, and the one team that has a plan failed to follow it. As a
                    result, the teams are not identifying risks, taking risk mitigation actions,
Ineffective         and tracking risk mitigation actions to completion. Moreover, none of the
                    teams measure and report to management on the status of acquisition risk
                    management activities. Consequently, management is not in a position to
                    correct problems promptly.

                    Figure 9.1 provides a comprehensive listing of the five projects’ strengths,
                    weaknesses, and observations for the acquisition risk management KPA.
                    The specific findings supporting the practice ratings in figure 9.1 are in
                    tables 9.1 through 9.5.




                    Page 115                                      GAO/AIMD-97-47 Air Traffic Control
                                                       Chapter 9
                                                       Acquisition Risk Management




Figure 9.1: Acquisition Risk Management Summary


                     Key Practice                                               ARTSIIIE         DSR          NIMS          VSCS          WARP


   Commitment 1      The acquisition organization has a written policy for       Strength      Strength     Weakness       Strength      Strength
                     the management of acquisition risk.


      Ability 1      A group that is responsible for coordinating               Weakness       Strength      Strength      Strength      Strength
                     acquisition risk management activities exists.


      Ability 2      Adequate resources are provided for acquisition risk       Weakness      Weakness      Weakness      Weakness      Weakness
                     management activities.

                     Individuals performing acquisition risk management
      Ability 3      activities have experience or receive required             Observation   Observation   Observation   Observation   Observation
                     training.


      Activity 1     Risk identification, analysis, and mitigation activities   Weakness       Strength      Strength     Weakness       Strength
                     are integrated into software acquisition planning.

                     The Software Risk Management Plan is developed
      Activity 2     according to the project's defined software                Weakness      Weakness      Weakness       Strength     Weakness
                     acquisition process.

                     The project's acquisition risk management activities
      Activity 3     are performed in accordance with its Software Risk         Weakness      Weakness      Weakness      Weakness       Weakness
                     Management Plan.

                     The project team identifies, analyzes, and takes
      Activity 4     appropriate software risk mitigation actions during        Observation   Weakness      Weakness      Weakness       Strength
                     acquisition planning.
                     Risk identification, analysis, and mitigation are
      Activity 5     conducted as an integral part of the project               Weakness       Strength      Not rated    Weakness       Strength
                     performance management and contract performance
                     management processes.


      Activity 6     Software risk mitigation actions are tracked to            Weakness      Weakness      Observation   Weakness       Strength
                     completion.


   Measurement 1     Measurements are made and used to determine the            Weakness      Weakness      Weakness      Weakness      Weakness
                     status of the acquisition risk management activities.


   Measurement 2     Resources expended for acquisition risk management         Weakness      Weakness      Weakness      Weakness      Weakness
                     activities are recorded and tracked.

                     The activities for acquisition risk management are
    Verification 1   reviewed by acquisition organization management on         Weakness      Weakness      Weakness      Weakness      Weakness
                     a periodic basis.




                                                       Page 116                                                  GAO/AIMD-97-47 Air Traffic Control
                                               Chapter 9
                                               Acquisition Risk Management




                 Key Practice                                             ARTSIIIE             DSR                NIMS               VSCS          WARP

                 The activities for acquisition risk management are
Verification 2   reviewed by the project manager on both a periodic       Weakness          Weakness           Weakness           Weakness        Weakness
                 and event-driven basis.


                                                        = Weakness      Key practice not implemented
                                                        = Strength      Key practice effectively implemented
                                                        = Observation Key practice evaluated but evidence inconclusive. Cannot characterize as either strength or
                                                                      weakness
                                                         = Not rated    Key practice not currently relevant to project, therefore not evaluated




                                               Page 117                                                              GAO/AIMD-97-47 Air Traffic Control
                                            Chapter 9
                                            Acquisition Risk Management




Table 9.1: Acquisition Risk Management Findings for ARTSIIIE
                                          Automated Radar Terminal System IIIE
                   Key Practice                                        Finding                                              Rating
Commitment 1       The acquisition organization has a written policy There is a policy for acquisition risk                 Strength
                   for the management of acquisition risk.           management.
Ability 1          A group that is responsible for coordinating        No group is assigned acquisition risk                Weakness
                   acquisition risk management activities exists.      management tasks.
Ability 2          Adequate resources are provided for                 No mechanism exists for identifying resources        Weakness
                   acquisition risk management activities.             required and for ensuring that the needed
                                                                       resources are provided to the project.
Ability 3          Individuals performing acquisition risk             The acquisition organization has no guidance         Observation
                   management activities have experience or            regarding training or experience requirements
                   receive required training.                          for project participation.
Activity 1         Risk identification, analysis, and mitigation       No risk identification, analysis, or mitigation     Weakness
                   activities are integrated into software acquisition activities are integrated into software acquisition
                   planning.                                           planning.
Activity 2         The Software Risk Management Plan is                There is no risk management plan.                    Weakness
                   developed according to the project’s defined
                   software acquisition process.
Activity 3         The project’s acquisition risk management           No software risk management plan exists,             Weakness
                   activities are performed in accordance with its     therefore, the activities are not performed in
                   Software Risk Management Plan.                      accordance with it.
Activity 4         The project team identifies, analyzes, and takes    Officials stated that the product team identifies,   Observation
                   appropriate software risk mitigation actions        analyzes, and mitigates risks, but could not
                   during acquisition planning.                        provide documents to support this.
Activity 5         Risk identification, analysis, and mitigation are   Officials could provide no evidence of risk          Weakness
                   conducted as an integral part of the project        identification, analysis, and mitigation being
                   performance management and contract                 conducted as part of project and contract
                   performance management processes.                   management.
Activity 6         Software risk mitigation actions are tracked to     Risks are not tracked to completion.                 Weakness
                   completion.
Measurement 1      Measurements are made and used to determine No internal process measurements are taken           Weakness
                   the status of the acquisition risk management and used to determine the status of activities for
                   activities.                                   any of the key process areas.
Measurement 2      Resources expended for acquisition risk             FAA does not track resources.                        Weakness
                   management activities are recorded and
                   tracked.
Verification 1     The activities for acquisition risk management      While the Integrated Product Team leader             Weakness
                   are reviewed by acquisition organization            reviews the status of the contract and the
                   management on a periodic basis.                     contractor’s cost and schedule, he does not
                                                                       review the status of the activities that are
                                                                       required to be performed for any of the key
                                                                       process areas.
Verification 2     The activities for acquisition risk management      While the product team leader reviews the            Weakness
                   are reviewed by the project manager on both a       status of the contract and the contractor’s cost
                   periodic and event-driven basis.                    and schedule, he does not review the status of
                                                                       the activities that are required to be performed
                                                                       for any of the key process areas.



                                            Page 118                                                GAO/AIMD-97-47 Air Traffic Control
                                            Chapter 9
                                            Acquisition Risk Management




Table 9.2: Acquisition Risk Management Findings for DSR
                                               Display System Replacement
                    Key Practice                                    Finding                                           Rating
Commitment 1        The acquisition organization has a written      There is a written policy for acquisition risk    Strength
                    policy for the management of acquisition        management.
                    risk.
Ability 1           A group that is responsible for coordinating    The product team is responsible for               Strength
                    acquisition risk management activities          acquisition risk management.
                    exists.
Ability 2           Adequate resources are provided for             No mechanism exists for identifying          Weakness
                    acquisition risk management activities.         resources required and for ensuring that the
                                                                    needed resources are provided to the
                                                                    project.
Ability 3           Individuals performing acquisition risk         The acquisition organization has no               Observation
                    management activities have experience or        guidance regarding training or experience
                    receive required training.                      requirements for project participation.
Activity 1          Risk identification, analysis, and mitigation   Risk identification, analysis, and mitigation     Strength
                    activities are integrated into software         are integrated into the software acquisition
                    acquisition planning.                           planning.
Activity 2          The Software Risk Management Plan is            While there is a risk management plan that        Weakness
                    developed according to the project’s            addresses software at a high level, there is
                    defined software acquisition process.           no software risk management plan.
Activity 3          The project’s acquisition risk management       Because there is no software risk                 Weakness
                    activities are performed in accordance with     management plan, activities are not
                    its Software Risk Management Plan.              performed in accordance with it.
Activity 4          The project team identifies, analyzes, and      The risk identification activities are not well   Weakness
                    takes appropriate software risk mitigation      defined. The product team used risk
                    actions during acquisition planning.            interchangeably with currently identified
                                                                    problems, action items, issues, and
                                                                    schedule tracking.
Activity 5          Risk identification, analysis, and mitigation   What the product team considers risks             Strength
                    are conducted as an integral part of the        (problems, action items, issues, etc.) are
                    project performance management and              identified, analyzed, and mitigated.
                    contract performance management
                    processes.
Activity 6          Software risk mitigation actions are tracked    Software risk mitigation is not tracked to        Weakness
                    to completion.                                  completion.
Measurement 1       Measurements are made and used to               No internal process measurements are              Weakness
                    determine the status of the acquisition risk    taken and used to determine the status of
                    management activities.                          activities for any of the key process areas.
Measurement 2       Resources expended for acquisition risk         FAA does not track resources.                     Weakness
                    management activities are recorded and
                    tracked.
                                                                                                                                 (continued)




                                            Page 119                                                  GAO/AIMD-97-47 Air Traffic Control
                                     Chapter 9
                                     Acquisition Risk Management




                                           Display System Replacement
                 Key Practice                             Finding                                         Rating
Verification 1   The activities for acquisition risk      While the Integrated Product Team leader     Weakness
                 management are reviewed by acquisition   reviews the status of the contract and the
                 organization management on a periodic    contractor’s cost and schedule, he does not
                 basis.                                   review the status of the activities that are
                                                          required to be performed for any of the key
                                                          process areas.
Verification 2   The activities for acquisition risk      While the product team leader reviews the       Weakness
                 management are reviewed by the project   status of the contract and the contractor’s
                 manager on both a periodic and           cost and schedule, he does not review the
                 event-driven basis.                      status of the activities that are required to
                                                          be performed for any of the key process
                                                          areas.




                                     Page 120                                              GAO/AIMD-97-47 Air Traffic Control
                                            Chapter 9
                                            Acquisition Risk Management




Table 9.3: Acquisition Risk Management Findings for NIMS
                                          NAS Infrastructure Management System
                    Key Practice                                    Finding                                         Rating
Commitment 1        The acquisition organization has a written      The Acquisition Management System is the        Weakness
                    policy for the management of acquisition        written policy for acquisition risk
                    risk.                                           management, but it does not address
                                                                    software.
Ability 1           A group that is responsible for coordinating    Both the Integrated Program Plan and the        Strength
                    acquisition risk management activities          Product Team Plan assign the risk
                    exists.                                         management plan activities to the team.
Ability 2           Adequate resources are provided for             No mechanism exists for identifying          Weakness
                    acquisition risk management activities.         resources required and for ensuring that the
                                                                    needed resources are provided to the
                                                                    project.
Ability 3           Individuals performing acquisition risk         The acquisition organization has no             Observation
                    management activities have experience or        guidance regarding training or experience
                    receive required training.                      requirements for project participation.
Activity 1          Risk identification, analysis, and mitigation   Acquisition risk management is part of          Strength
                    activities are integrated into software         acquisition planning as called for in both
                    acquisition planning.                           the Integrated Program Plan and the
                                                                    Product Team Plan.
Activity 2          The Software Risk Management Plan is            No software risk management plan exists.        Weakness
                    developed according to the project’s            The risk management plan does not
                    defined software acquisition process.           address software.
Activity 3          The project’s acquisition risk management       There is no software risk management plan, Weakness
                    activities are performed in accordance with     therefore, activities cannot be performed in
                    its Software Risk Management Plan.              accordance with it.
Activity 4          The project team identifies, analyzes, and      The project team does not identify, analyze, Weakness
                    takes appropriate software risk mitigation      or mitigate software risks.
                    actions during acquisition planning.
Activity 5          Risk identification, analysis, and mitigation   While plans call for risk identification,       Not rated
                    are conducted as an integral part of the        analysis, and mitigation, the project has not
                    project performance management and              reached a stage where this activity applies.
                    contract performance management
                    processes.
Activity 6          Software risk mitigation actions are tracked    A risk tracking system is being developed     Observation
                    to completion.                                  as called for in the Integrated Program Plan.
Measurement 1       Measurements are made and used to               No internal process measurements are            Weakness
                    determine the status of the acquisition risk    taken to determine the status of activities for
                    management activities.                          any of the key process areas.
Measurement 2       Resources expended for acquisition risk         FAA does not track resources.                   Weakness
                    management activities are recorded and
                    tracked.
                                                                                                                               (continued)




                                            Page 121                                                GAO/AIMD-97-47 Air Traffic Control
                                     Chapter 9
                                     Acquisition Risk Management




                                     NAS Infrastructure Management System
                 Key Practice                             Finding                                         Rating
Verification 1   The activities for acquisition risk      While the Integrated Product Team leader     Weakness
                 management are reviewed by acquisition   reviews the status of the contract and the
                 organization management on a periodic    contractor’s cost and schedule, he does not
                 basis.                                   review the status of the activities that are
                                                          required to be performed for any of the key
                                                          process areas.
Verification 2   The activities for acquisition risk      While the product team leader reviews the       Weakness
                 management are reviewed by the project   status of the contract and the contractor’s
                 manager on both a periodic and           cost and schedule, he does not review the
                 event-driven basis.                      status of the activities that are required to
                                                          be performed for any of the key process
                                                          areas.




                                     Page 122                                              GAO/AIMD-97-47 Air Traffic Control
                                            Chapter 9
                                            Acquisition Risk Management




Table 9.4: Acquisition Risk Management Findings for VSCS
                                           Voice Switching and Control System
                    Key Practice                                    Finding                                          Rating
Commitment 1        The acquisition organization has a written      FAA Order 1810.1F is the policy for              Strength
                    policy for the management of acquisition        acquisition risk management.
                    risk.
Ability 1           A group that is responsible for coordinating    The product team leader is the acquisition       Strength
                    acquisition risk management activities          risk manager.
                    exists.
Ability 2           Adequate resources are provided for             No mechanism exists for identifying the      Weakness
                    acquisition risk management activities.         resources required and for ensuring that the
                                                                    needed resources are provided to the
                                                                    project.
Ability 3           Individuals performing acquisition risk         The acquisition organization has no              Observation
                    management activities have experience or        guidance regarding training or experience
                    receive required training.                      requirements for project participation.
Activity 1          Risk identification, analysis, and mitigation   The risk management plan does not call for       Weakness
                    activities are integrated into software         risk activities to be integrated into software
                    acquisition planning.                           acquisition planning.
Activity 2          The Software Risk Management Plan is            There is a software risk management plan.        Strength
                    developed according to the project’s
                    defined software acquisition process.
Activity 3          The project’s acquisition risk management       Although there is a risk management plan,        Weakness
                    activities are performed in accordance with     there is no evidence that acquisition risk
                    its Software Risk Management Plan.              management is performed in accordance
                                                                    with the plan.
Activity 4          The project team identifies, analyzes, and      The project team does not identify, analyze, Weakness
                    takes appropriate software risk mitigation      and take appropriate actions during
                    actions during acquisition planning.            acquisition planning.
Activity 5          Risk identification, analysis, and mitigation   The product team does not ensure that risks Weakness
                    are conducted as an integral part of the        are identified, or that analyses and
                    project performance management and              mitigations are conducted as an integral
                    contract performance management                 part of project performance management
                    processes.                                      and contract performance management.
Activity 6          Software risk mitigation actions are tracked    There is no evidence that risk mitigation is     Weakness
                    to completion.                                  tracked to completion.
Measurement 1       Measurements are made and used to               No internal process measurements are             Weakness
                    determine the status of the acquisition risk    taken or used to determine the status of
                    management activities.                          activities for any of the key process areas.
Measurement 2       Resources expended for acquisition risk         FAA does not track resources.                    Weakness
                    management activities are recorded and
                    tracked.
Verification 1      The activities for acquisition risk             While the Integrated Product Team leader     Weakness
                    management are reviewed by acquisition          reviews the status of the contract and the
                    organization management on a periodic           contractor’s cost and schedule, he does not
                    basis.                                          review the status of the activities that are
                                                                    required to be performed for any of the key
                                                                    process areas.
                                                                                                                                (continued)


                                            Page 123                                                 GAO/AIMD-97-47 Air Traffic Control
                                     Chapter 9
                                     Acquisition Risk Management




                                       Voice Switching and Control System
                 Key Practice                             Finding                                         Rating
Verification 2   The activities for acquisition risk      While the product team leader reviews the       Weakness
                 management are reviewed by the project   status of the contract and the contractor’s
                 manager on both a periodic and           cost and schedule, he does not review the
                 event-driven basis.                      status of the activities that are required to
                                                          be performed for any of the key process
                                                          areas.




                                     Page 124                                              GAO/AIMD-97-47 Air Traffic Control
                                            Chapter 9
                                            Acquisition Risk Management




Table 9.5: Acquisition Risk Management Findings for WARP
                                              Weather and Radar Processor
                    Key Practice                                    Finding                                         Rating
Commitment 1        The acquisition organization has a written      FAA Order 1810.1F is the written policy.        Strength
                    policy for the management of acquisition
                    risk.
Ability 1           A group that is responsible for coordinating    The product team is responsible for             Strength
                    acquisition risk management activities          acquisition risk management activities.
                    exists.
Ability 2           Adequate resources are provided for             No mechanism exists for identifying and for     Weakness
                    acquisition risk management activities.         ensuring that the needed resources are
                                                                    provided to the project.
Ability 3           Individuals performing acquisition risk         The acquisition organization has no             Observation
                    management activities have experience or        guidance regarding training or experience
                    receive required training.                      requirements for project participation.
Activity 1          Risk identification, analysis, and mitigation   Risk identification, analysis, and mitigation   Strength
                    activities are integrated into software         activities are integrated into software
                    acquisition planning.                           acquisition planning.
Activity 2          The Software Risk Management Plan is            Although there is a risk management plan, it Weakness
                    developed according to the project’s            incorrectly identifies risks as currently
                    defined software acquisition process.           identified problems, action items, issues,
                                                                    etc.
Activity 3          The project’s acquisition risk management       While issues are reviewed at team              Weakness
                    activities are performed in accordance with     meetings, they are not specifically identified
                    its Software Risk Management Plan.              and managed as risks.
Activity 4          The project team identifies, analyzes, and      The product team identifies and analyzes        Strength
                    takes appropriate software risk mitigation      risks as defined in the risk management
                    actions during acquisition planning.            plan and takes appropriate actions during
                                                                    acquisition planning.
Activity 5          Risk identification, analysis, and mitigation   The product team identifies, analyzes, and      Strength
                    are conducted as an integral part of the        mitigates risks (as defined in the risk
                    project performance management and              management plan) as part of project and
                    contract performance management                 contract performance management.
                    processes.
Activity 6          Software risk mitigation actions are tracked    Software risks as defined in the risk           Strength
                    to completion.                                  management plan are tracked to
                                                                    completion.
Measurement 1       Measurements are made and used to               No internal process measurements are            Weakness
                    determine the status of the acquisition risk    taken and used to determine the status of
                    management activities.                          activities for any of the key process areas.
Measurement 2       Resources expended for acquisition risk         FAA does not track resources.                   Weakness
                    management activities are recorded and
                    tracked.
                                                                                                                               (continued)




                                            Page 125                                                 GAO/AIMD-97-47 Air Traffic Control
                                     Chapter 9
                                     Acquisition Risk Management




                                          Weather and Radar Processor
                 Key Practice                             Finding                                         Rating
Verification 1   The activities for acquisition risk      While the Integrated Product Team leader     Weakness
                 management are reviewed by acquisition   reviews the status of the contract and the
                 organization management on a periodic    contractor’s cost and schedule, he does not
                 basis.                                   review the status of the activities that are
                                                          required to be performed for any of the key
                                                          process areas.
Verification 2   The activities for acquisition risk      While the product team leader reviews the       Weakness
                 management are reviewed by the project   status of the contract and the contractor’s
                 manager on both a periodic and           cost and schedule, he does not review the
                 event-driven basis.                      status of the activities that are required to
                                                          be performed for any of the key process
                                                          areas.


                                     FAA’s software acquisition risk management process has many weaknesses
Conclusions                          and is, therefore, undefined and undisciplined. To become an effective
                                     acquirer of software, FAA must adopt a more structured, rigorous, and
                                     disciplined approach to ATC modernization software acquisition risk
                                     management.




                                     Page 126                                              GAO/AIMD-97-47 Air Traffic Control
Chapter 10

FAA Lacks an Effective Approach for
Improving ATC Modernization Software
Acquisition Processes
                       In order to be effective, the organization responsible for improving
                       software acquisition processes must have (1) organizational and/or
                       budgetary authority over units acquiring software; and (2) a
                       comprehensive plan to guide software acquisition process improvement
                       efforts and measure progress on each. At FAA, responsibility for ATC
                       software acquisition process maturity and improvement has been assigned
                       to three organizational entities over the last 4 years, and currently is
                       assigned to FAA’s Software Engineering Process Group (SEPG), a multilevel
                       committee structure chaired by a member of FAA’s Chief Information
                       Officer’s (CIO) staff. However, the SEPG, like the entities previously
                       responsible for software acquisition process improvement, has neither
                       organizational nor budgetary authority over the ATC modernization product
                       teams that acquire software. Further, the SEPG does not have a software
                       acquisition process improvement plan to guide its maturation efforts.

                       As a result, management of FAA’s software acquisition process
                       improvement effort is ad hoc and has not instilled software acquisition
                       process discipline into the ATC modernization program. While the SEPG is
                       now taking steps to establish the analytical basis for a defined and
                       comprehensive software process improvement plan, that plan does not yet
                       exist, no schedule has been established for completing it, and the SEPG,
                       like the organizational entities before it that have failed to institute
                       process improvements, has no authority to implement or to enforce
                       process change.


                       FAA has been attempting to strengthen its software acquisition processes
FAA Organization       since 1993. At that time it established the Software Engineering Specialty
Responsible for ATC    Group to, among other things, incrementally improve FAA’s software
Software Acquisition   acquisition processes, first establishing repeatable processes [SEI maturity
                       level 2], then defined processes [SEI maturity level 3], and eventually
Process Improvement    managed processes [SEI maturity level 4]. This group was to be FAA’s focal
Lacks Authority to     point for software process assessment and improvement initiatives
                       through 2002, and it developed a 10-year strategy and implementation plan
Affect Change          for doing so. It also produced guidance addressing software acquisition
                       management, software management indicators, and other software-related
                       processes and practices.

                       In 1995, FAA established the Office of the Chief Scientist for Software
                       Engineering under FAA’s CIO to lead FAA’s software-related process
                       improvement efforts, including software acquisition. According to FAA
                       officials, this change was made to strengthen software process



                       Page 127                                      GAO/AIMD-97-47 Air Traffic Control
Chapter 10
FAA Lacks an Effective Approach for
Improving ATC Modernization Software
Acquisition Processes




improvement through increased interaction with the ATC modernization
project offices. In May 1995, the Chief Scientist reaffirmed SEI’s CMM as the
basis for all FAA process improvement and set forth three broad “strategies
for improving the quality of FAA software.”1 The three strategies were
(1) “improve the software acquisition, development, certification, and
maintenance processes of the FAA and its suppliers,”2 (2) “accelerate the
adoption of open systems, COTS, NDI,3 re-engineering, and reuse by FAA
programs (without jeopardizing system safety or integrity),”4 and
(3) “promote the use of best software engineering practices throughout the
FAA and its supplier community.”5 The Chief Scientist also began about a
dozen loosely defined process improvement activities. Examples of these
activities include participating in national and international software
engineering activities, interacting with governmental and professional
software engineering organizations, meeting with FAA suppliers and
aviation groups, and assessing software engineering methods, tools, and
best practices.

Another of the Chief Scientist’s dozen activities was to form the SEPG as
FAA’s“focal point for initiating, planning, motivating, and facilitating the
improvement of ’acquisition life cycle processes’ for software intensive
systems.”6 Formed in October 1995, the SEPG includes representatives from
ATC modernization product teams and their parent organizations as well as
other FAA organizations, and it is chaired by the Chief Scientist for
Software Engineering. The SEPG is directed by the Software Engineering
Executive Committee (SEEC), which is chaired by the Associate
Administrator for Research and Acquisition (i.e., the head of the ATC
modernization), and is composed of senior FAA executives. The SEEC is
responsible for recommending and providing guidance on software
engineering issues. Additionally, some of the ATC modernization product
teams’ parent organizations have SEPGs.




1
 “Strategies and Tactics for FAA to Improve Software, CIP Steering Committee Meeting,” May 16, 1995,
page 8.
2
 See footnote 1.
3
 COTS is commercial, off-the-shelf; NDI is non-development item.
4
 See footnote 1.
5
 See footnote 1.
6
 In a document entitled “SEPG Purpose” Nov. 18, 1996, FAA defines a software intensive system as
“any system that is entirely software or whose principle (sic) functionality depends on the correct
functioning of software.”



Page 128                                                      GAO/AIMD-97-47 Air Traffic Control
                      Chapter 10
                      FAA Lacks an Effective Approach for
                      Improving ATC Modernization Software
                      Acquisition Processes




                      None of the organizations responsible for ATC modernization software
                      acquisition process improvement over the last 4 years, including the SEPG,
                      have had organizational and/or budgetary authority over the ATC
                      modernization product teams that acquire ATC software. As a result,
                      neither the SEPG nor its predecessor organizations have been positioned to
                      effectively and efficiently implement and enforce process changes.
                      Instead, they can only attempt to encourage and persuade product teams
                      to undertake process improvement activities.

                      To illustrate the ineffectiveness of this management structure, the Chief
                      Scientist proposed that each product team earmark 5 percent of its
                      software-related budgets to implement approved process improvement
                      initiatives. According to the Chief Scientist, however, the teams refused to
                      spend product team money on the FAA-wide software improvement
                      activities being proposed. One product team leader stated that the teams
                      are understaffed and there is not enough time and resources to support
                      these activities.


                      To properly focus and target software acquisition process improvements,
FAA Planning for      current process strengths and weaknesses (the capability baseline) must
Software Process      be fully identified and assessed, and an effective plan for systematically
Improvement Has Not   correcting weaknesses must be developed. At a minimum, this plan should
                      specify measurable goals, objectives, milestones, and needed resources,
Been Effective        should clearly assign responsibility and accountability for accomplishing
                      well-defined tasks, and should be documented and approved by FAA
                      leadership.

                      Despite 4 years of software acquisition process improvement effort, FAA
                      has not effectively baselined FAA’s ATC modernization software acquisition
                      processes, and has not developed a comprehensive plan for software
                      acquisition process improvement on the basis of this baseline. In 1995, the
                      Chief Scientist began an effort to assess software acquisition processes,
                      but completion of the effort was delayed 8 months and, because it lacked
                      the requisite depth and scope, it could not be used to produce an effective
                      baseline to guide improvement activities. Subsequent plans to perform
                      more detailed and comprehensive assessments were dropped.

                      FAA also has no comprehensive plan for software acquisition process
                      improvement. As a proxy, the Chief Scientist claims that a variety of
                      documents produced and activities conducted over the last 2 years
                      collectively form a complete and comprehensive plan. These include (1) a



                      Page 129                                      GAO/AIMD-97-47 Air Traffic Control
                        Chapter 10
                        FAA Lacks an Effective Approach for
                        Improving ATC Modernization Software
                        Acquisition Processes




                        document containing the “preliminary process improvement
                        recommendations” of a process improvement working group dated
                        September 4, 1996, (2) minutes of SEPG and SEEC meetings, (3) briefings on
                        software process improvement activities, and (4) the business plan for the
                        ATC modernization organization. However, these documents and activities,
                        which only briefly address process improvement initiatives, cite broad
                        strategies, and sometimes mention general schedules and resource needs,
                        do not constitute an effective plan. They do not specify well-defined and
                        measurable goals and milestones, assign responsibility, identify resource
                        needs for each initiative, or define how and when process improvements
                        will actually be implemented and enforced. Further, without a capability
                        maturity baseline, there is no analytical basis for selecting these initiatives.
                        According to product team officials, the modernization effort has no
                        software acquisition process improvement plan.


                        Since 1995, the Chief Scientist has planned or initiated a dozen activities.
Improvement             Some have never been started, some are behind schedule, and several
Initiatives Have Thus   have proceeded according to plan. For example, while the Chief Scientist
Far Not Instilled       planned to complete an assessment of ATC modernization software
                        acquisition capabilities using SEI level 2 and level 3 requirements by
Software Process        December 1996 and June 1997, respectively, these efforts were never
Discipline              performed. Other efforts are behind schedule. For example, software
                        engineering policies, guidance, and standards that were to be issued by
                        September 1996 are now scheduled for issuance the third quarter of
                        calendar year 1997; and a software life cycle tool that was to be developed
                        by October 1996 has been postponed indefinitely.

                        Several efforts have met their milestones and begun to build a foundation
                        for undertaking process improvements. For example, a software
                        engineering training plan was developed in May 1996, 1 month ahead of
                        schedule; product teams have been trained to use the SEI software
                        development CMM and the associated capability evaluation methodology to
                        evaluate contractors’ capabilities to develop software; one product team
                        used the results of a CMM evaluation as part of its source selection criteria;
                        and, according to the Chief Scientist, hundreds of members of various ATC
                        modernization product teams have been trained in software management
                        techniques, such as defining software processes, using software
                        management indicators, estimating software costs, and using standards
                        such as open systems standards.




                        Page 130                                        GAO/AIMD-97-47 Air Traffic Control
              Chapter 10
              FAA Lacks an Effective Approach for
              Improving ATC Modernization Software
              Acquisition Processes




              Other FAA activities relating to software process improvement include
              establishing an SEPG infrastructure, pilot testing selected software product
              and process metrics, creating a software quality assurance capability,
              reengineering configuration management processes, and studying
              software cost estimation tools. In addition, the SEPG and the Chief Scientist
              are now taking steps to establish an analytical basis for software
              acquisition process improvement. For example, the SEPG and the Chief
              Scientist intend to use the results of this GAO evaluation as the basis for
              planning future software acquisition process improvement activities. Also,
              the SEPG has begun to analyze the interrelationships among FAA’s various
              process improvement activities, link the activities to strategic goals and
              measurable outcomes, and explore ways to manage these activities in a
              coordinated fashion. Further, the Chief Scientist intends to formalize the
              results of these steps in a comprehensive plan of action, although no
              schedule has been set for doing so.

              However, none of these activities or steps, either individually or in the
              aggregate, have yet resulted in more repeatable, better defined, more
              disciplined ATC modernization software acquisition processes. In
              interviews, product team and SEPG officials confirmed that the software
              acquisition processes had not yet been improved. Instead, the activities
              have begun to lay a foundation for potential process improvements in the
              future.


              FAA has neither an effective management structure nor plan of action for
Conclusions   improving its software acquisition processes. As a result, software
              acquisition processes will remain immature and will not support effective,
              efficient, and economical acquisition of mission-critical software costing
              billions of dollars. Until responsibility and accountability for software
              acquisition process improvement is assigned to an FAA organizational
              entity with the requisite authority to affect process change, and until this
              organizational entity pursues a plan for change based on a complete and
              objective assessment of current process strengths and weaknesses, it is
              unlikely that significant ATC modernization software acquisition process
              improvements will be made, and ATC software acquisition processes will
              remain immature.




              Page 131                                      GAO/AIMD-97-47 Air Traffic Control
Chapter 11

Overall Conclusions and Recommendations


                      Leading software acquisition organizations rely on defined and disciplined
                      software acquisition processes to deliver promised software capabilities
                      on time and within budget, first on a project-by-project basis, and later, as
                      the organization’s processes become more mature, consistently across the
                      institution. FAA’s ATC modernization software acquisition processes are ad
                      hoc and sometimes chaotic, and are not repeatable even on a
                      project-by-project basis. As a result, FAA’s success or failure in acquiring
                      ATC software depends largely on specific individuals, rather than on
                      well-defined and disciplined software acquisition management practices.
                      This greatly reduces the probability that software-intense ATC
                      modernization projects will consistently perform as intended and be
                      delivered on schedule and within budget. For FAA to mature beyond this
                      initial level, it must implement basic management controls and instill
                      self-discipline in its software projects.

                      FAA recognizes the importance of software process maturity and the need
                      to improve its software acquisition processes. However, it lacks an
                      effective management structure for accomplishing this because the FAA
                      organization responsible for process improvement, the SEPG, lacks the
                      authority to implement management controls and instill process discipline
                      within the ATC modernization software acquiring organizations.
                      Additionally, despite 4 years of FAA process improvement activity, no
                      well-targeted, comprehensive, and coordinated plan of action for software
                      acquisition process improvement exists.


                      Given the importance and the magnitude of information technology at FAA,
Recommendations       we reiterate our recent recommendation that a CIO organizational structure
                      similar to the department-level CIOs as prescribed in the Clinger-Cohen Act
                      of 1996 be established for FAA.1

                      To improve FAA’s software acquisition capability for its ATC modernization
                      and thereby take the first step in institutionalizing mature acquisition
                      processes, we recommend that the Secretary of Transportation direct the
                      FAA Administrator to:


                  •   assign responsibility for software acquisition process improvement to
                      FAA’s CIO;
                  •   provide FAA’s CIO the authority needed to implement and enforce ATC
                      modernization software acquisition process improvement;

                      1
                       Air Traffic Control: Complete and Enforced Architecture Needed for FAA Systems Modernization
                      (GAO/AIMD-97-30, Feb. 3, 1997).



                      Page 132                                                   GAO/AIMD-97-47 Air Traffic Control
                         Chapter 11
                         Overall Conclusions and Recommendations




                     •   require the CIO to develop and implement a comprehensive plan for ATC
                         modernization software acquisition process improvement that is based on
                         the software capability evaluation results contained in this report and
                         specifies measurable goals and time frames, prioritizes initiatives,
                         estimates resource requirements, and assigns roles and responsibilities;
                     •   allocate adequate resources to ensure that these improvement efforts are
                         implemented and enforced; and
                     •   require that, before being approved, every ATC modernization acquisition
                         project have software acquisition processes that satisfy at least SA-CMM
                         level 2 requirements.


                         In its written comments on a draft of this report, the Department of
Agency Comments          Transportation recognized the importance of mature software acquisition
and Our Evaluation       processes, agreed that FAA’s processes are insufficiently mature,
                         acknowledged that FAA process improvement activities have yet to
                         produce greater process discipline, and reaffirmed FAA’s commitment to
                         improving its software acquisition capabilities using the SA-CMM. However,
                         the Department did not state what, if any, specific action it would take on
                         our recommendations. Additionally, the Department expressed several
                         concerns, each of which is presented below, along with our rebuttal.

                         First, the Department stated that FAA does not separately procure software
                         for its ATC systems. Rather, it procures systems that use software as a
                         major component. Therefore, its policies and procedures (i.e., processes)
                         are “geared” to system acquisitions, and evaluating only the
                         software-related aspects of its acquisition processes “is not an adequate
                         approach.”

                         GAO  Rebuttal: All major system modernizations, like the ATC modernization,
                         involve the acquisition of hardware, software, and firmware operating
                         interdependently. However, as FAA’s own experience with the Advanced
                         Automation System clearly proves, the software component is the source
                         of most system risk, and the component most frequently associated with
                         late deliveries, cost increases, and performance shortfalls. Moreover, there
                         is widespread recognition throughout the computer industry that the
                         billions of dollars being invested in complex, real-time, fault tolerant
                         systems, like FAA ATC systems, are jeopardized by inadequate management
                         attention to software in general, and undisciplined, ill-defined software
                         acquisition and development processes in particular. This is precisely why
                         SEI developed its software-related CMMs, why the CMMs have been endorsed
                         and accepted throughout industry and government, and why the scope of



                         Page 133                                      GAO/AIMD-97-47 Air Traffic Control
Chapter 11
Overall Conclusions and Recommendations




this evaluation focused on software acquisition processes rather than on
broader systems issues. Further, the FAA system acquisition policies and
procedures that the Department references do not explicitly or adequately
address software issues. For example, they do not address
software-specific acquisition planning for such KPAs as contract tracking
and oversight, requirements development and management, and risk
management. Additionally, they do not provide for measuring and
verifying the performance of software-specific acquisition activities.

Second, the Department commented that the SA-CMM “is not widely used,
adopted, or validated” and, while it has “significant merit, it is certainly not
to be taken as the same authoritative source for process improvement
guidance as the SW-CMM,2 which has been in use worldwide by thousands of
organizations for several years.”

GAO  Rebuttal: This position is clearly inconsistent with the Department’s
and FAA’s stated commitment to using the SA-CMM as the basis for efforts to
improve FAA’s software acquisition capabilities. More important however is
that this position is without substance. The SA-CMM does not promulgate
original or novel concepts of debatable value. Instead, it presents as
requisite processes and practices those activities that common sense have
validated as essential to effective software acquisition. For example, it
requires disciplined requirements development and management,
solicitation, contract tracking and oversight, and evaluation. Our
evaluations have for years made the same points without rational dispute.
The SA-CMM simply provides a coherent framework and standard
terminology for these concepts. The findings in this report, which have
been corroborated by SEI, are compelling not because of the age of the
model used, but because the criticality of the processes and practices
examined is undeniable.

Third, the Department claimed that “GAO may have misapplied the model”
by (1) giving inadequate consideration to equivalent alternative practices
when determining whether SA-CMM specified practices were performed
(e.g., DSR system acquisition planning being judged as an insufficient proxy
for software acquisition planning specified in the SA-CMM), (2) not
effectively tailoring the SA-CMM to focus only on project activities that
occurred after the cancellation of the Advanced Automation System, and
(3) reporting evaluation results in a way that “does not create an
environment conducive to process improvement” (i.e., reporting the


2
 SW-CMM is SEI’s software development capability maturity model.



Page 134                                                   GAO/AIMD-97-47 Air Traffic Control
Chapter 11
Overall Conclusions and Recommendations




results for each project, rather than either aggregating the results or
disguising the identity of the projects).

GAO  Rebuttal: We applied the model properly and correctly, and SEI has
attested to this. Every member of our evaluation team was trained by SEI;
the team leader was an SEI designated “lead evaluator” and has been
authorized by SEI to submit results for inclusion in SEI studies; three senior
SEI professionals, two of whom are authors of the SA-CMM, participated in
the evaluation to ensure that the model was properly used; and SEI
concurred with each practice determination in the report (e.g., strength,
weakness). With respect to each of the Department’s subsidiary points
regarding our application of the model:

(1)During the course of extensive interviews with FAA designated officials,
no evidence of reasonable alternative practices was provided. If such
evidence had been provided, we would have considered it. For example,
when FAA provided a system acquisition plan for DSR as evidence of
software acquisition planning, we reviewed the document and found that it
was not a reasonable alternative practice because it did not adequately
address the software component of the acquisition.

(2)As agreed with SEI, those practices that were deemed inapplicable were
not rated, and those performed years ago were so designated. Moreover,
even if all practices predating the Advance Automation System’s
cancellation were ignored, none of our conclusions and recommendations
would change.

(3)The model and evaluation method do not specify any reporting format.
In particular, they do not address whether results should be reported for
each project, or whether the identity of the projects should be disguised or
results reported only in the aggregate. Given the mission-critical
importance and billion dollar cost of these projects, full disclosure of all
relevant facts to the Congress and the public is both warranted and
appropriate.

Fourth, concerning FAA’s software acquisition process improvement
efforts, the Department stated that the report does not sufficiently
appreciate the “progress made to date, the difficulties involved in
achieving that progress, and the time that it takes for . . . changes of
this . . . magnitude.” Specifically, the many efforts underway are not a
“hodge podge” of activities, but are “a very healthy sign of the seriousness
and enthusiasm” that FAA assigns to process improvement and are



Page 135                                       GAO/AIMD-97-47 Air Traffic Control
Chapter 11
Overall Conclusions and Recommendations




“organized with respect to specific directorate objectives.” Also, since
FAA’s process improvement activities have been underway for less than 2
years, it is too early to expect results.

GAO Rebuttal: The Department’s position that FAA’s many process
improvement activities are not a “hodge podge,” but rather are part of an
organized and coordinated comprehensive plan of action, is not supported
by the facts. While FAA began drafting a plan during the course of our
evaluation, it had no schedule for finalizing this plan, and no analytical
basis for the software acquisition process improvement activities
underway. Just as its software acquisition processes lack maturity and
discipline, so do its efforts to improve these processes. Claims that FAA has
been engaged in software improvement efforts for less than 2 years, and
thus it is too early to evaluate results, are also unsupported. In fact,
software acquisition process maturity and improvement efforts began in
1993. Since SEI published statistics show that the median time to improve
from SW-CMM level 1 to level 2 is 26 months, and from SW-CMM level 2 to level
3 is 17 months, it is entirely reasonable to expect FAA to be able to
demonstrate some improvement in its processes after 4 years.

Fifth, while the report states that the SEPG has neither the organizational
nor the budgetary authority to effectively implement process change, the
Department stated that its “understanding . . . is that organizations do not
normally give their SEPG authority over product teams.” In FAA’s case, the
SEPG provides advice and counsel to the Software Engineering Executive
Committee, which consists of senior managers who have authority and
responsibility to direct process improvement actions. The SEPG is the
committee’s agent for implementing these improvements.

GAO Rebuttal: The issue is not whether FAA’s SEPG is organized as the
Department “understands” other SEPGs to be organized, but whether the
SEPG, or any FAA organizational entity responsible for implementing and
enforcing software process change, has the authority needed to
accomplish this task. Currently, no organizational entity in FAA has the
requisite authority. Accordingly, we have recommended that a CIO
organizational structure similar to the department-level CIOs prescribed in
the Clinger-Cohen Act of 1996 be established for FAA, and that it be
assigned the responsibility and resources needed to affect and enforce
software acquisition process improvement.




Page 136                                       GAO/AIMD-97-47 Air Traffic Control
Chapter 11
Overall Conclusions and Recommendations




Sixth, the Department contends that the report “leads the reader to
erroneously conclude that the five programs reviewed are in trouble”
relative to their cost and schedule goals.

GAO Rebuttal: The report addresses the maturity of FAA’s software
acquisition processes and concludes that these processes are ad hoc and
undisciplined, reducing the probability that software-intense ATC
modernization projects will consistently perform as intended and be
delivered on schedule and within budget. The report does not address the
overall status of the projects covered by GAO’s review, and, therefore,
provides no basis for drawing conclusions about the projects’ overall cost
and schedule performance.




Page 137                                     GAO/AIMD-97-47 Air Traffic Control
Appendix I

Comments From the Department of
Transportation




             Page 138        GAO/AIMD-97-47 Air Traffic Control
Appendix I
Comments From the Department of
Transportation




Page 139                          GAO/AIMD-97-47 Air Traffic Control
Appendix I
Comments From the Department of
Transportation




Page 140                          GAO/AIMD-97-47 Air Traffic Control
Appendix I
Comments From the Department of
Transportation




Page 141                          GAO/AIMD-97-47 Air Traffic Control
Appendix I
Comments From the Department of
Transportation




Page 142                          GAO/AIMD-97-47 Air Traffic Control
Appendix I
Comments From the Department of
Transportation




Page 143                          GAO/AIMD-97-47 Air Traffic Control
Appendix II

Major Contributors to This Report


                       Randolph C. Hite, Senior Assistant Director
Accounting and         Keith A. Rhodes, Technical Director
Information            Leonard J. Latham, Technical Assistant Director
Management Division,   Suzanne M. Burns, Senior Information Systems Analyst
                       Madhav S. Panwar, Senior ADP/Telecommunications Analyst
Washington, D.C.       Ira S. Sachs, Senior Information Systems Analyst




(511487)               Page 144                                GAO/AIMD-97-47 Air Traffic Control
Ordering Information

The first copy of each GAO report and testimony is free.
Additional copies are $2 each. Orders should be sent to the
following address, accompanied by a check or money order
made out to the Superintendent of Documents, when
necessary. VISA and MasterCard credit cards are accepted, also.
Orders for 100 or more copies to be mailed to a single address
are discounted 25 percent.

Orders by mail:

U.S. General Accounting Office
P.O. Box 6015
Gaithersburg, MD 20884-6015

or visit:

Room 1100
700 4th St. NW (corner of 4th and G Sts. NW)
U.S. General Accounting Office
Washington, DC

Orders may also be placed by calling (202) 512-6000
or by using fax number (301) 258-4066, or TDD (301) 413-0006.

Each day, GAO issues a list of newly available reports and
testimony. To receive facsimile copies of the daily list or any
list from the past 30 days, please call (202) 512-6000 using a
touchtone phone. A recorded menu will provide information on
how to obtain these lists.

For information on how to access GAO reports on the INTERNET,
send an e-mail message with "info" in the body to:

info@www.gao.gov

or visit GAO’s World Wide Web Home Page at:

http://www.gao.gov




PRINTED ON    RECYCLED PAPER
United States                       Bulk Rate
General Accounting Office      Postage & Fees Paid
Washington, D.C. 20548-0001           GAO
                                 Permit No. G100
Official Business
Penalty for Private Use $300

Address Correction Requested