oversight

FAA Procurement: Major Data-Processing Contract Should Not Be Awarded

Published by the Government Accountability Office on 1990-05-25.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

)(                                                              .*
     *,   IJnited States General Accounting   Office                 i
I
          Report to the Chairman, Subcommittee
          on Transportation and Related
          Agencies, Committee on
          Appropriations, House of
          Representatives

          FAA PROCUREMENT
          Major Data-Processing
          Contract Should Not
          Be Awarded


                                                       141576
Information Management and
Technology Division

B-234903
May 25,199O

The Honorable William Lehman
Chairman, Subcommittee on Transportation
  and Related Agencies
Committee on Appropriations
House of Representatives
Dear Mr. Chairman:
This report responds to your request that we review the Federal Aviation Administration’s
Computer ResourcesNucleus project. It presents our analysis, conclusions,and
recommendations regarding the project. In March 1989, we reported on the project’s
objectives, cost estimates, and implementation approach.
As arranged with your office, unless you publicly release its contents earlier, we plan no
further distribution of this report until June 15,199O.At that time, we will send copies of
the report to the Secretary of Transportation; the Administrator, Federal Aviation
Administration; the Director, Office of Management and Budget; the Administrator of
General Services;interested congressionalcommittees; and other interested parties.

This work was performed under the direction of JayEtta Hecker, Director, Resources,
Community, and Economic Development Information Systems,who can be reached at (202)
275-9676. Other major contributors are listed in the appendix.
Sincerely yours,




Ralph V. Carlone
Assistant Comptroller General
lihecutive Summary


                   In February 1989, the Federal Aviation Administration (FAA) issued a
Purpose            request for proposals for its largest and most complex general-purpose
                   data-processingacquisition to date: the Computer ResourcesNucleus
                   (CORN) project. This project is intended to meet the agency’s general-
                   purpose data-processingneedsfor 10 years and provide options for sup-
                   porting the processingneedsof other parts of the Department of Trans-
                   portation at an estimated cost of $1.6 billion. At the request of the
                   Chairman of the Subcommitteeon Transportation and Related Agencies,
                   House Committee on Appropriations, GAO reviewed FAA'S assessmentsof
                   its data-processingproblems and future needsused to justify CORN, its
                   methodology for validating vendors’ proposals, and its planning and
                   preparation for converting current applications’ software.

                   Currently, FAA'S 12 in-house “Common System” computer facilities pro-
Background         vide general-purposedata processingfor mission and administrative
                   areas such as airport and aviation activity (excluding real-time air traf-
                   fic control), aviation safety, national airspace facilities, and financial,
                   materiel, and human resources.Agency officials maintain that the Com-
                   mon System is not meeting FAA'S current processingneedsbecauseof
                   capacity and responsetime problems, and that it is not possible or desir-
                   able to meet long-term needsby upgrading this system.

                   To resolve these problems, FAA developed the CORN project, under which
                   a contractor would provide data processingon a fee-for-service basis for
                   up to 10 years from computer facilities owned, operated, and main-
                   tained by the contractor. The Common System would be closed down
                   once current applications systems are moved to the new system. In addi-
                   tion to meeting FAA'S anticipated needs,the project includes options for
                   providing service to other elements of the Department of Transporta-
                   tion. FAA estimates that the contract, scheduledto be awarded in the
                   summer of 1990, has a value of $1.5 billion.

                   The CORN project has not been properly justified and planned, and con-
Results in Brief   tains major unresolved problems. FAA'S claims about the causesof per-
                   ceived problems with its current system are poorly supported, as are its
                   projection of future needs,leading to fundamental doubts about the pro-
                   ject’s justification. In addition, the agency’smethodology for evaluating
                   technical and cost aspectsof vendor proposals is seriously flawed, a
                   problem that could have cost ramifications. Further, FAA'S estimates of
                   the cost of converting software to CORN and the amount of agency sup-
                   port neededfor the conversion are unreliable. The estimated time frame


                   Psge2
                                  Executive summary




                                  for the conversion has already doubled, from 18 months to 3 years, lead-
                                  ing to more cost growth. Finally, the conversion itself will not result in
                                  better managementinformation.


Principal Findings

FAA’s Justification for           In documents and briefings provided to the Congressand the Depart-
CORNIs Unsubstantiated            ment of Transportation, FAA has repeatedly cited CommonSystem
                                  capacity and responsetime problems as key justifications for CORN.
                                  However, FAA is unable to provide support for these perceived problems
                                  becauseit lacks a central capacity and performance managementpro-
                                  gram to gather and analyze data on system-wide utilization and
                                  responsetimes. GAO'S own analysis of available system monitoring data
                                  shows no evidence of computer processor capacity conditions that
                                  would causeresponsetime problems, indicating that perceived response
                                  time problems may not be solved by procuring more computer process-
                                  ing capacity through CORN. FAA itself has not determined the specific
                                  causesof such problems.
                                  FAA  also justifies CORN on the grounds that its future general-purpose
                                  data-processingneedswill increase at a rate of 30 percent per year over
                                  10 years, resulting in a system about 1300 percent larger than the cur-
                                  rent one. However, GAO found that this estimate is basedon sparse data,
                                  raising doubts about whether such extremely steep growth will occur.


Project’s Validation              FAA  needsmeaningful and accurate information to evaluate the most
Methodology Is Flawed             effective way of meeting agency requirements. However, one key FAA
                                  method used in validating the vendors’ CORNproposals is deficient
                                  becausethe sample work load to be used is extremely small and unrep-
                                  resentative of the agency’stotal work load. In addition, the performance
                                  information on the current system that FAA provided to the vendors to
                                  assist them in developing their proposed solutions was incomplete. Fail-
                                  ure to accurately validate vendors’ proposed solutions could have cost
                                  ramifications throughout the life of the contract.


Major Problerks Loom Over Converting computer application software to run on a new system is a
CORNConversion            costly, disruptive task that requires careful preparation. FAA'S efforts to
                          plan for the CORN conversion have been marred by its unreliable initial


                                  Page g                  GAO/IMTEC90-28 FM CORNContract Should Not Be.Awarded



                                                                    ’ ,
                  inventory of applications to be converted, calling into question the basic
                  accuracy of the conversion cost estimate of $74.6 million. The number of
                  applications to be converted has varied from 600 to 200, while estimates
                  of the amount of codeto be converted have varied from 18 million to 10
                  million lines. More seriously, the conversion cost estimate is incompiete
                  becausethe potentially substantial costs of rectifying problems with the
                  applications’ documentation were not included. FAA, nevertheless, con-
                  tinues to use this incomplete estimate in CORN budget information pro-
                  vided to the Congress.
                  FAA  states that it will need to provide 86 employee-yearsof staff sup
                  port to assist the CORN contractor in converting current applications to
                  the new system. However, the basis for this estimate is questionable
                  becauseit is not grounded in a review of each individual application to
                  take into account the specific characteristics, complexities, or problems
                  associatedwith converting each one. In addition, FAA has not determined
                  the extent to which staff actually will be available to support each con-
                  version. The estimated time frame for conversion has doubled from 18
                  months to 3 years, resulting in additional costs, since the current Com-
                  mon System is to run in parallel with CORN   during the conversion. These
                  parallel costs for a 3-year conversion period could be as much as $106
                  million. This cost is not part of the CORN project cost estimates.
                  Further, the agency has stipulated that the application code is to be con-
                  verted without functional changeto the applications. As a result, the
                  conversion will not enhancethe application code,improve its efficiency,
                  or result in better information. In addition, FAA has done no analysis to
                  determine whether specific applications are worth converting at all.
                  Existing problems, such as poor responsetime, that are associatedwith
                  FAA'S information practices and computer applications may simply be
                  transferred to CORN, at considerable expense.

                  GAO recommendsthat the Secretary of Transportation direct that the
Recommendations   CORN contract not be awarded. In addition, GAO recommendsthat the Sec-
                  retary direct the FAA Administrator to ensure that future procurements
                  of this type and magnitude be properly justified and planned.

                  Department of Transportation and FAA officials state that CORN should
Agency Comments   be assessedon its planned benefits-such as enabling FM to better man-
                  age its information resources,provide quality services as needed,and
                  perform its mandated missions. Although expressing general agreement
Ehecutive sulnmary




with the facts presented in GAO'S report, they believe that GAO has
judged CORN on grounds that are not fundamental to its purpose and not
material in the larger context. They maintain that CORN has been suffi-
ciently planned and justified to warrant its award. GAO maintains that
its findings deal with problems and deficiencies that are fundamental to
the project’s scopeand implementation and, consequently, support the
conclusion that the contract should not be awarded.
In responseto GAO'S recommendation not to award CORN, the FAA Admin-
istrator has ordered an independent review of the project. The House
Committee on Appropriations has directed the Department and FAA not
to award the CORN contract until (1) the Committee reviews the results
of GAO'S report and FAA'S written responseto it and (2) FAA and Depart-
ment officials subsequently discussthe project with the Committee to
resolve any outstanding concerns.




Page 5                 GAO/IMTEG90-38 FM CORN Contract Should Not Be Awarded
Executive Summary
Chapter I                                                                                          8
Introduction            How FAA Hopes CORNWill Meet Its Needs
                        Optional Levels of Service
                                                                                                   8
                                                                                                  10
                        Project Length, Estimated Cost, and Status                                10
                        Objectives,Scope,and Methodology                                          10

Chapter 2                                                                                         13
CORN Has Not Been       CORNIs Intended to Solve Perceived Chronic System
                            Problems
                                                                                                  13
Justified               FM Lacks Data on Current System Performance                               14
                        Limited Data Available on ResponseTime Problems                           16
                        lo-Year Growth Projection Is Basedon Sparse,                              18
                            Inadequate Data
                        The Importance of Accurate System Sizing                                  20

Chapter 3                                                                                         22
CORNEvaluation          Performance and Capability Validation Are Critical for
                            Large Acquisitions
                                                                                                  22
Methodology Is Faulty   FAA’s Validation Strategy Will Not Adequately Test the                    23
                            ProposedSystems

Chapter 4                                                                                         26
Major Uncertainties     Conversion Cost Estimate Is Unreliable
                        Staff Support for Conversion Inadequately Determined
                                                                                                  26
                                                                                                  29
Loom Over Conversion    Estimated Conversion Time Frame Has Doubled                               31
                        Conversion Will Not Result in Better Information                          32

Chapter 5                                                                                         34
Conclusions and         Recommendationsto the Secretary of Transportation
                        Agency Comments
                                                                                                  36
                                                                                                  36
Recommendations
Appendix                Appendix I: Major Contributors to This Report                             38

           Y




                        Page 6                 GAO/WTECW@    FM CORN Contract   Should Not Be Awarded
    Contenta




    Abbreviations
    CORN       Computer ResourcesNucleus
    DOT        Department of Transportation
               Federal Aviation Administration
Y

    FAA
    GAO        General Accounting Office
    IMTEZC     Information Managementand Technology Division


    Page 7                 GAO/lMTJW$bO-88 FM COIW Contract Should Not Be Awarded
Chapter I

Introduction


                        On February 27,1989, the Federal Aviation Administration (FAA) issued
                        a request for proposals for its largest and most complex general-purpose
                        data-processingacquisition to date: the Computer ResourcesNucleus
                        (CORN) project. CORN is intended to meet the agency’sgeneral-purpose
                        data-processingneedsfor up to 10 years in the following mission and
                        program areas:

                    l airport and aviation activity;
                    l air traffic control and airspace (excluding real-time air-traffic control
                      systems);
                    . aviation safety;
                    l national airspace system facilities;
                    l financial, materiel, and human resources;and
                    l managementsupport.
                        FAA currently supports these areas with its own in-house, general-pur-
                        pose data-processingresourcescalled the “Common System.” The Com-
                        mon System is made up of one International BusinessMachines 3084
                        computer and 22 Data General MV/16000 computers distributed among
                        12 agency facilities: headquarters, 9 regional offices, and 2 centers. The
                        major hardware componentsof this system were installed and upgraded
                        during the 1980s.
                        In March 1989, we issued a report that provided information about the
                        CORNproject’s objectives, cost estimates, and implementation approach.1
                        The report highlighted the tenfold increase in estimated project cost-
                        from $148 million to $1.6 billion-that occurred as the project evolved
                        between 1986 and 1987.

                        In project documents supporting the need for CORN, project officials
How FAA Hopes           maintain that the agency’scurrent general-purposedata-processing
CORNWill Meet Its       needs are not being met by the Common System. Specifically, they assert
Needs                   the capacity of the Common System is saturated, causing average
                        responsetimes of 4 secondsthat result in substantial loss of staff pro-
                        ductivity. They estimate that FAA'S general-purposedata-processing
                        needswill grow at a rate of 30 percent per year over the next 10 years.
                        They argue that this growth rate far exceedsany feasible timetable for
                        expanding the Common System becauseof the agency’slengthy procure-
                        ment process.Finally, they maintain that it is not desirable to upgrade

                                 rPnxurement: FAA's$1.6-BiUionComputerReaourcesNudeus~ject(GAO/
                                  -44F?$Mar.31,19~



                        Page 8                     GAO/IMTECfJO-3fl   FM CORN Contract Should Not Be Awarded
chapter I
Intzwduction




the current CommonSystem becauseit is a hedge-podgeof widely vary-
ing equipment that is inefficient and wasteful of critical staff and fund-
ing resources.
CORN   project officials have concludedthat in order to meet FAA’s pro-
jected growth in data-processingneedsover the next decade,the agency
must free itself from the administrative and technical burdens involved
in acquiring, managing, and operating its own general-purposecomputer
facilities. Consequently, under the CORN approach, FAA plans to procure
data-processingservicesfrom a contractor and closedown its Common
System operations. The work of FAA’S information systems staff would
shift from managing an in-house system to helping agency users meet
their information resourceneeds.
The CORN  approach calls for FAA to define its general-purposedata-
processingneedsfor the next 10 years. On the basis of this information,
the contractor-not FAA-determines the computer configuration most
appropriate and cost-effective for meeting these projected needs.The
contractor is to provide, operate, and maintain the new system, which is
to be located in at least two contractor-provided facilities that communi-
cate with each other and with FAA centers, regions, and offices, as well
as other Department of Transportation locations. The contractor is to
provide the agency with specified levels of data-processingserviceson a
fee-for-service basis.
In addition, the contractor is to convert the current CommonSystem
applications’ software and data to the new system on a fixed-price basis;
prepare full documentation for the converted applications; provide tech-
nical staffing, support, and training to agency users of the system; select
and managesubcontractors; and implement system upgrades.
Project officials maintain the CORN approach will allow the agencyto
easily order increasing levels of data-processingservice over the next 10
years just as if it were a basic utility, such as electrical service. System
users, however, will be required to internally budget and pay for their
use of CORN, something they do not do in obtaining data-processingfrom
the Common System. Consequently,the amount of service that FAA pro-
gram offices would receive under CORN would depend on their ability to
pay for it. On a broader level, project officials maintain that CORN will
promote data-processingstandardization, and data integration and
integrity.




Page 9                   GAO/IMTEG90-38 FM CQRN Contract Should Not Be Awarded
                        chapter1
                        Introdnction




                        CX~RNincludes options  for processingapplications other than those cur-
Optional Levels of      rently run on the Common System. These additional applications include
Service                 FAA data-processingrequirements that are not part of the Common Sys-
                        tem, as well as data-processingneedsof other elements of the Depart-
                        ment of Transportation, such as the Coast Guard, the Federal Highway
                        Administration, and the Federal Railroad Administration. Detailed
                        requirements, feasibility, and cost/benefit studies have not been done
                        for the optional levels of service. Instead, they are to be done before
                        implementing these options.

                        The CORN contract is expected to cover an initial S-year implementation
Project Length,         period, followed by five l-year renewals. FAA estimates the total con-
Estimated Cost, and     tract value to be about $1.6 billion: $876 million to meet FAA processing
Status                  requirements, and $619 million to meet optional data-processing
                        requirements.
                        The Secretary of Transportation designated CORN as a major systems
                        acquisition in September 1987. The project was reviewed by the Trans-
                        portation SystemsAcquisition Review Council and approved by the
                        Deputy Secretary of Transportation in January 1989. FAA issued the
                        CORN request for proposals on February 27,1989. Vendors’ cost and
                        technical proposals were due by August 30,1989. FAA is currently
                        reviewing the proposal material submitted and anticipates that the CORN
                        contract will be awarded in the summer of 1990. However, in the House
                        Committee on Appropriations report supporting the fiscal year 1990
                        appropriations bill for the Department of Transportation, FAA and the
                        Department were directed to defer awarding the CORN contract until (1)
                        the Committee reviews the results of our evaluation of CORN and a writ-
                        ten FAA responseto our report and (2) FAA and Department officials sub-
                        sequently discussthe project with the Committee to resolve any
                        outstanding concerns.
                        Regardlessof the future of CORN, FAA plans to upgrade the mainframe
                        portion of the CommonSystem. The existing contract for the minicom-
                        puter portion of the system already has provisions for implementing
                        upgrades.

                        At the request of the Chairman, HouseCommittee on Appropriations,
Objectives,Scope, and   Subcommittee on Transportation and Related Agencies,we reviewed
Methodology             FAA'S proposed procurement of the CORN system. As agreed with the
                        Chairman’s office, our objectives were to


                        Page10                  GAO/WTEC-gO-8SFAAC0RNContmctShouldNotBeAwarded
    Chapter I
    hdXOdWtiOll




l   determine the adequacy of FAA's assessmentof its current and future
    data-processingneeds,
l   determine the adequacy of FAA'S methodology for validating the ven-
    dors’ proposed solutions for meeting agency data-processingrequire-
    ments, and
l   determine the adequacy of FAA'S preparation and planning for the con-
    version of current applications to the CORN system.
    To determine the adequacy of FAA'S assessmentof its current data-
    processingproblems and future needs,we met with officials at the
    Department of Transportation and FAA to discussCORN'S development,
    goals, objectives, estimated cost, implementation approach, and manage-
    ment. We also reviewed key documents-such as the CORN requirements
    analysis, feasibility study, mission need statement, project charter,
    request for proposals, and draft implementation plan-used by FAA and
    the Department of Transportation to review and approve the CORN
    approach. We analyzed CommonSystem performance monitoring tapes
    to determine the extent of capacity and responsetime problems, and
    reviewed other data on the system’s performance. We visited FAA'S Mike
    Monroney Aeronautical Center, where about half of the agency’sgen-
    eral-purpose data processingis performed, to discusssystem perform-
    ance issuesand review performance data.
    To determine the appropriateness of FAA'S validation methodology, we
    reviewed the rationale for the chosenapproach and the method by
    which it is to be implemented, met with officials of the General Services
    Administration’s Federal SystemsIntegration and ManagementCenter
    to discusstheir independent assessmentof the approach, and considered
    applicable federal regulations.
    To assessthe adequacy of preparation and planning for the CORN con-
    version, we reviewed the project’s 1987 conversion study, the original
    and revised CORN documentation package,the methodology for estimat-
    ing agency resourcesneededto support the conversion, and the manage-
    ment plan for implementing the conversion. We discussedthe conversion
    with officials at the General ServicesAdministration’s Federal Software
    ManagementSupport Center, FAA program offices, the CORN project
    office, and the Department of Transportation’s Office of the Inspector
    General.




    Page11
chapm      I
Introduction




We performed our work at the Department of Transportation, FAA, the
General ServicesAdministration in Washington, D.C.;the Federal Sys-
tems Integration and ManagementCenter, the Federal Software Manage-
ment Support Center in Falls Church, Virginia; and at FAA’S Mike
Monroney Aeronautical Center in Oklahoma City, Oklahoma.
Our review, performed from January 1989 to April 1990, was conducted
according to generally acceptedgovernment auditing standards. We
obtained the views of Department of Transportation and FAA officials on
this report and have incorporated them where appropriate.




Page 12                GAOAMTEWWS    FM CXlRN Contract Should Not Be Awarded
Chapter 2

CORNHas Not Eken Justified


                     While the concept of CORN-t0 contract for data-processingservices
                     from a vendor-may be acceptable,key justifications for the CORN pro-
                     ject are not properly supported, raising fundamental questions about the
                     rationale for the project. In promoting CORN, project officials maintained
                     that CommonSystem users were receiving inadequate responsetimes
                     becausethe system was “at capacity” or “saturated.” However, we
                     found no evidenceproving that perceived responsetime problems are
                     being causedby a lack of processorcapacity or would be solved by CORN.
                     FM has virtually no data on responsetimes indicating the frequency,
                     magnitude, and causeof such problems. Project officials also maintain
                     that the agency’s data-processingneedswill increase faster than its abil-
                     ity to upgrade the current system. However, the data and methodology
                     used by FAA to make its growth projections are inadequate, raising fur-
                     ther doubts about the necessity for the CORN approach.

                     Project officials have repeatedly maintained that FAA needsCORN
CORNIs Intended to   becausethe current Common System doesnot adequately meet the
Solve Perceived      agency’s needsand becausethe level of service provided by the system
Chronic System       is continuously degrading. Specifically, they maintain that the Common
                     System is “saturated” and that current capacity must be increased by
Problems             160 percent to achieve required service levels. The officials also claim
                     that the system’s responsetime is inadequate, averaging 4 secondsand
                     causing a productivity loss of $37.6 million a year. In addition, they
                     maintain that the agency’sgeneral-purposedata-processingneedswill
                     grow at a compounded rate of 30 percent per year, over the next 10
                     years.
                     Thesejustifications have been continually stressedin documents and
                     briefings used to explain why CORN is needed.For example, the system is
                     described as “repeatedly saturated” in a February 1989 “Project CORN
                     Basic Facts” briefing document. The draft CORN implementation plan
                     provided to the Department of Transportation in April 1989 stated that
                     the current system has reached its saturation level, suppressing service
                     to FAA personnel. The system was also described as “saturated” in a Sep-
                     tember 1989 briefing document prepared for the FAA Administrator.
                     Earlier project documents and briefings used to justify CORN also
                     assertedthat the CommonSystem was “overloaded,” “at capacity,” or
                     “saturated.” Briefings given to the Congressand to us have also stressed
                     that lack of computer capacity is causing inadequate responsetimes and
                     impairing staff productivity.




                     Page 13                 GAO/IMlEC30-33   FM CORN Contract Should Not Be Awarded
                    chapter 2
                    C43RNIIaa Not Been Judfled




                    Project officials were unable to provide us with adequate support for
FAA Lacks Data on   their assertions about capacity and responsetime problems becauseFAA
Current System      lacks a central capacity and performance managementprogram for the
Performance         Common System. Such a program is important to ensure maximum use
                    of existing resourcesand adequate capacity for growth. Not only is
                    capacity and performance managementa commonly-acceptedbusiness
                    practice, but the Federal Information ResourcesManagementRegulation
                    Part 201-30 requires agenciesto perform capacity managementactivi-
                    ties in planning, acquiring, and using computer resources.
                    An effective capacity managementand performance monitoring pro-
                    gram needsto addressboth performance managementand capacity
                    planning. Performance managementinvolves analyzing the performance
                    of computer systems to determine how resourcesare currently utilized
                    and how such utilization can be improved. Capacity planning assistsin
                    forecasting computer resource requirements to ensure that enough
                    capacity exists when needed.
                    While data are captured and analyzed on several componentsof the
                    CommonSystem, officials at both FAA and the Department of Transpor-
                    tation stated that there is no central capacity managementprogram for
                    the Common System that captures and analyzes Common System utiliza-
                    tion and responsetime data on a systemwide basis. According to project
                    officials, instead of having a central program, each of the 12 facilities is
                    responsible for independently conducting capacity planning.
                    We found, however, that the lack of a central capacity and performance
                    managementprogram has resulted in inadequate data on the system’s
                    utilization and responsetimes. Very little data on utilization and
                    responsetimes is available for the minicomputer portion of the system.
                    Performance monitoring is much more in evidence for the mainframe
                    portion of the system. But even there, project officials could not provide
                    us with critical performance information, especially on responsetimes,
                    that is key to justifying the need for CORN. Consequently, the officials
                    were unable to provide support for the key assertion that responsetime
                    problems were causedby insufficient capacity and therefore could be
                    solved by abandoning the current system and moving to CORN-a capac-
                    ity-oriented approach.
                    Project officials state that the current mixture of mainframe and min-
                    icomputers doesnot lend itself to centralized capacity management,and
                    that it is difficult to monitor the minicomputers’ performance. They



                    Page 14                      GAO-       FM CORN Ckmtract Should Not Ee Awarded
                        chqmr2
                        CORNIIaa Not Been JuMfled




                        claim that CORN will create an environment that will allow                    FAA   to have
                        centralized capacity management.

                        Project officials claim that insufficient processorcapacity is causing the
Limited Data            Common System to provide users with poor responsetimes, averaging 4
Available on Response   seconds,resulting in annual productivity lossesof $37.6 million. They
Time Problems           calculated that by reducing responsetime to a maximum of 2 seconds,
                        CORN would allow FAA to realize cost-avoidancesavings of $376 million
                        over 10 years due to improved productivity.
                        Project officials, however, could not provide support either for their
                        assertion that the CommonSystem’s averageresponsetime is 4 seconds
                        or for their assertion that perceived responsetime problems are caused
                        by a lack of Common System processorcapacity. Information resource
                        staff responsible for the operation of the mainframe system stated that
                        they had not collected overall responsetime data during the last 2 years
                        becauseof the architecture of the agency’stelecommunications system.’
                        Further, no quantifiable data were provided to support the assertion
                        that the averageresponsetime on the minicomputer systems is 4
                        seconds.

                        Since project officials were unable to provide us with performance moni-
                        toring data to support their assertionsthat the 12 Common System facil-
                        ities are at capacity and were the causeof poor responsetime, we
                        independently analyzed the limited processorutilization data that were
                        available. Specifically, we analyzed July 1989 and January 1990 utiliza-
                        tion data from the system’s mainframe facility, which processesabout
                        44 percent of the work load; and utilization data from February through
                        May 1987 for 4 of the 12 minicomputer facilities that processthe
                        remaining 66 percent.2
                        For high-priority interactive work loads,3the July 1989 data showed
                        that the mainframe processorwas at or below 69 percent utilization for


                        10ffichls responsiblefor the operationof FAA’smainframestatedthat they have decidedto start
                        collectingvery limited responsetime data The telecommunicationslines beingmonitoredrepresent
                        2.6 percentof the mainframeusers.
                        2The 1989and 1900data were provided at our request.The minicomputerutilization data from Feb-
                        ruary through May 1987are the mostrecentdata collectedby FM on thesemachines.
                        %Iigh priority interactive work loadsinclude thoseapplicationsprogramsthat require on-line
                        PITXXSS~@  and are consideredmostcritical to FM.



                        Page 16                         GAO-               FM C0lW Contract Should Not Be Awarded
chapter 2
CORN Has Not Been Ju&Uled




99 percent of the time between 7 a.m. and 7 p.m. on nonholiday week-
days. Similarly, the January 1990 data showed the mainframe was at or
below 67 percent utilization for 99 percent of the samehours for the
sametype of work load. It is highly unlikely that these levels of proces-
sor utilization would causepoor responsetimes.
Our analysis of utilization levels for the minicomputer portion of the
Common System yielded similar results. Basedon the available data, the
mean prime-time (8 a.m. to 4 p.m. on nonholiday weekdays) utilization
was between 16 and 26 percent for all of the selectedsites except for
one facility, the Aviation Standards National Field Office, where it was
46 percent.4Since 1987, when these data on the minicomputers were col-
lected, FAA has upgraded all of its minicomputers. The upgraded
machines have over 3.6 times the processingpower of the old ones,
according to industry specifications. Although FAA doesnot regularly
collect utilization data from the minicomputer facilities, the available
chargeback reports provide data on total processorutilization at each
data-processingfacility. Using extrapolations of the 1987 data and 1989
chargeback reports, we estimate that prime-time, processorutilization
was at or below 66 percent 99 percent of the time between 8 a.m. and 4
p.m. on nonholiday weekdays for 7 of the 8 facilities analyzed.6It is
highly unlikely that processorutilization at these levels is the causeof
responsetime problems.

Given the absenceof direct measurementsof responsetime, we analyzed
the interactive, time-sharing option subsystem of the mainframe and
found responsetimes of less than 1 second.6Agency officials said that
the time-sharing option subsystem is not heavily used and that response
time problems occurred in other subsystems,namely their data base
managementsystem and communications handlers. However, agency
officials were unable to provide us with data documenting the existence
of poor responsetimes in these subsystems.Moreover, we observedthat
the data base managementsystem and the communications handlers are

4Although usedin the CORNrequirementsanalysis,thesedata are sparseand incomplete.During the
Cmonth monitoring period,the selectedsites had gapsin their utilization data ranging from about 1
to 3 months.

6Chsrgebackreports are generatedmonthly by FAA and distributed to eachof its data-processing
facilities for information purposesonly. No chargesare collscted.The reports contain information
describingthe total resources(Le.,central processorunit time, memory,tape storage,etc.)usedby
eachfacility and the dollar amountthe facility would be billed for the useof thoseresources.Our
analysis of prim&me utilization is basedon the extrapolation of data from thesereports.
6Thetime-sharingoption subsystemIs an option of the International BusinessMachines3034operat-
ing systemthat allows usersto interactively sharecomputertime and resources.



Page 16                         GAO/IMTECfKMg       FM CORN Contract Should Not Be Awarded




                                         .I   :,
                              chapter 2
                              CORN Iiaa Not Been JustifM




                              assigneda higher processingpriority than the time-sharing option sub-
                              system. Therefore, if the lower priority time-sharing subsystem was not
                              suffering from poor responsetime resulting from a saturated processor,
                              it is unlikely that these other higher priority systems were suffering in
                              that way either.


Perceived Slow Response       The absenceof data on the responsetimes does not, of course,mean that
Times May Be Caused by        responsetime problems do not exist. CommonSystem users have com-
                              plained about slow responsetimes. However, as our analysis of the sys-
Other Problems                tem’s available utilization data and processingpower indicates, response
                              time problems do not appear to be causedby insufficient processor
                              capacity.
                              Other factors that may be causing FAA’s responsetime problems include
                              the following:
                          9 Inefficiencies in the design of application programs may causeprocess-
                            ing delays that prevent the current system or the proposed CORN system
                            from providing a 2-secondresponsetime.
                          9 Contention for peripheral devices(e.g., disk and tape drives) may cause
                            processingdelays.
                          l The communication system linking users to the Common System may be
                            causing delays. An official at the mainframe facility believes that
                            responsetime problems may be causedby FAA’S Administrative Data
                            Transmission Network communications system. CORN requires 2-second
                            responsetime within the contractor’s facility, not at the users’ termi-
                            nals. As a result, delays due to telecommunications problems or the
                            input devices,such as microcomputers, will not be corrected by CORN.
                            Project officials have asserted,without support, that CORN and the
                            agency’s switch to the new FTS 2000 telecommunications system will
                            eliminate current responsetime problems,
                          . Inefficient managementof current system resourcesmay be causing
                            delays. For example, the mainframe system contains an operating sys-
                            tem module that is designedto optimize the tradeoff between
                            throughput and responsetime. In order to do this, the module must be
                            given control over which batch jobs are allowed in the computer’s mem-
                            ory and when they are allowed in. FAA is denying this control to the
                            module by not allowing many batch jobs to be initiated during prime-
                            time hours. As a result, we identified instances where jobs took only
                            minutes to complete and utilized only hundredths of a secondof proces-
                            sor time, yet waited hours before they were initiated. Oneof the worst



                              Page 17                      GAO/lMTECW   FM CORN Contract Should Not Be Awarded
                                                                                                             r


                             chapter 2
                             CORN Haa Not Been JumtUled




                             casesidentified was a job that unnecessarily waited 80 hours to be initi-
                             ated. Onceinitiated, this job was completed in 12 minutes and used only
                             0.06 secondsof processortime.

                             Project officials constructed a lo-year growth projection to estimate the
lo-Year Growth               data-processingresourcesrequired to meet FAA’S needsthroughout the
PrOjeCtiOn IS Based on       life of the project. Their projection is comprised of four components:
Sparse, Inadequate       . current demand, the measurable demand that existing application pro-
Data                       grams place on data-processingfacilities;
                         . latent demand, the difference between the measured demand and the
                           demand that would be exerted on a computer system if an acceptable
                           level of service were available;
                         l projected mission growth, the increase in the demand for data-process-
                           ing resourcesresulting from the agency’sadditional use of automated
                           systems to meet future mission needs;and
                         l new demand, the demand that planned or developing application pro-
                           grams will exert on the system when they are implemented.
                             According to FAA estimates, these four growth components,when com-
                             bined, result in a projected growth rate of 30 percent per year com-
                             pounded annually. Over 10 years, this amounts to demand growing by
                             about 1300 percent.
                             Making long-range data-processinggrowth projections is inherently dif-
                             ficult. To make accurate growth projections, it is essential to have a
                             complete and thorough understanding of a system’s historical utilization
                             patterns, current demands,and the effect that future applications and
                             changesin the operating environment will have on the system. However,
                             FAA lacks the data to assesscurrent demand, latent demand, and mission
                             growth.
                             FAA’S current demand estimate is flawed becauseit is based on incom-
                             plete and inadequate processor-utilization data. The minicomputer data
                             used by FAA only covered a 4-month time frame from 4 of the agency’s
                             12 minicomputer facilities, and these data had gaps of 1 to 3 months.
                             Project officials stated that the missing data resulted from periods when
                             the monitoring system used to extract utilization data from the system
                             was not operating. The incompletenessof the minicomputer utilization
                             data prevents FAA from accurately identifying daily, weekly, or monthly
                             levels of processorutilization for more than half the Common System’s
                             processing.The lack of complete data and the short measurementperiod


                             Page 18                      GAO/lMTJGBO-2S FM CORN Contract Should Not Be Awarded
chapter 2
C4MtN Has Not Been Jwtified




make it difficult for FAA to obtain a clear understanding of the demands
currently placed on the Common System.
The latent demand estimate is based on the increase in demand that pro-
ject officials say was observed after two upgrades to the mainframe.
Project officials, however, could not provide any utilization data docu-
menting this increase. Moreover, the officials could not furnish any data
that provide a basis for measuring the amount of latent demand in the
other half of the Common System-its minicomputer systems.Conse-
quently, the latent demand estimate is basedpartially on speculation,
rather than on quantifiable data.
FAA’S estimate of projected mission growth is also flawed. To make a
reliable forecast of this growth over the lo-year life of the contract, the
agency needs(1) complete and accurate historical data on system utili-
zation, and (2) an in-depth understanding of the projected changesin
agency operations that will affect FAA’S general-purposedata-processing
resources.

As previously discussed,FAA has a limited amount of historical utiliza-
tion data. These limited data do not provide a sound basis for making an
adequate lo-year growth projection. To compensatefor the lack of his-
torical utilization data, FAA officials relied on their knowledge of past
growth in the agency’s data-processingfacilities to validate their growth
projections. Project officials stated that, on the basis of their knowledge
of central processingunits obtained over the past 10 years, their projec-
tions for the next 10 years appear to be accurate.’
Regarding projected changesin agency operations, government and
industry experts recommendthat growth projections be tied to natural
forecasting units8 and thus be based on real-world actions. However,
project officials did not use natural forecasting units as part of the
growth analysis. As a result, the growth projections are not tied to any
factor that would causethe 30percent annual growth rate to slow down
or plateau at any point during the lo-year life of CORN. FAA’S projection


7Projectofficials statedthat the data-processingindustry will experiencea similar growth pattern
over the next 10 years ashas beencalculatedfor the CORNproject.Whenwe askedprojectofficiala
for the support for this statement,they said that the statementwas basedon a singlemagazine
article.
sA natural forecastingunit is a functionaUy+rient.edunit of measureof the work that systemusers
perform. For a Fmancialinstitution, for example,natural forecastingunits would include loan applica-
tions pmcesaed,credit reports prepared,and checksprocessed.



Page 19                         GAO/lMTEGfWP3 FAA CORNContract Should Not Be Awarded
                                                                                                              .
                             Chapter 2
                             CORN Has Not Been Justifkd




                             methodology simply continues to compound the 30-percent growth rate
                             year after year.


Optional Levels of Service   Embeddedin the growth projections are data-processingrequirements
Based on Unsupported         for applications that are not processedon the Common System. These
                             include some additional FAA applications, as well as applications used by
Assumptions                  other elements of the Department of Transportation such as the Coast
                             Guard, the Federal Highway Administration, and the Federal Railroad
                             Administration. Currently these applications are processedon dedicated
                             hardware, time-sharing systems, or outside FM. Levels of service are
                             included in CORN to accommodatethese additional applications on an
                             optional basis. These levels of service account for approximately 40 per-
                             cent of the estimated $1.6 billion value of the CORN contract. No detailed
                             requirements, feasibility, or cost/benefit studies were done for the
                             optional levels of service portion of the procurement. Sizing for these
                             levels of service was basedon the assumption that the growth rate,
                             acquisition schedule, and the ratio of peripheral devicesto the central
                             processingunit would be identical to that of the CommonSystem’s
                             requirements.

                             FAA'S growth methodology results in extremely large annual leaps in the
The Importance of            system’s size and capabilities, especially after the first few years of
Accurate System              CORN. By the tenth and final year of the CORN contract, the projected
Sizing                       growth rate results in a system almost 1300 percent the size of the cur-
                             rent one. Growth of this magnitude would, for example, result in a sys-
                             tem capable of providing 390,000 hours of system accesstime per
                             calendar day and producing over 16 million printed pagesof output per
                             calendar day in the final year of the CORN contract. A system of this -
                             magnitude would provide 32.6 hours of accessper person per day for
                             12,000 users. Similarly high growth is projected for system input/output
                             transfers, number of tape reels used, gigabyte9 of disk storage, and cen-
                             tral processoruse. However, project officials have not addressedbasic
                             issuesthat are naturally associatedwith growth of this speedand mag-
                             nitude, such as (1) determining how the 30-percent level of growth cor-
                             relates with expected increasesin the number of system users over the
                             next decade,or (2) determining whether FAA's and the Department’s
                             activities would in fact require the production of over 16 million pages
                             of output every day of the year.




                             Page 20                      GA0/IMTEG90&3   FM CORN Ckmtract Should Not Be Awarded
Chapter 2
CORN Has Not Been Justified




The larger and more demanding a computer growth requirement is, the
lower the number of vendors available to meet the requirement. A pro-
curement as large as CORN can only be handled by the largest computer
companies.If the 1300-percentgrowth requirement is in fact too high,
then it may have unnecessarily precluded other vendors from compet-
ing. In addition, vendors must design solutions and attempt to cover the
costs for a system that can handle a potential work load of this size.
Therefore, the per-unit cost can be expected to be high for lower levels
of usage.




Page 21                       GAO/lMTlW9O&S   FM CORN Contract Should Not Be Awarded
Chapter 3

CORNEvaluation Methodology Is Faulty


                         In a procurement as large and costly as CORN, it is critical for FAA to have
                         meaningful and accurate information for evaluating the most effective
                         solution for meeting agency requirements over the contract’s lo-year
                         period. However, a key element of FAA'S methodology for validating the
                         vendors’ proposed solutions-an operational capability demonstra-
                         tion-is deficient in two ways. First, information that FAA provided to
                         the vendors to assist them in developing their proposals was incomplete.
                         Second,the sample work load that FAA developed for use in the demon-
                         stration is extremely small and unrepresentative of the Common Sys-
                         tem’s total work load. As a result, the demonstration will not provide
                         adequate data for accurately evaluating the vendors’ proposals and
                         their proposed charges for data-processingservice. This deficiency
                         could have cost ramifications throughout the life of the contract.

                         Performance and capability validation techniques are important to
Performance and          agenciesthat are acquiring a large amount of data processingequipment
Capability Validation    or services.Performance and capability validation helps to reduce the
Are Critical for Large   risks of acquiring insufficient or excessivecapacity, inadequate func-
                         tional capability, and uneconomical capability.
Acquisitions
                         Federal Information ResourcesManagementRegulation 201-30.013-3
                         stipulates that the selection of performance evaluation techniques shall
                         be commensuratewith the programmatic risks of inappropriate or insuf-
                         ficient data-processingcapacity. Present and forecasted data-processing
                         work loads, anticipated system life costs, validation costs to the govern-
                         ment and offerors, and objectivenessand fairness in the acquisition pro-
                         cessare factors to be considered.
                         The most precise performance validation technique is benchmark test-
                         ing. A benchmark is a set of computer programs and associateddata
                         tailored to represent a particular work load. A benchmark test is a user-
                         witnessed demonstration on a vendor’s proposed computer system done
                         to validate system performance or cost. Benchmark tests are used to
                         assesshow a vendor’s system will processthe work load (e.g., process-
                         ing speed,resource consumption) and to compare the performance of
                         several systems.




                         Page 22                  GAO/MTEG9O-38   FM COBN Contract Should Not Be Awarded
                      CORN Evaluation Methodology b Faulty




FAA’s Validation      Under the validation methodology developed by FAA, vendors will be
                      required to perform an agency-designedoperational capability demon-
Strategy Will Not     stration in which they will processa portion of the agency’swork load
Adequately Test the   on a subset of their proposed systems. The makeup of this sample work
                      load is critical to the thoroughness and adequacy of the demonstration
Proposed Systems      becauseit will exercisethe vendors’ billing algorithms, which will be the
                      basis for charging the government for the servicesprovided,’ The dem-
                      onstration is intended to aid FAA in determining how well the vendors’
                      proposed solutions will meet the agency’sprojected requirements.


Problems Cited by     Problems with FAA’S demonstration were originally cited by the General
Independent Review    ServicesAdministration’s Federal Systems Integration and Management
                      Center, which was asked by FAA to perform an independent review of a
                      draft of the CORN request for proposals.2In its September 1988 report to
                      the agency,the Center raised methodological issuesregarding the dem-
                      onstration, stating that the agency’s demonstration methodology would
                      not accurately validate the cost of processingthe agency’swork load on
                      a vendor’s proposed system. The Center stressedthat it is crucial for the
                      agency to accurately validate its work load in terms of the billing units
                      proposed by each vendor. According to the Center, an error in validating
                      the number of billing units required to processFAA’S work load could
                      have “staggering cost consequencesover the lo-year contract life.” The
                      Center stated that the only reliable method for performing this valida-
                      tion is through the use of a “government-furnished benchmark.” We dis-
                      cussedFAA’s validation methodology as it appears in the final draft of
                      the CORN request for proposals with Center officials. The officials said
                      that they would prefer that FAA use a more rigorous test, which would
                      include a larger, more representative sample of the agency’swork load.


Data and Validation   In our own review of FAA’S proposed validation methodology, we found
Problems Remain       that serious problems remain in its approach. Onebasic problem is that
                      the data supplied by FAA to the vendor community to aid them in sizing a
                      system capable of meeting the agency’s general-purposedata-processing
                      needsfor the lo-year life of the contract are flawed. The agency sup-
                      plied performance data for its mainframe system on magnetic tape.
                      ‘The actual algorithmsdesignedby eachvendor will be extremely complex,including dozensof pric-
                      ing elementswith complicatedrelationshipsamongthem.Properly exercisingthesealgorithmsis crit-
                      ically important to understandingthe utilization costaof the resourcesbeingoffered.
                      2TheCenterwas establishedin 1972to assistfederal agenciesin acquiring,managing,and using
                      information systemsand technology.It provides agencieswith a wide rangeof technical and contrac-
                      tual assistanceon a cost-reimbursablebasis.



                      Page 22                        GAO/IlWEC-9O-2S FM CORN Contract Should Not Be. Awarded
Chupter 3
CORN Fivaluatlon Methodology Ia Faulty




However, portions of the data neededto accurately model FAA’S current
system were either unusable or missing from the tapes, making it impos-
sible for the vendors to obtain information on such things as disk drive
activity.
In addition, information neededto identify the characteristics of batch
and on-line application systems3and transaction volumes was not pro-
vided. Consequently, vendors would have to make assumptions about
key characteristics of the current mainframe system in order to perform
their modeling.

Another problem with FAA’S approach is that the sample work load to be
used in the demonstration doesnot represent the CommonSystem’s
work load. According to FAA, the CommonSystem’s current work load is
made up of almost 290 application systems,both batch and on-line,
totaling nearly 16 million lines of code.However, the sample work load
to be used in the demonstration is made up of only 12,000 lines of code
from parts of two application systems.
Project officials admit that the application subsystemsto be used in the
demonstration are only “slivers” of the CommonSystem’s total work
load and are not representative. Nevertheless,they maintain that these
subsystemsare typical examples of the types of applications that are
processedin FAA’S current data-processingenvironment becausethey
have qualitative attributes found in many of the agency’s applications.
For example, according to project officials, the programs both use a
high-level programming language and data base managementsystems.
In addition, one application is processedat a minicomputer facility,
while the other is processedat the mainframe facility.

Officials also maintain that a more rigorous validation methodology
would be hard to develop and more costly to both the agency and ven-
dors becauseof the difficulty in defining and constructing a larger, rep-
resentative work load.
We recognizethat the two subsystemschosenfor the demonstration
have attributes similar to other types of CommonSystem programs.
However, becauseof their small size and level of activity, they cannot
provide F&I with a realistic view of the resource utilization cost per
transaction or cost data resulting from work loads of increasing size. As

3A batch systemprocessesonegroup of transactionsentirely beforeprocessingthe next group.On-
line refers to users’ability to accessand interact with a computervia a terminal.



Page 24                        GAOjIMTEG90-3S FM co&N Contract Should Not Be Awarded



                          .W
                                                                                     a.2
chapter8
CORN Eval~tion   Methodology ImFaulty




a result, the demonstration would not provide FM with adequate data to
evaluate vendors’ proposals and their proposed chargesfor data-
processingservice.




Page 26                     GAO/RtiTECM   FM CORN Contract Should Not Be Awanled
Chapter 4                                                                                                                 I

Major Uncertainties Loom Over Conversion


                             The General ServicesAdministration’s Federal Software Management
                             Support Center starkly characterizes software conversion1as “labor
                             intensive, managementintensive, machine-resourceintensive, and dead-
                             line intensive. In short, it has all of the wrong attributes for a successful
                             enterprise, and many problems will arise.” Becauseof the extremely
                             large size of the CORN conversion- nearly 290 software applications
                             made up of over 14.8 million lines of code-and the fact that FAA staff
                             will need to assist the contractor in the effort, the conversion will be
                             costly, disruptive, and time-consuming. Unfortunately, FAA has not ade-
                             quately prepared for the conversion, resulting in major outstanding
                             problems and uncertainties. Specifically:
                         . The conversion could cost substantially more than the current $74.6mil-
                           lion estimate becauseit is basedon an unreliable inventory of applica-
                           tions and excludes the cost of fixing major documentation deficiencies.
                         . Project officials state that 86 employee-yearsof agency staff support
                           will be neededfor the conversion. This figure, however, does not include
                           all the conversion tasks and is not based on a review of each individual
                           application to determine the amount of conversion support needed.More
                           significantly, the availability of agency staff to support the conversion
                           will not be determined until after the CORN contract is awarded.
                         l The estimated time neededfor the conversion has doubled from 18
                           months to 3 years. The Common System will continue to operate until
                           the conversion is complete, at a potential 3-year cost of $106 million.
                           This cost will need to be funded in addition to CORN.
                         l The conversion will not correct problems in the applications, which may
                           be causing inefficient processingof data, nor will it improve the
                           agency’s information structures.

                             Project officials estimate that the conversion will cost $74.6 million.2 An
Conversion Cost              essential step in making such an estimate is preparing and validating the
Estimate Is Unreliable       inventory of applications software and files to be converted. The $74.6-
                             million estimate is based on project officials’ initial attempt in 1987 to
                             establish an inventory of applications to be converted. This effort did
                             ‘Software conversionis the transformation,without functional change,of computerprogramsand
                             data to permit their useon replacementdata-processingequipment.
                             2Thecomplexity and costof the conversionwill vary with the type of hardwarebeingproposedby a
                             vendor.The projectteamdevelopedcostestimatesfor severalhardware scenarios-rsnging from
                             $10.6million for a replacementsystemfully compatiblewith the current system,to $126million for a
                             completelynoncompatiblereplacementsystem.The $74.~million figure consistentlyusedin budget
                             documentscorrespondsto the estimatefor convertingto an International BusinessMachines-compati-
                             ble system.This estimatedoesnot include the costof convertingother FAA or Departmentof Trans-
                             portation applicationsaspart of CORN’soptional levels of service.



                             Page 26                        GAO/IMTEGgO&3 FM CORN Contract Should Not Be Awarded
chapter 4
Mejor Uncertaintie    Loom Over Conversion




not succeedin establishing reliable results, and therefore the cost esti-
mate remains questionable.
Between I987 and 1989, estimates of the number of applications that
need to be converted ranged between about 200 and 600, while esti-
mates of the lines of applications codeto be converted varied from
about 10 million to 18 million. Project officials maintain that the vari-
ance in the count of applications to be converted is largely a result of
different ways in which they grouped related applications. They have
no explanation for the changesin the corresponding line counts, how-
ever. As late as February 1989 project officials concededin a CORN
“basic facts” briefing document that the conversion will not be easy
because“FAA’S knowledge of what currently exists to be converted is far
from perfect.”
This uncertainty over the applications inventory reached a critical point
shortly after the request for proposals was issued in February 1989. FAA
received vendor complaints that the CORN documentation package-
which was supposedto include current code for all applications to be
converted-was confusing, incomplete, and did not provide an adequate
basis for making a firm, fixed-price offere3In April 1989 FAA informed
prospective offerors that the documentation package “is being revised
to include the information neededto fix price the effort.” FAA analysts
subsequently reviewed the documentation package and found that it
contained obsoleteversions of applications, along with applications no
longer in use. In addition, someapplications systems were listed for con-
version in the CORN request for proposals, but had no code included in
the documentation package.Project officials concluded that the materi-
als provided by its staff for this package “were not sufficiently accu-
rate.” They subsequently characterized the documentation package as a
first attempt to gather code for the vendors’ use in preparing proposals
for CORN. They maintain that the initial documentation package pro-
vided information “to enable offerors to analyze FAA’s source codeto
appreciate the scopeof the conversion effort.”
Between February and August 1989, the conversion count continued to
vary from 204 applications to 247 to 279, while the line count changed
from 12.8 million to nearly 14.8 million (with an interim estimate of 10



3An FM fraud hotline complainttriggereda review of the issueby the Departmentof Transporta-
tion’s Office of the InspectorGeneral.The resulting report is scheduledfor releaselater this year.



Page 21                          GAO/lMTEGSO-2SFM CORN Contract Should Not Be Awarded
ChaPtm 4
Major Uncertainties Loom Over Convereion




million lines done between these two line counts). The revised documen-
tation package and related amendment to the request for proposals, list-
ing 279 applications with nearly 14.8 million lines of code,was issued
only 3 weeks before vendor proposals were due in August 1989. Accord-
ing to an FAA memorandum, the revised package still had missing and
incorrectly labeled components.Project officials issued another revision
in February 1990 that raised the number of applications to 289 and
increased the line count to slightly more than 14.8 million. Continued
changesin the documentation package between the time of expected
contract award and the actual conversionscould drive up conversion
costs by triggering cost-adjustment provisions specified in the request
for proposals.
The conversion cost estimate is also unreliable becauseproject officials
did not include an estimate of the cost of enhancing or updating out-of-
date applications’ documentation- such as system, program, and users’
manuals-or creating such documentation where none exists. Providing
new documentation for all converted applications, except for functional
requirements, is required of the contractor under CORN. These documen-
tation costs could be substantial becauseproject officials estimate that
the documentation is, on average,only about 60 percent current and
complete.
Project officials originally told us that the additional costs of fixing doc-
umentation deficiencies would be offset by an anticipated decreasein
the number of lines to be converted as a result of revising the applica-
tions inventory. When the new inventory did not result in the antici-
pated reduction, project officials speculated that the additional
documentation would probably add less than 10 percent to the $74.6-
million conversion cost estimate. Officials now maintain that they have
developed a new cost estimate, for use during review of vendors’ pro-
posals, that includes the documentation costs. For planning and budget-
ing purposes, however, FAA is continuing to use the original $745million
estimate and continues to submit this figure to the Congressin CORN
budget information.




Page 28                      GAO/lMTEGBO3S FM CORN Contract Should Not Be Awarded
                           chapter 4
                           Mqjor U&tiea         Loom Over Convemion




                           FM information systems staff and program office staff will need to sup-
Staff Support for          port the conversion by helping the contractor understand the functional-
Conversion                 ities of the applications being converted, assisting in resolving technical
Inadequately               issuesand documentation problems, and reviewing the converted sys-
                           tems prior to accepting them. In presenting the project to the Depart-
Determined                 ment of Transportation for approval, FAA committed itself to a series of
                           actions that it termed “critical to the successof CORN."
                           Among these actions, FAA stated that it planned to “establish an estimate
                           of FAA/m [Department of Transportation] staffing resourcesrequired
                           and their availability for each [application] system conversion prior to
                           contract negotiations.” We found, however, that project officials devel-
                           oped only aggregatedestimates of the amount of agency employee-years
                           neededfor the conversion and have not determined the extent to which
                           agency staff with appropriate skills will be available to support the
                           conversion.


Estimated Amount of        Project officials state that a total of 86 employee-yearsof staff support
Employee-Years Is Poorly   will be neededto assist in the conversion, excluding employee training.4
                           However, this figure is not basedon a review of each individual applica-
Supported                  tion to take into account the specific characteristics, complexities, condi-
                           tion of documentation, or other problems associatedwith converting
                           each one. Instead, project officials developed a formula basedon general
                           assumptions about the amount of time neededto perform typical con-
                           version tasks and the number of lines of code involved. They told us
                           that they lumped the mainframe-based applications into one group and
                           applied this formula to the aggregate,and used the sameprocedure for
                           the minicomputer-based applications. Project officials were unable to
                           provide us with documentation showing the basis for their general
                           assumptions or how they carried out their calculations.
                           FAA initially informed the Department of Transportation that the con-
                           version would require 46 employee-yearsof FAA staff support6 Project

                           4Projectofficials have identified the needfor an additional 6 employeeyearsto tram agencystaff in
                           using the CORNsystem.This estimateis very soft, however.Projectofficials informed the Depart-
                           ment of Transportation that “the numben of employeesinvolved in the application systemsopera-
                           tion varies too much from application systemto application systemto makea meaningfulestimate
                           for conversion/transition.” They maintain that a fm estimatecannotbe madeuntil after contract
                           award.
                           ‘The 46 employeeyearestimateis cited in a projectmilestonechart provided by FAA to the Depart-
                           ment of Transportation in April 1989in responseto the Department’srequestfor additional mforma-
                           tion on projectplanning.



                           Page 29                         GAOjIMTECf%28       FM CORN Cm~tract Should Not Be Awarded
                         Chnpter 4
                         Major Uncertainties Loom Over Conversion




                         officials said that this number was a typographical error, since the esti-
                         mate they had actually calculated was 86 employee-years.The 86
                         employee-year estimate is used in the CORN request for proposals and
                         remains current. However, this estimate doesnot include the staff time
                         neededto prepare the conversion packagesto be provided to the con-
                         tractor for each application. Project officials estimate that this task will
                         require about 28 employee years.
                         Project officials maintain that their employee-year estimate cannot be
                         refined until after the CORN contract is awarded, since the amount of
                         staff resourcesneededwill depend to a large degreeon how compatible
                         the current applications software is with the contractor’s system. They
                         are nevertheless confident that they have a “pretty solid feel” for the
                         amount of employee-yearsneededfor the conversion, In the CORN
                         request for proposals, they stipulated that vendors bidding on the con-
                         tract must develop preliminary conversion plans that require no more
                         than 86 FAA employee-yearsof support (42 years for the mainframe
                         applications and 43 years for the minicomputer applications). Project
                         officials acknowledge, though, that this doesnot preclude FAA from pro-
                         viding a higher level of support, if necessary.For example, they said
                         that if funding for the conversion is constrained, someof the contrac-
                         tor’s conversion work could be shifted to agency staff.


FAA Staff Availability   Along with estimating the number of employee-yearsneededto support
Remains Undetermined     the conversion, it is necessaryto determine the extent to which agency
                         employeeswith the neededskills can actually be made available from
                         their regular work to support the additional work involved in the con-
                         version As noted, FAA'S promise to determine staff availability for each
                         application conversion prior to contract negotiations has not been car-
                         ried out. Project officials told us in March 1990 that they were in the
                         processof beginning to determine staff availability. They maintain,
                         however, that this activity cannot be made final until after the contract
                         is awarded and the contractor’s conversion plan is completed. This
                         leaves unresolved the crucial issue of the extent to which appropriate
                         FAA staff will actually be available to support the conversion. A senior
                         information resourcesmanagementofficial acknowledged that the con-
                         version work load would put a heavy burden on the staff, but said that
                         the agency would find a way of working through it.
                         If staff are not available to the extent neededto support the conversion,
                         agency offices may need to turn to other contractors for help. For exam-
                         ple, officials in FAA'S Human ResourceInformation Division said that


                         Page 80                      GAO/IMTECBO-3S FM CORN Contract Should Not Be Awarded
                       chapter4
                       Mnjor Uncertaintie   Loom Over Conversion




                       they do not have enough staff to maintain the current personnel man-
                       agementsystem while supporting the complex task of reengineering the
                       system for CORN. Consequently, they have requested $2.3 million in con-
                       tract help for the conversion to CORN. The manager of the payroll system
                       expressedsimilar concern about the lack of staff to support the conver-
                       sion. Other program offices responsible for several less complex applica-
                       tions may need similar contract help in coping with their aggregate
                       conversion work load.
                       Project officials acknowledge that FAA is planning to support the CORN
                       contractor by using a combination of agency personnel and assistance
                       from additional contractors. They have determined that the agency’s
                       information servicesstaff will need to rely more heavily on their current
                       contractors, and have asked FAA to provide $1.2 million a year during
                       the 3-year conversion period for this purpose. However, the extent and
                       cost of additional contractor help neededby FAA program offices
                       remains undetermined.

                       Project officials said that they are currently developing a preliminary
                       conversion schedulethat will propose the sequencefor converting the
                       applications and apportion the 86 employee-year estimate to individual
                       applications. The officials said that prior to contract award, they will
                       ask the staff designated to managethe conversion to comment on the
                       plan. They said that they would not send the plan to the program
                       offices, although the offices would be heavily involved in the conver-
                       sion Sincethis plan is to be used in contract negotiations, project offi-
                       cials maintain that it is procurement-sensitive and not for general
                       release.The plan originally was to be made final in August 1989, but
                       was still being reviewed internally when we completed our audit work in
                       April 1990.

                       The amount of time estimated to complete the conversion has repeatedly
Estimated Conversion   escalated.Project officials originally assumedthat the conversion would
Time Frame Has         take 18 months, which is the estimate found in the conversion studv and
Doubled                various documents leading to the project’s approval by the Department
                       of Transportation in January 1989. They said that this l&month sched-
                       ule was an optimum estimate basedon advice from several contractors.
                       However, F&l program offices maintained that they could not meet this
                       estimate becausethey lacked the requisite staff and other resources
                       neededto satisfy CORN'S requirements while concurrently performing
                       their regular duties. Consequently, project officials increasedtheir esti-
                       mate to 24 months and then to 30 months. The officials said that this 30-


                       Page 31                       GAO/IMTEG9O-3f3 FM CORN Contract Should Not Be Awarded
                      -p&W 4
                      Ib&jor Uncertninties Loom Over Conversion




                      month estimate was judgmental, basedon discussionswith agency infor-
                      mation systems staff. By November 1989, project officials had raised
                      the estimated time frame to 3 years, citing budgeting concernsthat
                      could further slow the pace of the conversion.
                      Conversion delays will have cost consequences.Project documents
                      repeatedly state that the CommonSystem will remain fully operational
                      in parallel with CORN until all of the system’s applications are success-
                      fully converted. Project officials estimate that it costs $36 million a year
                      to operate the CommonSystem (excluding an additional $14 million a
                      year in personnel, space,and supply coststhat would not be avoided by
                      moving to CORN). The conversion time frame delay from 18 months to 3
                      years could result in additional parallel operations costsof $106 million.
                      The officials noted that the cost of parallel operations might be reduced
                      by phasing out portions of the CommonSystem before the conversion is
                      completed, depending on the conversion plan agreedto by FAA and the
                      contractor.6In any event, the parallel operations costs will be substan-
                      tial and would have to be funded in addition to CORN contract costs.

                      Oneof FAA’S goals is to create automated systems that provide the staff
Conversion Will Not   with better information for managing their programs. The CORNjustifi-
Result in Better      cation documents stress this goal by pointing out the agency’sdesire to
Information           integrate data into more useful information structures to improve man-
                      agementpolicymaking and decisionmaking. Project officials claim that
                      the diversity of hardware in the current system inhibits meeting this
                      goal, while CORN would promote it.
                      For all of its expense,however, the CORN conversion in itself will not
                      move FAA toward this goal. Although the conversion would move the
                      applications and data basesonto a new hardware environment, it would
                      not improve the applications themselves or integrate the data bases
                      becauseof the way the conversion is structured. Project officials insist
                      that existing applications and data basesbe converted so that they are
                      functionally equivalent to the original applications7 That is to say, the
                      sameinputs to both the original and converted applications are to pro-
                      duce identical outputs. Vendors were informed that enhancementof the
                      %omeapplicationsprocessedon the CommonSystem,such asoffice automationapplications,are not
                      part of the CORNconversion.They, too, will needto be movedoff of the CommonSystem-mainly to
                      microcomputers-before the systemcan be closeddown.
                      ?lkvo exceptionsare the ConsolidatedPersonnelManagementInformation Systemand the Aeronauti-
                      cal Information System,which are to be put onto a modemcommercialdata basemanagement
                      system.



                      Page 32                       GAO/IMTEC3O-33 FM COW Contract Should Not Be Awarded
Mqjor Unoertaintlsr   Loom Over Conversion




application codefor purposes of increased functionality, presentation
improvement, or performance improvement are neither encouragednor
desired. The conversion will result in information being provided in the
sameway it is now. The officials said that they adopted this approach in
order to be able to determine whether the conversion was carried out
successfully.
Project officials have not determined whether all of the applications
identified for conversion are worth converting. Following contract
award, but prior to ordering the conversion of a particular application,
the FAA office responsible for the application is supposedto determine
whether there is a continued need for it. Software problems with the
applications are to be corrected after they have been converted. The
officials maintain that correcting any such problems is the responsibility
of the agency staff in charge of the applications and is outside the scope
of CORN, which was never intended to be a software development con-
tract. This includes correcting errors in the applications, enhancing their
functionality, or providing new functional requirements documenta-
tion-all of which are tasks not required of the contractor under CORN.
Thus, if the current software is inefficient and causing poor response
times, the possibility exists that the converted software will still be inef-
ficient, and CORNwill not achieve one of its primary benefits-improved
responsetime.




Page 33                        GAO/MTEGQO4l3 FM CORN Contract Should Not Be Awnrded
Chapter 6

Conclusionsand Recommendations


             A project of CORN'S magnitude, both in terms of cost and effect on opera-
             tions, requires careful preparation and well-supported technical analy-
             sis. CORN envisions a total commitment: the project is meant to
             completely replace the current system, not merely to augment it, by
             transferring FAA’S general-purposedata processingto one vendor for up
             to 10 years. The project’s viability dependsheavily on FAA’s ability to
             understand its current system, define problems and their causes,fore-
             cast future needsfor a decade,adequately evaluate vendor-proposed
             systems, and adequately plan for conversion to the new system. To date
             FAA has not demonstrated satisfactory performance in these areas, Con-
             sequently, it is not ready to proceed with awarding the CORN contract.
             CORN is, in  essence,a request to the vendor community for a comprehen-
             sive technical solution to FAA’S long-term data-processingproblems and
             needs.While the concept of contracting for data-processingservices sup-
             port may be acceptable,FAA has not adequately defined, measured, and
             analyzed its current technical problems and needs.Most notably, FAA
             has not adequately documented responsetime problems, nor defined the
             causesof perceived responsetime problems with its current system. Its
             assertion that these problems exist and are causedby the lack of proces-
             sor capacity is not supported by available data. By not establishing a
             central capacity managementprogram for the Common System’s 12
             facilities, FAA has done a poor job of tracking Common System utiliza-
             tion, monitoring the system’s performance, optimizing the use of its cur-
             rent resources,and identifying the causesof perceived responsetime
             and performance problems. BecauseFAA has not identified the causeof
             responsetime problems in the current system, it cannot be assumedthat
             CORN will eliminate these problems. Without improved responsetimes,
             CORN will not achieve one of its major objectives.

             Since past and current performance data are an indispensable element
             with which to build projections of future growth, FAA is not able to make
             a reasonableprojection of its anticipated growth in general-purposedata
             processingover 10 years. Its projected growth rate of 30 percent per
             year compoundedfor 10 years is basedon inadequate data and oversim-
             plified analysis of the agency’swork load. The methodology does not
             take into account any factors that would causethis rate of growth to
             change over the contract period, especially in the later years when the
             cumulative effect of the 30 percent growth leads to extremely large
             annual jumps in system capacity-eventually reaching a point where
             the system would be 1300 percent larger than the current one. Further,
             project officials know even less about the future processinggrowth of
             other elements of the Department of Transportation. They have simply


             Page 34                GAO/IMTEC9O-3El FM CORN Contract Should Not Be Awarded
Chapter 5
Conclueiona   find Recommend~~tiona




assumeda similar 30-percent annual growth rate and increased CORN'S
estimated contract value by about 40 percent to accommodateit.
Although FAA and the Department are not obliged to order all of the ser-
vice built into CORN, vendors must develop a proposed system capable of
accommodatingthis high rate of growth. And they must develop fixed
prices for levels of service that such a system would provide over 10
years. FAA, however, has not provided vendors with key information on
the performance of the current system that they should have in order to
properly develop a proposal to meet FAA'S projected needs.In the
absenceof such information, vendors have to make assumptions about
crucial characteristics of FAA's current system in order to model their
proposed solutions to the agency’sneedsas specified in the request for
proposals.

FAA should have meaningful and accurate information with which to
evaluate the most effective solution for meeting the agency’sprojected
needs.one of FAA's critical evaluation tools for doing this-the agency-
designedoperational capability demonstration-is seriously deficient.
The sample work load used in the demonstration is extremely small and
not representative of the agency’swork load. As a result, the demon-
stration will not provide adequate data for accurately evaluating the
vendors’ proposals. This problem, coupled with FAA'S failure to provide
the vendors with adequate data with which to model a new system,
could have cost ramifications throughout the life of the contract.
A critical hurdle for implementing CORN is the conversion of Common
System applications. Project officials have not adequately prepared for
the conversion, as evidenced by the long-standing confusion over the
inventory of applications to be converted to CORN. The officials have also
not provided agency managementand the Congresswith reliable infor-
mation on the funding, staff resources,and time frames of the conver-
sion The conversion cost estimate of $74.6 million, used for project
approval and budget requests, is unreliable and incomplete. The project
officials’ assertion that the conversion will require 86 employee-yearsis
broadly estimated on a aggregatebasis. Most critically, staff availability
to support the conversion will not be determined until after contract
award, when the agency’s staff will be confronted with the necessity of
working through the conversion someway or another.
As for time frames, CORN passedthrough the justification and approval
processon the unsupported assumption that the conversion would
involve a quick, 18-month effort. The time frame was lengthened to 3


Page35                          GAO/lMTEGM   FM CORN Contact Should Not Be Awarded
                        chapt4Ts
                        Ca~chusio~ and Recommendation




                        years after the project was approved, when reactions from agency staff
                        made clear the unreasonablenessof the project officials’ original esti-
                        mate. The conversion inherently has a high risk for scheduledelays
                        becauseof its extremely large size, as underscored by the fact that the
                        original time frame has already doubled before any conversion work has
                        begun. Delays in the conversion would further increasethe cost of run-
                        ning the current system in parallel with CORN during the transition
                        period. For the current 3-year time frame, these costs could run as high
                        as $106 million.

                        Finally, the conversion approach chosenby FAA will not result in
                        improvements to the applications or their data bases.Problems cur-
                        rently found in them would be transferred to the new system at consid-
                        erable cost.

                        Becausethe CORN acquisition has not been adequately justified or
Recommendationsto       planned and has major unresolved problems, we recommendthat the
the Secretary of        Secretary of Transportation direct that the CORN contract not be
Transportation          awarded.@ addition, we recommendthat the Secretary direct the
                        Administrator, FAA, to ensure that future procurements of this type and
                        magnitude are properly justified and planned prior to contract award.
                        Specifically, the Administrator must ensure that the following is done
                        priorto proceeding with a comprehensiveprocurement similar to CORN:

                    . Existing system deficiencies need to be accurately and completely iden-
                      tified and a solution needsto be designedthat addressesthese deficien-
                      cies. Direct, periodic, systemwide monitoring, accomplishedthrough the
                      implementation of a computer capacity and performance management
                      program for FAA'S general-purposesystems, should be used to determine
                      the presence,extent, and causesof performance problems-such as
                      poor responsetimes.
                    l Evaluation of vendors’ proposals should involve the use of a representa-
                      tive work load sample.
                    . Planning for conversion should include an accurate inventory of the
                      existing applications, an assessmentof their continued need, and com-
                      plete estimates of the cost and employee-yearsneededto support the
                      conversion, including the extent to which qualified staff are available.

         fl
Agency Comments         way of doing businessand of making data-processingservices available
                        to the agency’sprogram offices during the 1990s.Accordingly, they


                        Page 86                    GAO/lMTElCW3S   FM C4MW Contract Should Not Be Awarded
.



    ChaPtm 6
    C4mclumlon13and bc4mmendarione




                                                            _-     ~~ ~-~~
    maintain CORN should be assessedon its planned benefits-such as ena-
    bling FAA to better manageits information resources,provide quality
    services as needed,and perform its mandated missions. Although
    expressing general agreementwith the facts presented in our report,
    they disagree with the conclusions.They maintain that the report has
    not consideredCORN'S planned benefits and has judged the project on
    grounds that are not fundamental to its purpose and not material in the
    larger context. They maintain that CORN is sufficiently planned and justi-
    fied to warrant its award. We maintain that our findings deal with
    problems and deficiencies that are fundamental to the project’s scope
    and implementation and, consequently, support the conclusion that the
    contract should not be awarded.
    After receiving our draft report for comment, the FAA Administrator
    decided to order an independent review of CORNto determine if it should
    proceed as planned. The HouseCommittee on Appropriations has
    directed the Department and FAA not to award the CORN contract until
    (1) the Committee reviews the results of our report and FAA's written
    responseto it and (2) FAA and Department officials subsequently discuss
    the project with the Committee to resolve any outstanding concerns.




    Page 37                     GAO-       FM CORN Contract Should Not Be Awarded
Appendix I

Major Contributors to This Report


                   411
                         Joel Willemssen, Assistant Director
Information              John P. Finedore, Evaluator-in-Charge
Management and           Dr. Rona B. Stillman, Chief Scientist
Technology Division,     Frank Reilly, Senior Technical Adviser
                         SusanMaciorowski, Presidential Exchange Executive
Washington, D.C.         David M. Bruno, Computer Scientist
                         Leonard J. Latham, Technical Adviser
                         Bruce Herbert, Senior Technical Specialist




(aloasl)                 Page 38               GAO/IMTEG9033   FM CORN Contract Should Not Be Awarded
United States
General Accounting Office
Washington, IX. 20648
Official Business
Penalty for Private Use $300