)( .* *, IJnited States General Accounting Office i I Report to the Chairman, Subcommittee on Transportation and Related Agencies, Committee on Appropriations, House of Representatives FAA PROCUREMENT Major Data-Processing Contract Should Not Be Awarded 141576 Information Management and Technology Division B-234903 May 25,199O The Honorable William Lehman Chairman, Subcommittee on Transportation and Related Agencies Committee on Appropriations House of Representatives Dear Mr. Chairman: This report responds to your request that we review the Federal Aviation Administration’s Computer ResourcesNucleus project. It presents our analysis, conclusions,and recommendations regarding the project. In March 1989, we reported on the project’s objectives, cost estimates, and implementation approach. As arranged with your office, unless you publicly release its contents earlier, we plan no further distribution of this report until June 15,199O.At that time, we will send copies of the report to the Secretary of Transportation; the Administrator, Federal Aviation Administration; the Director, Office of Management and Budget; the Administrator of General Services;interested congressionalcommittees; and other interested parties. This work was performed under the direction of JayEtta Hecker, Director, Resources, Community, and Economic Development Information Systems,who can be reached at (202) 275-9676. Other major contributors are listed in the appendix. Sincerely yours, Ralph V. Carlone Assistant Comptroller General lihecutive Summary In February 1989, the Federal Aviation Administration (FAA) issued a Purpose request for proposals for its largest and most complex general-purpose data-processingacquisition to date: the Computer ResourcesNucleus (CORN) project. This project is intended to meet the agency’s general- purpose data-processingneedsfor 10 years and provide options for sup- porting the processingneedsof other parts of the Department of Trans- portation at an estimated cost of $1.6 billion. At the request of the Chairman of the Subcommitteeon Transportation and Related Agencies, House Committee on Appropriations, GAO reviewed FAA'S assessmentsof its data-processingproblems and future needsused to justify CORN, its methodology for validating vendors’ proposals, and its planning and preparation for converting current applications’ software. Currently, FAA'S 12 in-house “Common System” computer facilities pro- Background vide general-purposedata processingfor mission and administrative areas such as airport and aviation activity (excluding real-time air traf- fic control), aviation safety, national airspace facilities, and financial, materiel, and human resources.Agency officials maintain that the Com- mon System is not meeting FAA'S current processingneedsbecauseof capacity and responsetime problems, and that it is not possible or desir- able to meet long-term needsby upgrading this system. To resolve these problems, FAA developed the CORN project, under which a contractor would provide data processingon a fee-for-service basis for up to 10 years from computer facilities owned, operated, and main- tained by the contractor. The Common System would be closed down once current applications systems are moved to the new system. In addi- tion to meeting FAA'S anticipated needs,the project includes options for providing service to other elements of the Department of Transporta- tion. FAA estimates that the contract, scheduledto be awarded in the summer of 1990, has a value of $1.5 billion. The CORN project has not been properly justified and planned, and con- Results in Brief tains major unresolved problems. FAA'S claims about the causesof per- ceived problems with its current system are poorly supported, as are its projection of future needs,leading to fundamental doubts about the pro- ject’s justification. In addition, the agency’smethodology for evaluating technical and cost aspectsof vendor proposals is seriously flawed, a problem that could have cost ramifications. Further, FAA'S estimates of the cost of converting software to CORN and the amount of agency sup- port neededfor the conversion are unreliable. The estimated time frame Psge2 Executive summary for the conversion has already doubled, from 18 months to 3 years, lead- ing to more cost growth. Finally, the conversion itself will not result in better managementinformation. Principal Findings FAA’s Justification for In documents and briefings provided to the Congressand the Depart- CORNIs Unsubstantiated ment of Transportation, FAA has repeatedly cited CommonSystem capacity and responsetime problems as key justifications for CORN. However, FAA is unable to provide support for these perceived problems becauseit lacks a central capacity and performance managementpro- gram to gather and analyze data on system-wide utilization and responsetimes. GAO'S own analysis of available system monitoring data shows no evidence of computer processor capacity conditions that would causeresponsetime problems, indicating that perceived response time problems may not be solved by procuring more computer process- ing capacity through CORN. FAA itself has not determined the specific causesof such problems. FAA also justifies CORN on the grounds that its future general-purpose data-processingneedswill increase at a rate of 30 percent per year over 10 years, resulting in a system about 1300 percent larger than the cur- rent one. However, GAO found that this estimate is basedon sparse data, raising doubts about whether such extremely steep growth will occur. Project’s Validation FAA needsmeaningful and accurate information to evaluate the most Methodology Is Flawed effective way of meeting agency requirements. However, one key FAA method used in validating the vendors’ CORNproposals is deficient becausethe sample work load to be used is extremely small and unrep- resentative of the agency’stotal work load. In addition, the performance information on the current system that FAA provided to the vendors to assist them in developing their proposed solutions was incomplete. Fail- ure to accurately validate vendors’ proposed solutions could have cost ramifications throughout the life of the contract. Major Problerks Loom Over Converting computer application software to run on a new system is a CORNConversion costly, disruptive task that requires careful preparation. FAA'S efforts to plan for the CORN conversion have been marred by its unreliable initial Page g GAO/IMTEC90-28 FM CORNContract Should Not Be.Awarded ’ , inventory of applications to be converted, calling into question the basic accuracy of the conversion cost estimate of $74.6 million. The number of applications to be converted has varied from 600 to 200, while estimates of the amount of codeto be converted have varied from 18 million to 10 million lines. More seriously, the conversion cost estimate is incompiete becausethe potentially substantial costs of rectifying problems with the applications’ documentation were not included. FAA, nevertheless, con- tinues to use this incomplete estimate in CORN budget information pro- vided to the Congress. FAA states that it will need to provide 86 employee-yearsof staff sup port to assist the CORN contractor in converting current applications to the new system. However, the basis for this estimate is questionable becauseit is not grounded in a review of each individual application to take into account the specific characteristics, complexities, or problems associatedwith converting each one. In addition, FAA has not determined the extent to which staff actually will be available to support each con- version. The estimated time frame for conversion has doubled from 18 months to 3 years, resulting in additional costs, since the current Com- mon System is to run in parallel with CORN during the conversion. These parallel costs for a 3-year conversion period could be as much as $106 million. This cost is not part of the CORN project cost estimates. Further, the agency has stipulated that the application code is to be con- verted without functional changeto the applications. As a result, the conversion will not enhancethe application code,improve its efficiency, or result in better information. In addition, FAA has done no analysis to determine whether specific applications are worth converting at all. Existing problems, such as poor responsetime, that are associatedwith FAA'S information practices and computer applications may simply be transferred to CORN, at considerable expense. GAO recommendsthat the Secretary of Transportation direct that the Recommendations CORN contract not be awarded. In addition, GAO recommendsthat the Sec- retary direct the FAA Administrator to ensure that future procurements of this type and magnitude be properly justified and planned. Department of Transportation and FAA officials state that CORN should Agency Comments be assessedon its planned benefits-such as enabling FM to better man- age its information resources,provide quality services as needed,and perform its mandated missions. Although expressing general agreement Ehecutive sulnmary with the facts presented in GAO'S report, they believe that GAO has judged CORN on grounds that are not fundamental to its purpose and not material in the larger context. They maintain that CORN has been suffi- ciently planned and justified to warrant its award. GAO maintains that its findings deal with problems and deficiencies that are fundamental to the project’s scopeand implementation and, consequently, support the conclusion that the contract should not be awarded. In responseto GAO'S recommendation not to award CORN, the FAA Admin- istrator has ordered an independent review of the project. The House Committee on Appropriations has directed the Department and FAA not to award the CORN contract until (1) the Committee reviews the results of GAO'S report and FAA'S written responseto it and (2) FAA and Depart- ment officials subsequently discussthe project with the Committee to resolve any outstanding concerns. Page 5 GAO/IMTEG90-38 FM CORN Contract Should Not Be Awarded Executive Summary Chapter I 8 Introduction How FAA Hopes CORNWill Meet Its Needs Optional Levels of Service 8 10 Project Length, Estimated Cost, and Status 10 Objectives,Scope,and Methodology 10 Chapter 2 13 CORN Has Not Been CORNIs Intended to Solve Perceived Chronic System Problems 13 Justified FM Lacks Data on Current System Performance 14 Limited Data Available on ResponseTime Problems 16 lo-Year Growth Projection Is Basedon Sparse, 18 Inadequate Data The Importance of Accurate System Sizing 20 Chapter 3 22 CORNEvaluation Performance and Capability Validation Are Critical for Large Acquisitions 22 Methodology Is Faulty FAA’s Validation Strategy Will Not Adequately Test the 23 ProposedSystems Chapter 4 26 Major Uncertainties Conversion Cost Estimate Is Unreliable Staff Support for Conversion Inadequately Determined 26 29 Loom Over Conversion Estimated Conversion Time Frame Has Doubled 31 Conversion Will Not Result in Better Information 32 Chapter 5 34 Conclusions and Recommendationsto the Secretary of Transportation Agency Comments 36 36 Recommendations Appendix Appendix I: Major Contributors to This Report 38 Y Page 6 GAO/WTECW@ FM CORN Contract Should Not Be Awarded Contenta Abbreviations CORN Computer ResourcesNucleus DOT Department of Transportation Federal Aviation Administration Y FAA GAO General Accounting Office IMTEZC Information Managementand Technology Division Page 7 GAO/lMTJW$bO-88 FM COIW Contract Should Not Be Awarded Chapter I Introduction On February 27,1989, the Federal Aviation Administration (FAA) issued a request for proposals for its largest and most complex general-purpose data-processingacquisition to date: the Computer ResourcesNucleus (CORN) project. CORN is intended to meet the agency’sgeneral-purpose data-processingneedsfor up to 10 years in the following mission and program areas: l airport and aviation activity; l air traffic control and airspace (excluding real-time air-traffic control systems); . aviation safety; l national airspace system facilities; l financial, materiel, and human resources;and l managementsupport. FAA currently supports these areas with its own in-house, general-pur- pose data-processingresourcescalled the “Common System.” The Com- mon System is made up of one International BusinessMachines 3084 computer and 22 Data General MV/16000 computers distributed among 12 agency facilities: headquarters, 9 regional offices, and 2 centers. The major hardware componentsof this system were installed and upgraded during the 1980s. In March 1989, we issued a report that provided information about the CORNproject’s objectives, cost estimates, and implementation approach.1 The report highlighted the tenfold increase in estimated project cost- from $148 million to $1.6 billion-that occurred as the project evolved between 1986 and 1987. In project documents supporting the need for CORN, project officials How FAA Hopes maintain that the agency’scurrent general-purposedata-processing CORNWill Meet Its needs are not being met by the Common System. Specifically, they assert Needs the capacity of the Common System is saturated, causing average responsetimes of 4 secondsthat result in substantial loss of staff pro- ductivity. They estimate that FAA'S general-purposedata-processing needswill grow at a rate of 30 percent per year over the next 10 years. They argue that this growth rate far exceedsany feasible timetable for expanding the Common System becauseof the agency’slengthy procure- ment process.Finally, they maintain that it is not desirable to upgrade rPnxurement: FAA's$1.6-BiUionComputerReaourcesNudeus~ject(GAO/ -44F?$Mar.31,19~ Page 8 GAO/IMTECfJO-3fl FM CORN Contract Should Not Be Awarded chapter I Intzwduction the current CommonSystem becauseit is a hedge-podgeof widely vary- ing equipment that is inefficient and wasteful of critical staff and fund- ing resources. CORN project officials have concludedthat in order to meet FAA’s pro- jected growth in data-processingneedsover the next decade,the agency must free itself from the administrative and technical burdens involved in acquiring, managing, and operating its own general-purposecomputer facilities. Consequently, under the CORN approach, FAA plans to procure data-processingservicesfrom a contractor and closedown its Common System operations. The work of FAA’S information systems staff would shift from managing an in-house system to helping agency users meet their information resourceneeds. The CORN approach calls for FAA to define its general-purposedata- processingneedsfor the next 10 years. On the basis of this information, the contractor-not FAA-determines the computer configuration most appropriate and cost-effective for meeting these projected needs.The contractor is to provide, operate, and maintain the new system, which is to be located in at least two contractor-provided facilities that communi- cate with each other and with FAA centers, regions, and offices, as well as other Department of Transportation locations. The contractor is to provide the agency with specified levels of data-processingserviceson a fee-for-service basis. In addition, the contractor is to convert the current CommonSystem applications’ software and data to the new system on a fixed-price basis; prepare full documentation for the converted applications; provide tech- nical staffing, support, and training to agency users of the system; select and managesubcontractors; and implement system upgrades. Project officials maintain the CORN approach will allow the agencyto easily order increasing levels of data-processingservice over the next 10 years just as if it were a basic utility, such as electrical service. System users, however, will be required to internally budget and pay for their use of CORN, something they do not do in obtaining data-processingfrom the Common System. Consequently,the amount of service that FAA pro- gram offices would receive under CORN would depend on their ability to pay for it. On a broader level, project officials maintain that CORN will promote data-processingstandardization, and data integration and integrity. Page 9 GAO/IMTEG90-38 FM CQRN Contract Should Not Be Awarded chapter1 Introdnction CX~RNincludes options for processingapplications other than those cur- Optional Levels of rently run on the Common System. These additional applications include Service FAA data-processingrequirements that are not part of the Common Sys- tem, as well as data-processingneedsof other elements of the Depart- ment of Transportation, such as the Coast Guard, the Federal Highway Administration, and the Federal Railroad Administration. Detailed requirements, feasibility, and cost/benefit studies have not been done for the optional levels of service. Instead, they are to be done before implementing these options. The CORN contract is expected to cover an initial S-year implementation Project Length, period, followed by five l-year renewals. FAA estimates the total con- Estimated Cost, and tract value to be about $1.6 billion: $876 million to meet FAA processing Status requirements, and $619 million to meet optional data-processing requirements. The Secretary of Transportation designated CORN as a major systems acquisition in September 1987. The project was reviewed by the Trans- portation SystemsAcquisition Review Council and approved by the Deputy Secretary of Transportation in January 1989. FAA issued the CORN request for proposals on February 27,1989. Vendors’ cost and technical proposals were due by August 30,1989. FAA is currently reviewing the proposal material submitted and anticipates that the CORN contract will be awarded in the summer of 1990. However, in the House Committee on Appropriations report supporting the fiscal year 1990 appropriations bill for the Department of Transportation, FAA and the Department were directed to defer awarding the CORN contract until (1) the Committee reviews the results of our evaluation of CORN and a writ- ten FAA responseto our report and (2) FAA and Department officials sub- sequently discussthe project with the Committee to resolve any outstanding concerns. Regardlessof the future of CORN, FAA plans to upgrade the mainframe portion of the CommonSystem. The existing contract for the minicom- puter portion of the system already has provisions for implementing upgrades. At the request of the Chairman, HouseCommittee on Appropriations, Objectives,Scope, and Subcommittee on Transportation and Related Agencies,we reviewed Methodology FAA'S proposed procurement of the CORN system. As agreed with the Chairman’s office, our objectives were to Page10 GAO/WTEC-gO-8SFAAC0RNContmctShouldNotBeAwarded Chapter I hdXOdWtiOll l determine the adequacy of FAA's assessmentof its current and future data-processingneeds, l determine the adequacy of FAA'S methodology for validating the ven- dors’ proposed solutions for meeting agency data-processingrequire- ments, and l determine the adequacy of FAA'S preparation and planning for the con- version of current applications to the CORN system. To determine the adequacy of FAA'S assessmentof its current data- processingproblems and future needs,we met with officials at the Department of Transportation and FAA to discussCORN'S development, goals, objectives, estimated cost, implementation approach, and manage- ment. We also reviewed key documents-such as the CORN requirements analysis, feasibility study, mission need statement, project charter, request for proposals, and draft implementation plan-used by FAA and the Department of Transportation to review and approve the CORN approach. We analyzed CommonSystem performance monitoring tapes to determine the extent of capacity and responsetime problems, and reviewed other data on the system’s performance. We visited FAA'S Mike Monroney Aeronautical Center, where about half of the agency’sgen- eral-purpose data processingis performed, to discusssystem perform- ance issuesand review performance data. To determine the appropriateness of FAA'S validation methodology, we reviewed the rationale for the chosenapproach and the method by which it is to be implemented, met with officials of the General Services Administration’s Federal SystemsIntegration and ManagementCenter to discusstheir independent assessmentof the approach, and considered applicable federal regulations. To assessthe adequacy of preparation and planning for the CORN con- version, we reviewed the project’s 1987 conversion study, the original and revised CORN documentation package,the methodology for estimat- ing agency resourcesneededto support the conversion, and the manage- ment plan for implementing the conversion. We discussedthe conversion with officials at the General ServicesAdministration’s Federal Software ManagementSupport Center, FAA program offices, the CORN project office, and the Department of Transportation’s Office of the Inspector General. Page11 chapm I Introduction We performed our work at the Department of Transportation, FAA, the General ServicesAdministration in Washington, D.C.;the Federal Sys- tems Integration and ManagementCenter, the Federal Software Manage- ment Support Center in Falls Church, Virginia; and at FAA’S Mike Monroney Aeronautical Center in Oklahoma City, Oklahoma. Our review, performed from January 1989 to April 1990, was conducted according to generally acceptedgovernment auditing standards. We obtained the views of Department of Transportation and FAA officials on this report and have incorporated them where appropriate. Page 12 GAOAMTEWWS FM CXlRN Contract Should Not Be Awarded Chapter 2 CORNHas Not Eken Justified While the concept of CORN-t0 contract for data-processingservices from a vendor-may be acceptable,key justifications for the CORN pro- ject are not properly supported, raising fundamental questions about the rationale for the project. In promoting CORN, project officials maintained that CommonSystem users were receiving inadequate responsetimes becausethe system was “at capacity” or “saturated.” However, we found no evidenceproving that perceived responsetime problems are being causedby a lack of processorcapacity or would be solved by CORN. FM has virtually no data on responsetimes indicating the frequency, magnitude, and causeof such problems. Project officials also maintain that the agency’s data-processingneedswill increase faster than its abil- ity to upgrade the current system. However, the data and methodology used by FAA to make its growth projections are inadequate, raising fur- ther doubts about the necessity for the CORN approach. Project officials have repeatedly maintained that FAA needsCORN CORNIs Intended to becausethe current Common System doesnot adequately meet the Solve Perceived agency’s needsand becausethe level of service provided by the system Chronic System is continuously degrading. Specifically, they maintain that the Common System is “saturated” and that current capacity must be increased by Problems 160 percent to achieve required service levels. The officials also claim that the system’s responsetime is inadequate, averaging 4 secondsand causing a productivity loss of $37.6 million a year. In addition, they maintain that the agency’sgeneral-purposedata-processingneedswill grow at a compounded rate of 30 percent per year, over the next 10 years. Thesejustifications have been continually stressedin documents and briefings used to explain why CORN is needed.For example, the system is described as “repeatedly saturated” in a February 1989 “Project CORN Basic Facts” briefing document. The draft CORN implementation plan provided to the Department of Transportation in April 1989 stated that the current system has reached its saturation level, suppressing service to FAA personnel. The system was also described as “saturated” in a Sep- tember 1989 briefing document prepared for the FAA Administrator. Earlier project documents and briefings used to justify CORN also assertedthat the CommonSystem was “overloaded,” “at capacity,” or “saturated.” Briefings given to the Congressand to us have also stressed that lack of computer capacity is causing inadequate responsetimes and impairing staff productivity. Page 13 GAO/IMlEC30-33 FM CORN Contract Should Not Be Awarded chapter 2 C43RNIIaa Not Been Judfled Project officials were unable to provide us with adequate support for FAA Lacks Data on their assertions about capacity and responsetime problems becauseFAA Current System lacks a central capacity and performance managementprogram for the Performance Common System. Such a program is important to ensure maximum use of existing resourcesand adequate capacity for growth. Not only is capacity and performance managementa commonly-acceptedbusiness practice, but the Federal Information ResourcesManagementRegulation Part 201-30 requires agenciesto perform capacity managementactivi- ties in planning, acquiring, and using computer resources. An effective capacity managementand performance monitoring pro- gram needsto addressboth performance managementand capacity planning. Performance managementinvolves analyzing the performance of computer systems to determine how resourcesare currently utilized and how such utilization can be improved. Capacity planning assistsin forecasting computer resource requirements to ensure that enough capacity exists when needed. While data are captured and analyzed on several componentsof the CommonSystem, officials at both FAA and the Department of Transpor- tation stated that there is no central capacity managementprogram for the Common System that captures and analyzes Common System utiliza- tion and responsetime data on a systemwide basis. According to project officials, instead of having a central program, each of the 12 facilities is responsible for independently conducting capacity planning. We found, however, that the lack of a central capacity and performance managementprogram has resulted in inadequate data on the system’s utilization and responsetimes. Very little data on utilization and responsetimes is available for the minicomputer portion of the system. Performance monitoring is much more in evidence for the mainframe portion of the system. But even there, project officials could not provide us with critical performance information, especially on responsetimes, that is key to justifying the need for CORN. Consequently, the officials were unable to provide support for the key assertion that responsetime problems were causedby insufficient capacity and therefore could be solved by abandoning the current system and moving to CORN-a capac- ity-oriented approach. Project officials state that the current mixture of mainframe and min- icomputers doesnot lend itself to centralized capacity management,and that it is difficult to monitor the minicomputers’ performance. They Page 14 GAO- FM CORN Ckmtract Should Not Ee Awarded chqmr2 CORNIIaa Not Been JuMfled claim that CORN will create an environment that will allow FAA to have centralized capacity management. Project officials claim that insufficient processorcapacity is causing the Limited Data Common System to provide users with poor responsetimes, averaging 4 Available on Response seconds,resulting in annual productivity lossesof $37.6 million. They Time Problems calculated that by reducing responsetime to a maximum of 2 seconds, CORN would allow FAA to realize cost-avoidancesavings of $376 million over 10 years due to improved productivity. Project officials, however, could not provide support either for their assertion that the CommonSystem’s averageresponsetime is 4 seconds or for their assertion that perceived responsetime problems are caused by a lack of Common System processorcapacity. Information resource staff responsible for the operation of the mainframe system stated that they had not collected overall responsetime data during the last 2 years becauseof the architecture of the agency’stelecommunications system.’ Further, no quantifiable data were provided to support the assertion that the averageresponsetime on the minicomputer systems is 4 seconds. Since project officials were unable to provide us with performance moni- toring data to support their assertionsthat the 12 Common System facil- ities are at capacity and were the causeof poor responsetime, we independently analyzed the limited processorutilization data that were available. Specifically, we analyzed July 1989 and January 1990 utiliza- tion data from the system’s mainframe facility, which processesabout 44 percent of the work load; and utilization data from February through May 1987 for 4 of the 12 minicomputer facilities that processthe remaining 66 percent.2 For high-priority interactive work loads,3the July 1989 data showed that the mainframe processorwas at or below 69 percent utilization for 10ffichls responsiblefor the operationof FAA’smainframestatedthat they have decidedto start collectingvery limited responsetime data The telecommunicationslines beingmonitoredrepresent 2.6 percentof the mainframeusers. 2The 1989and 1900data were provided at our request.The minicomputerutilization data from Feb- ruary through May 1987are the mostrecentdata collectedby FM on thesemachines. %Iigh priority interactive work loadsinclude thoseapplicationsprogramsthat require on-line PITXXSS~@ and are consideredmostcritical to FM. Page 16 GAO- FM C0lW Contract Should Not Be Awarded chapter 2 CORN Has Not Been Ju&Uled 99 percent of the time between 7 a.m. and 7 p.m. on nonholiday week- days. Similarly, the January 1990 data showed the mainframe was at or below 67 percent utilization for 99 percent of the samehours for the sametype of work load. It is highly unlikely that these levels of proces- sor utilization would causepoor responsetimes. Our analysis of utilization levels for the minicomputer portion of the Common System yielded similar results. Basedon the available data, the mean prime-time (8 a.m. to 4 p.m. on nonholiday weekdays) utilization was between 16 and 26 percent for all of the selectedsites except for one facility, the Aviation Standards National Field Office, where it was 46 percent.4Since 1987, when these data on the minicomputers were col- lected, FAA has upgraded all of its minicomputers. The upgraded machines have over 3.6 times the processingpower of the old ones, according to industry specifications. Although FAA doesnot regularly collect utilization data from the minicomputer facilities, the available chargeback reports provide data on total processorutilization at each data-processingfacility. Using extrapolations of the 1987 data and 1989 chargeback reports, we estimate that prime-time, processorutilization was at or below 66 percent 99 percent of the time between 8 a.m. and 4 p.m. on nonholiday weekdays for 7 of the 8 facilities analyzed.6It is highly unlikely that processorutilization at these levels is the causeof responsetime problems. Given the absenceof direct measurementsof responsetime, we analyzed the interactive, time-sharing option subsystem of the mainframe and found responsetimes of less than 1 second.6Agency officials said that the time-sharing option subsystem is not heavily used and that response time problems occurred in other subsystems,namely their data base managementsystem and communications handlers. However, agency officials were unable to provide us with data documenting the existence of poor responsetimes in these subsystems.Moreover, we observedthat the data base managementsystem and the communications handlers are 4Although usedin the CORNrequirementsanalysis,thesedata are sparseand incomplete.During the Cmonth monitoring period,the selectedsites had gapsin their utilization data ranging from about 1 to 3 months. 6Chsrgebackreports are generatedmonthly by FAA and distributed to eachof its data-processing facilities for information purposesonly. No chargesare collscted.The reports contain information describingthe total resources(Le.,central processorunit time, memory,tape storage,etc.)usedby eachfacility and the dollar amountthe facility would be billed for the useof thoseresources.Our analysis of prim&me utilization is basedon the extrapolation of data from thesereports. 6Thetime-sharingoption subsystemIs an option of the International BusinessMachines3034operat- ing systemthat allows usersto interactively sharecomputertime and resources. Page 16 GAO/IMTECfKMg FM CORN Contract Should Not Be Awarded .I :, chapter 2 CORN Iiaa Not Been JustifM assigneda higher processingpriority than the time-sharing option sub- system. Therefore, if the lower priority time-sharing subsystem was not suffering from poor responsetime resulting from a saturated processor, it is unlikely that these other higher priority systems were suffering in that way either. Perceived Slow Response The absenceof data on the responsetimes does not, of course,mean that Times May Be Caused by responsetime problems do not exist. CommonSystem users have com- plained about slow responsetimes. However, as our analysis of the sys- Other Problems tem’s available utilization data and processingpower indicates, response time problems do not appear to be causedby insufficient processor capacity. Other factors that may be causing FAA’s responsetime problems include the following: 9 Inefficiencies in the design of application programs may causeprocess- ing delays that prevent the current system or the proposed CORN system from providing a 2-secondresponsetime. 9 Contention for peripheral devices(e.g., disk and tape drives) may cause processingdelays. l The communication system linking users to the Common System may be causing delays. An official at the mainframe facility believes that responsetime problems may be causedby FAA’S Administrative Data Transmission Network communications system. CORN requires 2-second responsetime within the contractor’s facility, not at the users’ termi- nals. As a result, delays due to telecommunications problems or the input devices,such as microcomputers, will not be corrected by CORN. Project officials have asserted,without support, that CORN and the agency’s switch to the new FTS 2000 telecommunications system will eliminate current responsetime problems, . Inefficient managementof current system resourcesmay be causing delays. For example, the mainframe system contains an operating sys- tem module that is designedto optimize the tradeoff between throughput and responsetime. In order to do this, the module must be given control over which batch jobs are allowed in the computer’s mem- ory and when they are allowed in. FAA is denying this control to the module by not allowing many batch jobs to be initiated during prime- time hours. As a result, we identified instances where jobs took only minutes to complete and utilized only hundredths of a secondof proces- sor time, yet waited hours before they were initiated. Oneof the worst Page 17 GAO/lMTECW FM CORN Contract Should Not Be Awarded r chapter 2 CORN Haa Not Been JumtUled casesidentified was a job that unnecessarily waited 80 hours to be initi- ated. Onceinitiated, this job was completed in 12 minutes and used only 0.06 secondsof processortime. Project officials constructed a lo-year growth projection to estimate the lo-Year Growth data-processingresourcesrequired to meet FAA’S needsthroughout the PrOjeCtiOn IS Based on life of the project. Their projection is comprised of four components: Sparse, Inadequate . current demand, the measurable demand that existing application pro- Data grams place on data-processingfacilities; . latent demand, the difference between the measured demand and the demand that would be exerted on a computer system if an acceptable level of service were available; l projected mission growth, the increase in the demand for data-process- ing resourcesresulting from the agency’sadditional use of automated systems to meet future mission needs;and l new demand, the demand that planned or developing application pro- grams will exert on the system when they are implemented. According to FAA estimates, these four growth components,when com- bined, result in a projected growth rate of 30 percent per year com- pounded annually. Over 10 years, this amounts to demand growing by about 1300 percent. Making long-range data-processinggrowth projections is inherently dif- ficult. To make accurate growth projections, it is essential to have a complete and thorough understanding of a system’s historical utilization patterns, current demands,and the effect that future applications and changesin the operating environment will have on the system. However, FAA lacks the data to assesscurrent demand, latent demand, and mission growth. FAA’S current demand estimate is flawed becauseit is based on incom- plete and inadequate processor-utilization data. The minicomputer data used by FAA only covered a 4-month time frame from 4 of the agency’s 12 minicomputer facilities, and these data had gaps of 1 to 3 months. Project officials stated that the missing data resulted from periods when the monitoring system used to extract utilization data from the system was not operating. The incompletenessof the minicomputer utilization data prevents FAA from accurately identifying daily, weekly, or monthly levels of processorutilization for more than half the Common System’s processing.The lack of complete data and the short measurementperiod Page 18 GAO/lMTJGBO-2S FM CORN Contract Should Not Be Awarded chapter 2 C4MtN Has Not Been Jwtified make it difficult for FAA to obtain a clear understanding of the demands currently placed on the Common System. The latent demand estimate is based on the increase in demand that pro- ject officials say was observed after two upgrades to the mainframe. Project officials, however, could not provide any utilization data docu- menting this increase. Moreover, the officials could not furnish any data that provide a basis for measuring the amount of latent demand in the other half of the Common System-its minicomputer systems.Conse- quently, the latent demand estimate is basedpartially on speculation, rather than on quantifiable data. FAA’S estimate of projected mission growth is also flawed. To make a reliable forecast of this growth over the lo-year life of the contract, the agency needs(1) complete and accurate historical data on system utili- zation, and (2) an in-depth understanding of the projected changesin agency operations that will affect FAA’S general-purposedata-processing resources. As previously discussed,FAA has a limited amount of historical utiliza- tion data. These limited data do not provide a sound basis for making an adequate lo-year growth projection. To compensatefor the lack of his- torical utilization data, FAA officials relied on their knowledge of past growth in the agency’s data-processingfacilities to validate their growth projections. Project officials stated that, on the basis of their knowledge of central processingunits obtained over the past 10 years, their projec- tions for the next 10 years appear to be accurate.’ Regarding projected changesin agency operations, government and industry experts recommendthat growth projections be tied to natural forecasting units8 and thus be based on real-world actions. However, project officials did not use natural forecasting units as part of the growth analysis. As a result, the growth projections are not tied to any factor that would causethe 30percent annual growth rate to slow down or plateau at any point during the lo-year life of CORN. FAA’S projection 7Projectofficials statedthat the data-processingindustry will experiencea similar growth pattern over the next 10 years ashas beencalculatedfor the CORNproject.Whenwe askedprojectofficiala for the support for this statement,they said that the statementwas basedon a singlemagazine article. sA natural forecastingunit is a functionaUy+rient.edunit of measureof the work that systemusers perform. For a Fmancialinstitution, for example,natural forecastingunits would include loan applica- tions pmcesaed,credit reports prepared,and checksprocessed. Page 19 GAO/lMTEGfWP3 FAA CORNContract Should Not Be Awarded . Chapter 2 CORN Has Not Been Justifkd methodology simply continues to compound the 30-percent growth rate year after year. Optional Levels of Service Embeddedin the growth projections are data-processingrequirements Based on Unsupported for applications that are not processedon the Common System. These include some additional FAA applications, as well as applications used by Assumptions other elements of the Department of Transportation such as the Coast Guard, the Federal Highway Administration, and the Federal Railroad Administration. Currently these applications are processedon dedicated hardware, time-sharing systems, or outside FM. Levels of service are included in CORN to accommodatethese additional applications on an optional basis. These levels of service account for approximately 40 per- cent of the estimated $1.6 billion value of the CORN contract. No detailed requirements, feasibility, or cost/benefit studies were done for the optional levels of service portion of the procurement. Sizing for these levels of service was basedon the assumption that the growth rate, acquisition schedule, and the ratio of peripheral devicesto the central processingunit would be identical to that of the CommonSystem’s requirements. FAA'S growth methodology results in extremely large annual leaps in the The Importance of system’s size and capabilities, especially after the first few years of Accurate System CORN. By the tenth and final year of the CORN contract, the projected Sizing growth rate results in a system almost 1300 percent the size of the cur- rent one. Growth of this magnitude would, for example, result in a sys- tem capable of providing 390,000 hours of system accesstime per calendar day and producing over 16 million printed pagesof output per calendar day in the final year of the CORN contract. A system of this - magnitude would provide 32.6 hours of accessper person per day for 12,000 users. Similarly high growth is projected for system input/output transfers, number of tape reels used, gigabyte9 of disk storage, and cen- tral processoruse. However, project officials have not addressedbasic issuesthat are naturally associatedwith growth of this speedand mag- nitude, such as (1) determining how the 30-percent level of growth cor- relates with expected increasesin the number of system users over the next decade,or (2) determining whether FAA's and the Department’s activities would in fact require the production of over 16 million pages of output every day of the year. Page 20 GA0/IMTEG90&3 FM CORN Ckmtract Should Not Be Awarded Chapter 2 CORN Has Not Been Justified The larger and more demanding a computer growth requirement is, the lower the number of vendors available to meet the requirement. A pro- curement as large as CORN can only be handled by the largest computer companies.If the 1300-percentgrowth requirement is in fact too high, then it may have unnecessarily precluded other vendors from compet- ing. In addition, vendors must design solutions and attempt to cover the costs for a system that can handle a potential work load of this size. Therefore, the per-unit cost can be expected to be high for lower levels of usage. Page 21 GAO/lMTlW9O&S FM CORN Contract Should Not Be Awarded Chapter 3 CORNEvaluation Methodology Is Faulty In a procurement as large and costly as CORN, it is critical for FAA to have meaningful and accurate information for evaluating the most effective solution for meeting agency requirements over the contract’s lo-year period. However, a key element of FAA'S methodology for validating the vendors’ proposed solutions-an operational capability demonstra- tion-is deficient in two ways. First, information that FAA provided to the vendors to assist them in developing their proposals was incomplete. Second,the sample work load that FAA developed for use in the demon- stration is extremely small and unrepresentative of the Common Sys- tem’s total work load. As a result, the demonstration will not provide adequate data for accurately evaluating the vendors’ proposals and their proposed charges for data-processingservice. This deficiency could have cost ramifications throughout the life of the contract. Performance and capability validation techniques are important to Performance and agenciesthat are acquiring a large amount of data processingequipment Capability Validation or services.Performance and capability validation helps to reduce the Are Critical for Large risks of acquiring insufficient or excessivecapacity, inadequate func- tional capability, and uneconomical capability. Acquisitions Federal Information ResourcesManagementRegulation 201-30.013-3 stipulates that the selection of performance evaluation techniques shall be commensuratewith the programmatic risks of inappropriate or insuf- ficient data-processingcapacity. Present and forecasted data-processing work loads, anticipated system life costs, validation costs to the govern- ment and offerors, and objectivenessand fairness in the acquisition pro- cessare factors to be considered. The most precise performance validation technique is benchmark test- ing. A benchmark is a set of computer programs and associateddata tailored to represent a particular work load. A benchmark test is a user- witnessed demonstration on a vendor’s proposed computer system done to validate system performance or cost. Benchmark tests are used to assesshow a vendor’s system will processthe work load (e.g., process- ing speed,resource consumption) and to compare the performance of several systems. Page 22 GAO/MTEG9O-38 FM COBN Contract Should Not Be Awarded CORN Evaluation Methodology b Faulty FAA’s Validation Under the validation methodology developed by FAA, vendors will be required to perform an agency-designedoperational capability demon- Strategy Will Not stration in which they will processa portion of the agency’swork load Adequately Test the on a subset of their proposed systems. The makeup of this sample work load is critical to the thoroughness and adequacy of the demonstration Proposed Systems becauseit will exercisethe vendors’ billing algorithms, which will be the basis for charging the government for the servicesprovided,’ The dem- onstration is intended to aid FAA in determining how well the vendors’ proposed solutions will meet the agency’sprojected requirements. Problems Cited by Problems with FAA’S demonstration were originally cited by the General Independent Review ServicesAdministration’s Federal Systems Integration and Management Center, which was asked by FAA to perform an independent review of a draft of the CORN request for proposals.2In its September 1988 report to the agency,the Center raised methodological issuesregarding the dem- onstration, stating that the agency’s demonstration methodology would not accurately validate the cost of processingthe agency’swork load on a vendor’s proposed system. The Center stressedthat it is crucial for the agency to accurately validate its work load in terms of the billing units proposed by each vendor. According to the Center, an error in validating the number of billing units required to processFAA’S work load could have “staggering cost consequencesover the lo-year contract life.” The Center stated that the only reliable method for performing this valida- tion is through the use of a “government-furnished benchmark.” We dis- cussedFAA’s validation methodology as it appears in the final draft of the CORN request for proposals with Center officials. The officials said that they would prefer that FAA use a more rigorous test, which would include a larger, more representative sample of the agency’swork load. Data and Validation In our own review of FAA’S proposed validation methodology, we found Problems Remain that serious problems remain in its approach. Onebasic problem is that the data supplied by FAA to the vendor community to aid them in sizing a system capable of meeting the agency’s general-purposedata-processing needsfor the lo-year life of the contract are flawed. The agency sup- plied performance data for its mainframe system on magnetic tape. ‘The actual algorithmsdesignedby eachvendor will be extremely complex,including dozensof pric- ing elementswith complicatedrelationshipsamongthem.Properly exercisingthesealgorithmsis crit- ically important to understandingthe utilization costaof the resourcesbeingoffered. 2TheCenterwas establishedin 1972to assistfederal agenciesin acquiring,managing,and using information systemsand technology.It provides agencieswith a wide rangeof technical and contrac- tual assistanceon a cost-reimbursablebasis. Page 22 GAO/IlWEC-9O-2S FM CORN Contract Should Not Be. Awarded Chupter 3 CORN Fivaluatlon Methodology Ia Faulty However, portions of the data neededto accurately model FAA’S current system were either unusable or missing from the tapes, making it impos- sible for the vendors to obtain information on such things as disk drive activity. In addition, information neededto identify the characteristics of batch and on-line application systems3and transaction volumes was not pro- vided. Consequently, vendors would have to make assumptions about key characteristics of the current mainframe system in order to perform their modeling. Another problem with FAA’S approach is that the sample work load to be used in the demonstration doesnot represent the CommonSystem’s work load. According to FAA, the CommonSystem’s current work load is made up of almost 290 application systems,both batch and on-line, totaling nearly 16 million lines of code.However, the sample work load to be used in the demonstration is made up of only 12,000 lines of code from parts of two application systems. Project officials admit that the application subsystemsto be used in the demonstration are only “slivers” of the CommonSystem’s total work load and are not representative. Nevertheless,they maintain that these subsystemsare typical examples of the types of applications that are processedin FAA’S current data-processingenvironment becausethey have qualitative attributes found in many of the agency’s applications. For example, according to project officials, the programs both use a high-level programming language and data base managementsystems. In addition, one application is processedat a minicomputer facility, while the other is processedat the mainframe facility. Officials also maintain that a more rigorous validation methodology would be hard to develop and more costly to both the agency and ven- dors becauseof the difficulty in defining and constructing a larger, rep- resentative work load. We recognizethat the two subsystemschosenfor the demonstration have attributes similar to other types of CommonSystem programs. However, becauseof their small size and level of activity, they cannot provide F&I with a realistic view of the resource utilization cost per transaction or cost data resulting from work loads of increasing size. As 3A batch systemprocessesonegroup of transactionsentirely beforeprocessingthe next group.On- line refers to users’ability to accessand interact with a computervia a terminal. Page 24 GAOjIMTEG90-3S FM co&N Contract Should Not Be Awarded .W a.2 chapter8 CORN Eval~tion Methodology ImFaulty a result, the demonstration would not provide FM with adequate data to evaluate vendors’ proposals and their proposed chargesfor data- processingservice. Page 26 GAO/RtiTECM FM CORN Contract Should Not Be Awanled Chapter 4 I Major Uncertainties Loom Over Conversion The General ServicesAdministration’s Federal Software Management Support Center starkly characterizes software conversion1as “labor intensive, managementintensive, machine-resourceintensive, and dead- line intensive. In short, it has all of the wrong attributes for a successful enterprise, and many problems will arise.” Becauseof the extremely large size of the CORN conversion- nearly 290 software applications made up of over 14.8 million lines of code-and the fact that FAA staff will need to assist the contractor in the effort, the conversion will be costly, disruptive, and time-consuming. Unfortunately, FAA has not ade- quately prepared for the conversion, resulting in major outstanding problems and uncertainties. Specifically: . The conversion could cost substantially more than the current $74.6mil- lion estimate becauseit is basedon an unreliable inventory of applica- tions and excludes the cost of fixing major documentation deficiencies. . Project officials state that 86 employee-yearsof agency staff support will be neededfor the conversion. This figure, however, does not include all the conversion tasks and is not based on a review of each individual application to determine the amount of conversion support needed.More significantly, the availability of agency staff to support the conversion will not be determined until after the CORN contract is awarded. l The estimated time neededfor the conversion has doubled from 18 months to 3 years. The Common System will continue to operate until the conversion is complete, at a potential 3-year cost of $106 million. This cost will need to be funded in addition to CORN. l The conversion will not correct problems in the applications, which may be causing inefficient processingof data, nor will it improve the agency’s information structures. Project officials estimate that the conversion will cost $74.6 million.2 An Conversion Cost essential step in making such an estimate is preparing and validating the Estimate Is Unreliable inventory of applications software and files to be converted. The $74.6- million estimate is based on project officials’ initial attempt in 1987 to establish an inventory of applications to be converted. This effort did ‘Software conversionis the transformation,without functional change,of computerprogramsand data to permit their useon replacementdata-processingequipment. 2Thecomplexity and costof the conversionwill vary with the type of hardwarebeingproposedby a vendor.The projectteamdevelopedcostestimatesfor severalhardware scenarios-rsnging from $10.6million for a replacementsystemfully compatiblewith the current system,to $126million for a completelynoncompatiblereplacementsystem.The $74.~million figure consistentlyusedin budget documentscorrespondsto the estimatefor convertingto an International BusinessMachines-compati- ble system.This estimatedoesnot include the costof convertingother FAA or Departmentof Trans- portation applicationsaspart of CORN’soptional levels of service. Page 26 GAO/IMTEGgO&3 FM CORN Contract Should Not Be Awarded chapter 4 Mejor Uncertaintie Loom Over Conversion not succeedin establishing reliable results, and therefore the cost esti- mate remains questionable. Between I987 and 1989, estimates of the number of applications that need to be converted ranged between about 200 and 600, while esti- mates of the lines of applications codeto be converted varied from about 10 million to 18 million. Project officials maintain that the vari- ance in the count of applications to be converted is largely a result of different ways in which they grouped related applications. They have no explanation for the changesin the corresponding line counts, how- ever. As late as February 1989 project officials concededin a CORN “basic facts” briefing document that the conversion will not be easy because“FAA’S knowledge of what currently exists to be converted is far from perfect.” This uncertainty over the applications inventory reached a critical point shortly after the request for proposals was issued in February 1989. FAA received vendor complaints that the CORN documentation package- which was supposedto include current code for all applications to be converted-was confusing, incomplete, and did not provide an adequate basis for making a firm, fixed-price offere3In April 1989 FAA informed prospective offerors that the documentation package “is being revised to include the information neededto fix price the effort.” FAA analysts subsequently reviewed the documentation package and found that it contained obsoleteversions of applications, along with applications no longer in use. In addition, someapplications systems were listed for con- version in the CORN request for proposals, but had no code included in the documentation package.Project officials concluded that the materi- als provided by its staff for this package “were not sufficiently accu- rate.” They subsequently characterized the documentation package as a first attempt to gather code for the vendors’ use in preparing proposals for CORN. They maintain that the initial documentation package pro- vided information “to enable offerors to analyze FAA’s source codeto appreciate the scopeof the conversion effort.” Between February and August 1989, the conversion count continued to vary from 204 applications to 247 to 279, while the line count changed from 12.8 million to nearly 14.8 million (with an interim estimate of 10 3An FM fraud hotline complainttriggereda review of the issueby the Departmentof Transporta- tion’s Office of the InspectorGeneral.The resulting report is scheduledfor releaselater this year. Page 21 GAO/lMTEGSO-2SFM CORN Contract Should Not Be Awarded ChaPtm 4 Major Uncertainties Loom Over Convereion million lines done between these two line counts). The revised documen- tation package and related amendment to the request for proposals, list- ing 279 applications with nearly 14.8 million lines of code,was issued only 3 weeks before vendor proposals were due in August 1989. Accord- ing to an FAA memorandum, the revised package still had missing and incorrectly labeled components.Project officials issued another revision in February 1990 that raised the number of applications to 289 and increased the line count to slightly more than 14.8 million. Continued changesin the documentation package between the time of expected contract award and the actual conversionscould drive up conversion costs by triggering cost-adjustment provisions specified in the request for proposals. The conversion cost estimate is also unreliable becauseproject officials did not include an estimate of the cost of enhancing or updating out-of- date applications’ documentation- such as system, program, and users’ manuals-or creating such documentation where none exists. Providing new documentation for all converted applications, except for functional requirements, is required of the contractor under CORN. These documen- tation costs could be substantial becauseproject officials estimate that the documentation is, on average,only about 60 percent current and complete. Project officials originally told us that the additional costs of fixing doc- umentation deficiencies would be offset by an anticipated decreasein the number of lines to be converted as a result of revising the applica- tions inventory. When the new inventory did not result in the antici- pated reduction, project officials speculated that the additional documentation would probably add less than 10 percent to the $74.6- million conversion cost estimate. Officials now maintain that they have developed a new cost estimate, for use during review of vendors’ pro- posals, that includes the documentation costs. For planning and budget- ing purposes, however, FAA is continuing to use the original $745million estimate and continues to submit this figure to the Congressin CORN budget information. Page 28 GAO/lMTEGBO3S FM CORN Contract Should Not Be Awarded chapter 4 Mqjor U&tiea Loom Over Convemion FM information systems staff and program office staff will need to sup- Staff Support for port the conversion by helping the contractor understand the functional- Conversion ities of the applications being converted, assisting in resolving technical Inadequately issuesand documentation problems, and reviewing the converted sys- tems prior to accepting them. In presenting the project to the Depart- Determined ment of Transportation for approval, FAA committed itself to a series of actions that it termed “critical to the successof CORN." Among these actions, FAA stated that it planned to “establish an estimate of FAA/m [Department of Transportation] staffing resourcesrequired and their availability for each [application] system conversion prior to contract negotiations.” We found, however, that project officials devel- oped only aggregatedestimates of the amount of agency employee-years neededfor the conversion and have not determined the extent to which agency staff with appropriate skills will be available to support the conversion. Estimated Amount of Project officials state that a total of 86 employee-yearsof staff support Employee-Years Is Poorly will be neededto assist in the conversion, excluding employee training.4 However, this figure is not basedon a review of each individual applica- Supported tion to take into account the specific characteristics, complexities, condi- tion of documentation, or other problems associatedwith converting each one. Instead, project officials developed a formula basedon general assumptions about the amount of time neededto perform typical con- version tasks and the number of lines of code involved. They told us that they lumped the mainframe-based applications into one group and applied this formula to the aggregate,and used the sameprocedure for the minicomputer-based applications. Project officials were unable to provide us with documentation showing the basis for their general assumptions or how they carried out their calculations. FAA initially informed the Department of Transportation that the con- version would require 46 employee-yearsof FAA staff support6 Project 4Projectofficials have identified the needfor an additional 6 employeeyearsto tram agencystaff in using the CORNsystem.This estimateis very soft, however.Projectofficials informed the Depart- ment of Transportation that “the numben of employeesinvolved in the application systemsopera- tion varies too much from application systemto application systemto makea meaningfulestimate for conversion/transition.” They maintain that a fm estimatecannotbe madeuntil after contract award. ‘The 46 employeeyearestimateis cited in a projectmilestonechart provided by FAA to the Depart- ment of Transportation in April 1989in responseto the Department’srequestfor additional mforma- tion on projectplanning. Page 29 GAOjIMTECf%28 FM CORN Cm~tract Should Not Be Awarded Chnpter 4 Major Uncertainties Loom Over Conversion officials said that this number was a typographical error, since the esti- mate they had actually calculated was 86 employee-years.The 86 employee-year estimate is used in the CORN request for proposals and remains current. However, this estimate doesnot include the staff time neededto prepare the conversion packagesto be provided to the con- tractor for each application. Project officials estimate that this task will require about 28 employee years. Project officials maintain that their employee-year estimate cannot be refined until after the CORN contract is awarded, since the amount of staff resourcesneededwill depend to a large degreeon how compatible the current applications software is with the contractor’s system. They are nevertheless confident that they have a “pretty solid feel” for the amount of employee-yearsneededfor the conversion, In the CORN request for proposals, they stipulated that vendors bidding on the con- tract must develop preliminary conversion plans that require no more than 86 FAA employee-yearsof support (42 years for the mainframe applications and 43 years for the minicomputer applications). Project officials acknowledge, though, that this doesnot preclude FAA from pro- viding a higher level of support, if necessary.For example, they said that if funding for the conversion is constrained, someof the contrac- tor’s conversion work could be shifted to agency staff. FAA Staff Availability Along with estimating the number of employee-yearsneededto support Remains Undetermined the conversion, it is necessaryto determine the extent to which agency employeeswith the neededskills can actually be made available from their regular work to support the additional work involved in the con- version As noted, FAA'S promise to determine staff availability for each application conversion prior to contract negotiations has not been car- ried out. Project officials told us in March 1990 that they were in the processof beginning to determine staff availability. They maintain, however, that this activity cannot be made final until after the contract is awarded and the contractor’s conversion plan is completed. This leaves unresolved the crucial issue of the extent to which appropriate FAA staff will actually be available to support the conversion. A senior information resourcesmanagementofficial acknowledged that the con- version work load would put a heavy burden on the staff, but said that the agency would find a way of working through it. If staff are not available to the extent neededto support the conversion, agency offices may need to turn to other contractors for help. For exam- ple, officials in FAA'S Human ResourceInformation Division said that Page 80 GAO/IMTECBO-3S FM CORN Contract Should Not Be Awarded chapter4 Mnjor Uncertaintie Loom Over Conversion they do not have enough staff to maintain the current personnel man- agementsystem while supporting the complex task of reengineering the system for CORN. Consequently, they have requested $2.3 million in con- tract help for the conversion to CORN. The manager of the payroll system expressedsimilar concern about the lack of staff to support the conver- sion. Other program offices responsible for several less complex applica- tions may need similar contract help in coping with their aggregate conversion work load. Project officials acknowledge that FAA is planning to support the CORN contractor by using a combination of agency personnel and assistance from additional contractors. They have determined that the agency’s information servicesstaff will need to rely more heavily on their current contractors, and have asked FAA to provide $1.2 million a year during the 3-year conversion period for this purpose. However, the extent and cost of additional contractor help neededby FAA program offices remains undetermined. Project officials said that they are currently developing a preliminary conversion schedulethat will propose the sequencefor converting the applications and apportion the 86 employee-year estimate to individual applications. The officials said that prior to contract award, they will ask the staff designated to managethe conversion to comment on the plan. They said that they would not send the plan to the program offices, although the offices would be heavily involved in the conver- sion Sincethis plan is to be used in contract negotiations, project offi- cials maintain that it is procurement-sensitive and not for general release.The plan originally was to be made final in August 1989, but was still being reviewed internally when we completed our audit work in April 1990. The amount of time estimated to complete the conversion has repeatedly Estimated Conversion escalated.Project officials originally assumedthat the conversion would Time Frame Has take 18 months, which is the estimate found in the conversion studv and Doubled various documents leading to the project’s approval by the Department of Transportation in January 1989. They said that this l&month sched- ule was an optimum estimate basedon advice from several contractors. However, F&l program offices maintained that they could not meet this estimate becausethey lacked the requisite staff and other resources neededto satisfy CORN'S requirements while concurrently performing their regular duties. Consequently, project officials increasedtheir esti- mate to 24 months and then to 30 months. The officials said that this 30- Page 31 GAO/IMTEG9O-3f3 FM CORN Contract Should Not Be Awarded -p&W 4 Ib&jor Uncertninties Loom Over Conversion month estimate was judgmental, basedon discussionswith agency infor- mation systems staff. By November 1989, project officials had raised the estimated time frame to 3 years, citing budgeting concernsthat could further slow the pace of the conversion. Conversion delays will have cost consequences.Project documents repeatedly state that the CommonSystem will remain fully operational in parallel with CORN until all of the system’s applications are success- fully converted. Project officials estimate that it costs $36 million a year to operate the CommonSystem (excluding an additional $14 million a year in personnel, space,and supply coststhat would not be avoided by moving to CORN). The conversion time frame delay from 18 months to 3 years could result in additional parallel operations costsof $106 million. The officials noted that the cost of parallel operations might be reduced by phasing out portions of the CommonSystem before the conversion is completed, depending on the conversion plan agreedto by FAA and the contractor.6In any event, the parallel operations costs will be substan- tial and would have to be funded in addition to CORN contract costs. Oneof FAA’S goals is to create automated systems that provide the staff Conversion Will Not with better information for managing their programs. The CORNjustifi- Result in Better cation documents stress this goal by pointing out the agency’sdesire to Information integrate data into more useful information structures to improve man- agementpolicymaking and decisionmaking. Project officials claim that the diversity of hardware in the current system inhibits meeting this goal, while CORN would promote it. For all of its expense,however, the CORN conversion in itself will not move FAA toward this goal. Although the conversion would move the applications and data basesonto a new hardware environment, it would not improve the applications themselves or integrate the data bases becauseof the way the conversion is structured. Project officials insist that existing applications and data basesbe converted so that they are functionally equivalent to the original applications7 That is to say, the sameinputs to both the original and converted applications are to pro- duce identical outputs. Vendors were informed that enhancementof the %omeapplicationsprocessedon the CommonSystem,such asoffice automationapplications,are not part of the CORNconversion.They, too, will needto be movedoff of the CommonSystem-mainly to microcomputers-before the systemcan be closeddown. ?lkvo exceptionsare the ConsolidatedPersonnelManagementInformation Systemand the Aeronauti- cal Information System,which are to be put onto a modemcommercialdata basemanagement system. Page 32 GAO/IMTEC3O-33 FM COW Contract Should Not Be Awarded Mqjor Unoertaintlsr Loom Over Conversion application codefor purposes of increased functionality, presentation improvement, or performance improvement are neither encouragednor desired. The conversion will result in information being provided in the sameway it is now. The officials said that they adopted this approach in order to be able to determine whether the conversion was carried out successfully. Project officials have not determined whether all of the applications identified for conversion are worth converting. Following contract award, but prior to ordering the conversion of a particular application, the FAA office responsible for the application is supposedto determine whether there is a continued need for it. Software problems with the applications are to be corrected after they have been converted. The officials maintain that correcting any such problems is the responsibility of the agency staff in charge of the applications and is outside the scope of CORN, which was never intended to be a software development con- tract. This includes correcting errors in the applications, enhancing their functionality, or providing new functional requirements documenta- tion-all of which are tasks not required of the contractor under CORN. Thus, if the current software is inefficient and causing poor response times, the possibility exists that the converted software will still be inef- ficient, and CORNwill not achieve one of its primary benefits-improved responsetime. Page 33 GAO/MTEGQO4l3 FM CORN Contract Should Not Be Awnrded Chapter 6 Conclusionsand Recommendations A project of CORN'S magnitude, both in terms of cost and effect on opera- tions, requires careful preparation and well-supported technical analy- sis. CORN envisions a total commitment: the project is meant to completely replace the current system, not merely to augment it, by transferring FAA’S general-purposedata processingto one vendor for up to 10 years. The project’s viability dependsheavily on FAA’s ability to understand its current system, define problems and their causes,fore- cast future needsfor a decade,adequately evaluate vendor-proposed systems, and adequately plan for conversion to the new system. To date FAA has not demonstrated satisfactory performance in these areas, Con- sequently, it is not ready to proceed with awarding the CORN contract. CORN is, in essence,a request to the vendor community for a comprehen- sive technical solution to FAA’S long-term data-processingproblems and needs.While the concept of contracting for data-processingservices sup- port may be acceptable,FAA has not adequately defined, measured, and analyzed its current technical problems and needs.Most notably, FAA has not adequately documented responsetime problems, nor defined the causesof perceived responsetime problems with its current system. Its assertion that these problems exist and are causedby the lack of proces- sor capacity is not supported by available data. By not establishing a central capacity managementprogram for the Common System’s 12 facilities, FAA has done a poor job of tracking Common System utiliza- tion, monitoring the system’s performance, optimizing the use of its cur- rent resources,and identifying the causesof perceived responsetime and performance problems. BecauseFAA has not identified the causeof responsetime problems in the current system, it cannot be assumedthat CORN will eliminate these problems. Without improved responsetimes, CORN will not achieve one of its major objectives. Since past and current performance data are an indispensable element with which to build projections of future growth, FAA is not able to make a reasonableprojection of its anticipated growth in general-purposedata processingover 10 years. Its projected growth rate of 30 percent per year compoundedfor 10 years is basedon inadequate data and oversim- plified analysis of the agency’swork load. The methodology does not take into account any factors that would causethis rate of growth to change over the contract period, especially in the later years when the cumulative effect of the 30 percent growth leads to extremely large annual jumps in system capacity-eventually reaching a point where the system would be 1300 percent larger than the current one. Further, project officials know even less about the future processinggrowth of other elements of the Department of Transportation. They have simply Page 34 GAO/IMTEC9O-3El FM CORN Contract Should Not Be Awarded Chapter 5 Conclueiona find Recommend~~tiona assumeda similar 30-percent annual growth rate and increased CORN'S estimated contract value by about 40 percent to accommodateit. Although FAA and the Department are not obliged to order all of the ser- vice built into CORN, vendors must develop a proposed system capable of accommodatingthis high rate of growth. And they must develop fixed prices for levels of service that such a system would provide over 10 years. FAA, however, has not provided vendors with key information on the performance of the current system that they should have in order to properly develop a proposal to meet FAA'S projected needs.In the absenceof such information, vendors have to make assumptions about crucial characteristics of FAA's current system in order to model their proposed solutions to the agency’sneedsas specified in the request for proposals. FAA should have meaningful and accurate information with which to evaluate the most effective solution for meeting the agency’sprojected needs.one of FAA's critical evaluation tools for doing this-the agency- designedoperational capability demonstration-is seriously deficient. The sample work load used in the demonstration is extremely small and not representative of the agency’swork load. As a result, the demon- stration will not provide adequate data for accurately evaluating the vendors’ proposals. This problem, coupled with FAA'S failure to provide the vendors with adequate data with which to model a new system, could have cost ramifications throughout the life of the contract. A critical hurdle for implementing CORN is the conversion of Common System applications. Project officials have not adequately prepared for the conversion, as evidenced by the long-standing confusion over the inventory of applications to be converted to CORN. The officials have also not provided agency managementand the Congresswith reliable infor- mation on the funding, staff resources,and time frames of the conver- sion The conversion cost estimate of $74.6 million, used for project approval and budget requests, is unreliable and incomplete. The project officials’ assertion that the conversion will require 86 employee-yearsis broadly estimated on a aggregatebasis. Most critically, staff availability to support the conversion will not be determined until after contract award, when the agency’s staff will be confronted with the necessity of working through the conversion someway or another. As for time frames, CORN passedthrough the justification and approval processon the unsupported assumption that the conversion would involve a quick, 18-month effort. The time frame was lengthened to 3 Page35 GAO/lMTEGM FM CORN Contact Should Not Be Awarded chapt4Ts Ca~chusio~ and Recommendation years after the project was approved, when reactions from agency staff made clear the unreasonablenessof the project officials’ original esti- mate. The conversion inherently has a high risk for scheduledelays becauseof its extremely large size, as underscored by the fact that the original time frame has already doubled before any conversion work has begun. Delays in the conversion would further increasethe cost of run- ning the current system in parallel with CORN during the transition period. For the current 3-year time frame, these costs could run as high as $106 million. Finally, the conversion approach chosenby FAA will not result in improvements to the applications or their data bases.Problems cur- rently found in them would be transferred to the new system at consid- erable cost. Becausethe CORN acquisition has not been adequately justified or Recommendationsto planned and has major unresolved problems, we recommendthat the the Secretary of Secretary of Transportation direct that the CORN contract not be Transportation awarded.@ addition, we recommendthat the Secretary direct the Administrator, FAA, to ensure that future procurements of this type and magnitude are properly justified and planned prior to contract award. Specifically, the Administrator must ensure that the following is done priorto proceeding with a comprehensiveprocurement similar to CORN: . Existing system deficiencies need to be accurately and completely iden- tified and a solution needsto be designedthat addressesthese deficien- cies. Direct, periodic, systemwide monitoring, accomplishedthrough the implementation of a computer capacity and performance management program for FAA'S general-purposesystems, should be used to determine the presence,extent, and causesof performance problems-such as poor responsetimes. l Evaluation of vendors’ proposals should involve the use of a representa- tive work load sample. . Planning for conversion should include an accurate inventory of the existing applications, an assessmentof their continued need, and com- plete estimates of the cost and employee-yearsneededto support the conversion, including the extent to which qualified staff are available. fl Agency Comments way of doing businessand of making data-processingservices available to the agency’sprogram offices during the 1990s.Accordingly, they Page 86 GAO/lMTElCW3S FM C4MW Contract Should Not Be Awarded . ChaPtm 6 C4mclumlon13and bc4mmendarione _- ~~ ~-~~ maintain CORN should be assessedon its planned benefits-such as ena- bling FAA to better manageits information resources,provide quality services as needed,and perform its mandated missions. Although expressing general agreementwith the facts presented in our report, they disagree with the conclusions.They maintain that the report has not consideredCORN'S planned benefits and has judged the project on grounds that are not fundamental to its purpose and not material in the larger context. They maintain that CORN is sufficiently planned and justi- fied to warrant its award. We maintain that our findings deal with problems and deficiencies that are fundamental to the project’s scope and implementation and, consequently, support the conclusion that the contract should not be awarded. After receiving our draft report for comment, the FAA Administrator decided to order an independent review of CORNto determine if it should proceed as planned. The HouseCommittee on Appropriations has directed the Department and FAA not to award the CORN contract until (1) the Committee reviews the results of our report and FAA's written responseto it and (2) FAA and Department officials subsequently discuss the project with the Committee to resolve any outstanding concerns. Page 37 GAO- FM CORN Contract Should Not Be Awarded Appendix I Major Contributors to This Report 411 Joel Willemssen, Assistant Director Information John P. Finedore, Evaluator-in-Charge Management and Dr. Rona B. Stillman, Chief Scientist Technology Division, Frank Reilly, Senior Technical Adviser SusanMaciorowski, Presidential Exchange Executive Washington, D.C. David M. Bruno, Computer Scientist Leonard J. Latham, Technical Adviser Bruce Herbert, Senior Technical Specialist (aloasl) Page 38 GAO/IMTEG9033 FM CORN Contract Should Not Be Awarded United States General Accounting Office Washington, IX. 20648 Official Business Penalty for Private Use $300
FAA Procurement: Major Data-Processing Contract Should Not Be Awarded
Published by the Government Accountability Office on 1990-05-25.
Below is a raw (and likely hideous) rendition of the original report. (PDF)