* United States General Accounting Omcs J Lf /d6 j-c * . * GAO Testimony 1111111 II 141265 For Release on Forest Service Not Ready to Del ivery Acquire a Nat ionwide Geographic Expected at IO:00 a.m. EDT Information System Wednesday May 2, 1990 Statement of JayEtta Z. Hecker, Director, Resources, Community, and Economic Development Information Systems Information Management and Technology Division Before the Subcommittee on Interior and Related Agencies Committee on Appropriations House of Representatives C#4@+$3~ GAO/T-IMTEC-90-10 Mr. Chairman and Members of the Subcommittee: I appreciate this'opportunity to testify on our review of the Forest Service's plans to acquire a nationwide geographic information system (GIS). Our review, done at your request, was aimed at identifying any unresolved issues that could adversely affect the planned acquisition. As you know, the Forest Service plans to acquire a computer-based GIS to store, retrieve, analyze, and present spatially referenced information about the nearly 200 million acres of national forests and grasslands that it manages. This information associates land ownership data; vegetation types, such as tree species: soil types: water location: land elevation; and other characteristics with a specific location. The nationwide system will be composed of commercially available GIS software at 880 offices1 operating on new computers that will be linked together through an existing telecommunications network. The new GIS will automate diverse manual tasks and replace about 130 GISs procured in recent years on an ad-hoc basis by various Forest Service offices. The Service estimates the GIS will cost $1.2 billion over a la-year period, of which about $900 million is for management and overhead costs. lThis includes 653 ranger districts; 123 supervisors1 offices: 9 regional offices: 8 Washington, D.C., office sites: 9 research stations: 74 research labs: and 4 state or private forestry offices. Y 1 . Whils a GIS holds the promise of significantly enhancing the ability of the Forest Service td manage and preserve natural resources, the agency is not yet ready to procure a $1.2.billion nationwide system. The Service did not analyze a full range of alternatives, made assumptions that unnecessarily limited the alternatives it considered, and did not analyze how these alternatives- would affect its organization. Further, it did not estimate the dollar value of specific benefits it expected to achieve from the planned GIS, but instead used what we believe to be an invalid representation of future benefits. Finally, the Forest Service did not adequately define its information and system performance needs, thereby failing to comply with regulations governing functional requirements analyses. As a result, there is increased and unnecessary risk that the proposed GIS will'not result in an effective and cost-beneficial system. COST/BENEFIT M Department of Agriculture regulations require the preparation of a feasibility study prior to proceeding with systems development or acquisition. A feasibility study should identify and evaluate various alternatives for achieving an agency's objectives and il 2 include an analysis of organizational impact, such as reorganization and changes in staffing levels and staff skills that might result. The feasibility study together with a cost/benefit analysis should provide managers with enough information to determine the best alternative. We found that the Forest Service made two critical assumptions that limited the alternatives it considered in its September 1989 feasibility study. First, it assumed that all 880 offices would need GIS hardware and software and therefore did not consider selective placement of GIS technology. Second, it assumed that staffing levels wduld not change and therefore did not consider alternatives that might have involved such changes. The Senice's feasibility study identified seven alternatives within the limits of these assumptions. These alternatives included an option, for analytical and comparison purposes, of allowing offices to continue acquiring individual GISs on an ad-hoc basis through 1991. A second alternative was to allow offices to continue acquiring individual GISs on an ad-hoc basis for 10 years. The remaining five alternatives provide GISs'to all 880 offices, just varying the organizational level with primary responsibility for storing, maintaining, and managing the data. None of the alternatives addressed the organizational impact that Service officials expect to result from GIS implementation. We believe that by not considering alternatives that might alter thr organization or reduce, shift, or change the type or distribution of personnel, the Service essentially failed to address how to bqst harness the full capabilities of GIS technology. The Forest Service also did not make valid estimates of the economic benefits of.the alternatives and therefore does not have the economic information needed to select the best alternative. Department of Agriculture regulations require the Service to identify and quantify the benefits of alternatives considered. This includes estimating the dollar value of cost savings such as staff and budgetary reductions, or economic benefits from improved program delivery, such as improved utilization of resources, improved operational effectiveness, or increased accuracy of information. The Service's calculation of benefits from the proposed GIS contains two basic components, an estimate of staff productivity increases resulting from the implementation of the GIS and the costs of the way it currently processes spatially referenced information. On the basis of a work load analysis, the Service estimated that staff productivity related to processing and analyzing spatially referenced information will increase by 400 percent with full GIS implementation. We believe a 400-percent productivity increase, which is the equivalent of freeing about 4,900 employees if u 4 information production is held constant, strongly implies that the organization could change as a result of GIS. However, neither the alternatives contained in the feasibility study nor the cost/benefit analysis clearly states how the Service will use the increased productivity or how much it is worth. The Service then estimated total life cycle benefits of $4.6 billion for its preferred GIS alternative by multiplying the estimated 4000percent productivity increase by its estimate of current relevant personnel costs. We believe this benefit estimate is not valid because the Service failed to identify (1) how staffing or other'reductions would save $4.6 billion or (2) economic benefits of $4.6 billion from the increased quantity or quality of information products or improved program delivery. In fact, while staffing levels are assumed to remain the same, the GIS is expected to increase costs by $1.2 billion over the life of the system, and the worth of improved products or program delivery is not estimated. CE =S NOT ADEQUATELYADDWSED SANALysIs The functional requirements analysis for the GIS does not adequately address the full range of its information and system performance needs. A functional requirements analysis should provide support for a full description of the information processing requirements needed to accomplish the agency's mission. Agriculture regulations, which incorporate the use of Federal Information Processing Standards guidance, and the Federal ,Informatioq Resources Management Regulation rsquire'the articulation of data and performance specifications to meet user needs. Tho functional requirements analysis performed by the Service, however, provides only some of the data and.none of the performance specifications required. In particular, the functional rrguirements analysis does not provide specific information on the volume and frequency of inputs and,outputst sources, volume, and timeliness of data; data-accuracy requirements; data-validation- requirements: performance requirements, such as response times, updat8 processing times, consequences of system failures, and data transfar and transmission times: and data characteristics, including projected growth in storage requirements. For example, the agency has not established requirements for the maximum time to complete user-initiated operations or transmit -. data between offices, or the maximum time lost due to malfunctions. Since Agriculture 'regulations require that requests for proposals be consistent with the agency's needs as established in the roguirementlr analysis, we believe that by failing to adequately describe the required data and performance specifications, the Service is unnecessarily increasing its risks that vendors' offers Y 6 will not meet its functionality, processing, or performance needs. There is also an increased risk that the specifications will be insufficient to make appropriate tradeoffs and decisions among systgs' price and performance and that costs may increase because the performance needed at the sites may be different from the configurations offered by vendors. ed D&a Remts Not SDecified In addition to inadequate specification of requirements for existing sources of data, the Service has not determined a strategy as to how it will collect, store, and process satellite or remotely sensed image data. For example, the Service's functional requirements analysis describes GIS use in which satellite and aerial photographic products provide fundamental data to Service staff. However, the analysis does not specify how the GIS will obtain image products, the accuracy needed, how frequently image data will be collected, the storage capacities needed, or the processing requirements for image products. Sewice officials acknowledged that image data from satellites and aerial remote sensing systems have great potential for data- collection cost savings, verification of existing maps, and detection of changes in natural resources, and anticipate that supplemental procurements and funds will be needed in the future to add the capability to acquire, input, process, and analyze image u 7 data. The Service has been working on, but has not yet completed, a plan to provide a framework for integrating image data with the new GIS. We believe that selection of the GIS prior to the development of these specificat&ons unnecessarily increases the risk of procuring a system that does not provide all of the functionality needed, including image processing, at the lowest cost. To reduce the risk that the currently proposed GIS procurement strategy may not be cost-beneficial or satisfy-mission needs, we are recommending in a report to be issued shortly that, before procooding with the produrement, the Forest Service -- evaluate the feasibility, costs, benefits, and organizational impact of alternatives, including selective placement of GIS Capabilities and associated analytical resources, to achieve mission based objectives. The Service should demonstrate, in accordance with OWBguidance, that the benefits of the selected alternative exceed projected costs: and -- develop a comprehensive functional requirements analysis that includes sources, flow, timing, accuracy levels, validation, and performance requirements for processing a complete range of data that include planned as well a# existing data bources.' Mr. Chairman, a8 you know, the Forest Service differs strongly with both our conclusions and recommendations. I would now like to discuss the Forest Sentice reaction to our position. S ANDOUREV~ In commenting on our draft report, the Forest Service stated that it is well prepared to proceed with the acquisition and implementation of GIS capability. It said that its documentation meets all requirements and has been approved by the Department of Agriculture and the General Services Administration, and that additional studies would make the technology appear more advantageous and would-not yield information useful to potential offerors. The Senrice disagreed with our recommendation to evaluate the feasibility, costs, benefits, and organizational impact of alternative systems before proceeding with the procurement. It stated that a new study would cost between $1 and $2 million and * 9 . I re8ult in the same recommendation to management--that.G IS is a co&-effective technology. The Formt Service said it developed alternatives that focused on the noed to manage information and get work done on site at forest locations, and analyzed several alternatives. It said it will conduct two additional studies as it implements the planned G IS to determine whether it is reasonable to-place G IS technology at all site8. The Service will study employee productivity and the costs to implement full G IS capability, m inimal G IS capability, or no CIS capability at various sites. It said it will also study combinations of conditions, such as office size, work load, and levrl of public pressure, in order to determine a break-even point below which G IS technology may not be economically justified. The Service expects the results of these two studies to be especially useful in deciding whether the G IS is economically justified for those offices having 25 or fewer employees. (More than half of Forest Service offices have 25 or fewer employees.) The Service added that these studies will become supplements to the cost/benefit analysis, as well as policy documents to guide the implementation decisions of all offices. The Service's intention to conduct further studies during the implementation phase to determine the reasonableness of placing G IS capability at small offices indicates that the assumption that all 880 offices need G IS hardware and software needs to be reassessed. r) 10 Although this addresses one of our concerns about the feasibility study, these studies would amount to a reevaluation of the Service's preferred alternative after the agency issues a request ,for proposals and perhaps after awarding a contract on the basis of its preferred alternative. We believe that selectively placing GIS at the agency.'6 880 offices is a new alternative and should be analyzed as such before moving the project into the procurement stage. We also remain concerned that the Service did not adequately address the organizational impact of GIS in its feasibility study and cost/benefit analysis.. Service officials acknowledged that GIS technology will lead to significant changes in the organization resulting from transformations in how and where data are collected and analyzed, staffing level changes among offices, changes in personnel skill mix, and restructuring of jobs. They agreed that the feasibility study did not address the organizational changes they expect to result from the introduction of a GIS and new computers. ts on Need to Reestimat& The Senrice disagreed that its benefits estimate is invalid and stated that it followed traditional methods of analyzing benefits associated with computer technology. It said it will review cost data, especially management and overhead costs, to determine Ei 11 whether they have been correctly presented. The Senrice also said it will use a bontraotor to develop a methodology and periodically . as6es8 the benefits realized from the GIS implementation. This information will be used to adjust the Service's implementation 8trateg-y. After further review of the Service's methodology and consideration of its comments, we continue to believe that the Service did not use valid economic methods to estimate the quantifiable benefits of the GIS and that its estimate is invalid. The Service did not estimate the value of increased productivity or how it would use the increased productivity, the value of increased quality or quantity of information products, or the value of improved program delivery. Instead, it assumed that the current value of handling spatial data and producing informatign is equal to the personnel costs it now incurs for such work. The Service estimated the total quantifiable benefits of the GIS by in&easing the estimated current personnel costs fourfold because it expects the GIS to increase productivity 400 percent,. In effect, this methodology assumes that the value of GIS products would be equal to the current value of spatial data handling (current estimated personnel costs) increased fourfold. The increase assumes it would take four times as many staff to perform spatial data handling using current, largely manual, methods as might be achieved by a fully implemented GIS operated at the current staff year level of about 6,530. Since the quantifiable benefits of GIS were not properly estimated, the co8t/benefit basis for selecting the agency's preferred alternative wa8 not sound. The Service',8 delegation of procurement authority is al8o based on this analysis. ts on Need &r mive m The Service disagreed with our recommendation to develop a more comprehensive functional requirements analysis that includes data sources, -flow, timing, accuracy levels, validation, and performance requirements for processing a complete range of data that include planned as well as existing data sources. The Service believes that its information requirements have been adequately specified, stating that it has been developing and refining data, information, and functional requirements for 5 years. The Service said its functional requirements have been distributed in two requests for information with little adverse comment from potential vendors. It said that it has followed the direction of Departmental regulations, used Federal Information Processing Standards Publication 38 as a guideline, and considered the factors referenced in the Federal Information Resources Management Regulation. The Service also said that the planned performance evaluation of offered systems will be based on the u 13 c * . actual data of a Fore& Service office that will br realistically waled to a small- and large-sized field unit; The Service said that it is developing an attachment to the request for proposals that describes the Service@s current and anticipated usee,of remotely sensed data, and how the GIS is expected to use remotely sensed data in the future. The Service will also require each office to analyze its specific data. requirements, as part of the implementation process, to identify the type and configuration of equipment that is needed to meet its specific needs. While- we believe the Service's plan to describe its expected use of remotely sensed data is an improvement, we believe that its plan to further study its data and performance requirements during the implementation phase is inadequate. The functional requirements analysis provides only some of the data and none of the performance specifications required by Agriculture regulations which incorporate Federal Information Processing Standards Publication 38 (as guidance). The Service's plan to test the performance of offered systems is not a substitute for defining its data and performance specifications. The performance test is a one-time event and does not establish the continuing data and'performance requirements that must*be met as the system is implemented and used by hundreds of offices. We believe that by failing to adequately describe the required data and performance specifications, the P - 14 Service is unnecessarily increasing its risks of acquiring a system that will not meet its functionality, processing, and performance needs. If the Service doe8 not address these concerns and implement our recommendations, it will unnecessarily increase the risk that the proposed GIS will not result in an effective, efficient system-to support the Service's management and protection of the public lands and natural resources entrusted to it. We have found repeatedly ,that the lack of clear identification of needs, inadequate consideration of alternatives and costs, and inadequate definition of requirements are primary causes of problems with government civilian and military computer systems. We have no reason to believe that the GIS procurement would be immune from such problems. Wr. Chairman, this concludes my statement. I will be happy to answer any questions that you or other members of the Subcommittee may have at this time. 15
Forest Service Not Ready to Acquire a Nationwide Geographic Information System
Published by the Government Accountability Office on 1990-05-02.
Below is a raw (and likely hideous) rendition of the original report. (PDF)