oversight

Forest Service Not Ready to Acquire a Nationwide Geographic Information System

Published by the Government Accountability Office on 1990-05-02.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

*

                        United States General Accounting Omcs J Lf /d6          j-c
                                                                 * .
* GAO                   Testimony

                                                                  1111111
                                                                    II 141265


    For Release on       Forest    Service   Not Ready to
    Del ivery            Acquire    a Nat ionwide   Geographic
    Expected at
    IO:00 a.m. EDT       Information     System
    Wednesday
    May 2, 1990




                         Statement of
                         JayEtta Z. Hecker, Director,   Resources, Community,    and
                         Economic Development Information   Systems
                         Information  Management and Technology Division



                         Before the
                         Subcommittee on Interior    and
                         Related Agencies
                         Committee on Appropriations
                         House of Representatives




                                  C#4@+$3~
    GAO/T-IMTEC-90-10
Mr. Chairman and Members of the Subcommittee:

I appreciate           this'opportunity           to testify      on our review       of the
Forest       Service's       plans to acquire           a nationwide       geographic
information        system (GIS).             Our review,       done at your request,             was
aimed       at identifying          any unresolved          issues that      could adversely
affect       the planned acquisition.


As       you know, the Forest           Service       plans to acquire        a computer-based
GIS to store,           retrieve,       analyze,       and present       spatially    referenced
information        about the nearly            200 million        acres of national            forests
and grasslands           that     it manages.          This information         associates       land
ownership       data;      vegetation        types,     such as tree       species:     soil     types:
water location:            land elevation;            and other    characteristics        with      a
specific       location.


The nationwide           system will         be composed of commercially              available
GIS software           at 880 offices1         operating       on new computers that             will
be linked       together         through     an existing       telecommunications         network.
The new GIS will            automate diverse            manual tasks       and replace         about
130 GISs procured               in recent     years on an ad-hoc basis by various
Forest       Service     offices.          The Service       estimates     the GIS will         cost
$1.2 billion           over a la-year         period,       of which about $900 million                 is
for management and overhead costs.

lThis includes 653 ranger districts;    123 supervisors1
offices:  9 regional offices:   8 Washington, D.C., office sites:                                   9
research stations:   74 research labs: and 4 state or private
forestry  offices.
     Y
                                                      1 .
Whils a GIS holds the promise of significantly                                        enhancing the
ability          of    the        Forest     Service         td manage and preserve              natural
resources,             the agency is not yet ready to procure                             a $1.2.billion
nationwide             system.             The Service          did not analyze         a full      range of
alternatives,                 made assumptions                that    unnecessarily       limited        the
alternatives                 it    considered,             and did not analyze how these
alternatives-            would affect                its     organization.


Further,          it    did not estimate                    the dollar      value of specific            benefits
it        expected      to achieve            from the planned GIS, but instead                         used what
we believe             to be an invalid                    representation       of future        benefits.
Finally,          the Forest               Service         did not adequately         define      its
information             and system performance                       needs, thereby       failing        to comply
with        regulations             governing          functional        requirements         analyses.


As        a result,      there         is increased            and unnecessary         risk      that   the
proposed GIS will'not                        result         in an effective        and cost-beneficial
system.


                                   COST/BENEFIT M


Department of Agriculture                            regulations         require    the preparation            of a
feasibility             study prior            to proceeding             with   systems development            or
acquisition.                  A feasibility             study should identify             and evaluate
various         alternatives                for achieving            an agency's      objectives        and

     il
                                                               2
include       an analysis          of organizational                 impact,      such as
reorganization             and changes in staffing                    levels      and staff          skills        that
might result.              The feasibility                study together          with     a cost/benefit
analysis       should provide               managers with            enough information               to
determine        the best alternative.


We found that            the Forest          Service        made two critical              assumptions             that
limited       the alternatives               it        considered     in its      September          1989
feasibility           study.       First,         it     assumed that      all     880 offices             would
need GIS hardware and software                            and therefore          did not consider
selective       placement          of GIS technology.                  Second, it          assumed that
staffing       levels         wduld not change and therefore                       did not consider
alternatives           that      might have involved                 such changes.


The Senice's             feasibility          study identified             seven alternatives
within       the limits          of these assumptions.                  These alternatives
included       an option,          for analytical              and comparison purposes,                       of
allowing       offices          to continue            acquiring      individual          GISs on an ad-hoc
basis       through      1991.      A second alternative                 was to allow           offices            to
continue       acquiring          individual             GISs on an ad-hoc basis               for     10 years.
The remaining            five     alternatives             provide     GISs'to      all     880 offices,
just     varying      the organizational                   level     with primary          responsibility
for storing,          maintaining,            and managing the data.                      None of the
alternatives          addressed the organizational                       impact that           Service
officials       expect to result                  from GIS implementation.
We believe        that     by not considering                 alternatives       that     might alter
thr       organization       or reduce,           shift,      or change the type or
distribution         of personnel,              the Service        essentially          failed      to
address how to bqst harness the full                            capabilities        of GIS
technology.


The Forest        Service      also did not make valid                    estimates       of the
economic benefits             of.the          alternatives       and therefore           does not have
the economic information                  needed to select               the best alternative.
Department of Agriculture                      regulations       require       the Service          to
identify        and quantify          the benefits            of alternatives           considered.
This includes            estimating       the dollar           value of cost        savings         such as
staff       and budgetary       reductions,                or economic benefits           from improved
program delivery,             such as improved utilization                      of resources,
improved operational             effectiveness,                or increased       accuracy          of
information.             The Service's           calculation        of benefits          from the
proposed GIS contains              two basic               components,     an estimate           of staff
productivity         increases         resulting            from the implementation               of the GIS
and the costs            of the way it           currently       processes       spatially
referenced        information.


On the basis        of a work load analysis,                     the Service       estimated             that
staff      productivity        related          to processing        and analyzing             spatially
referenced        information          will      increase       by 400 percent          with      full     GIS
implementation.             We believe           a 400-percent        productivity             increase,
which is the equivalent                 of freeing            about 4,900 employees if

      u
                                                       4
information          production         is held constant,          strongly          implies    that       the
organization           could change as a result                 of GIS.       However, neither
the alternatives              contained        in the feasibility            study nor the
cost/benefit           analysis      clearly      states      how the Service           will    use the
increased          productivity         or how much it          is worth.


The Service          then estimated          total     life     cycle     benefits      of $4.6
billion       for    its    preferred       GIS alternative             by multiplying         the
estimated          4000percent      productivity           increase       by its     estimate        of
current      relevant         personnel      costs.        We believe       this     benefit     estimate
is not valid          because the Service              failed     to identify          (1) how
staffing       or other'reductions               would save $4.6 billion                or (2)
economic benefits               of $4.6 billion         from the increased              quantity          or
quality      of information           products        or improved program delivery.                        In
fact,      while     staffing      levels      are assumed to remain the same, the GIS
is expected to increase                 costs by $1.2 billion               over the life            of the
system, and the worth of improved products                               or program delivery               is
not estimated.


                                                      CE =S        NOT ADEQUATELYADDWSED
                                          SANALysIs


The functional             requirements        analysis       for the GIS does not
adequately          address the full           range of its        information          and system
performance          needs.       A functional        requirements          analysis      should
provide      support        for a full      description         of the information
processing       requirements          needed to accomplish                 the agency's           mission.
Agriculture        regulations,         which incorporate                the use of Federal
Information        Processing         Standards        guidance,         and the        Federal
,Informatioq      Resources Management Regulation                         rsquire'the
articulation        of    data and performance                 specifications            to meet user
needs.


Tho functional           requirements       analysis           performed      by the Service,
however, provides           only some of the             data      and.none of the performance
specifications           required.       In particular,                the functional
rrguirements        analysis         does not provide            specific       information          on the
volume and frequency            of inputs         and,outputst            sources,        volume,     and
timeliness       of data;      data-accuracy            requirements;           data-validation-
requirements:        performance         requirements,             such as response times,
updat8 processing           times,      consequences of system failures,                           and data
transfar       and transmission          times:        and data characteristics,
including       projected      growth in storage                requirements.             For     example,
the agency has not established                    requirements            for the maximum time to
complete user-initiated                operations        or transmit
                                                                -.    data between
offices,       or the maximum time lost                 due to malfunctions.


Since Agriculture           'regulations      require           that     requests        for proposals
be consistent        with    the agency's          needs as established                   in the
roguirementlr      analysis,         we believe         that     by failing         to adequately
describe       the required       data and performance                   specifications,            the
Service       is unnecessarily          increasing        its     risks      that    vendors'        offers

    Y
                                                   6
will       not meet its          functionality,           processing,        or performance         needs.
There is also an increased                    risk     that    the specifications           will      be
insufficient           to    make      appropriate        tradeoffs       and decisions      among
systgs'        price        and performance           and that        costs may increase           because
the performance              needed at the sites              may be different          from the
configurations              offered      by vendors.


                                              ed D&a Remts                         Not SDecified


In addition        to inadequate             specification            of requirements       for
existing       sources of data,              the Service         has not determined          a
strategy       as to how it will              collect,        store,      and process     satellite
or remotely        sensed image data.                 For example, the Service's
functional       requirements             analysis        describes      GIS    use in which
satellite       and aerial             photographic        products      provide      fundamental
data to Service              staff.       However, the analysis                does not specify        how
the GIS will           obtain         image products,         the accuracy         needed, how
frequently       image data will              be collected,            the storage      capacities
needed, or the processing                   requirements          for    image products.


Sewice       officials          acknowledged that             image data from satellites               and
aerial      remote sensing              systems have great             potential      for data-
collection       cost savings,             verification         of existing         maps, and
detection       of changes in natural                 resources,         and anticipate       that
supplemental        procurements            and funds will             be needed in the future             to
add the capability                to acquire,        input,     process,       and analyze         image

       u
                                                      7
data.       The Service       has been working             on, but has not yet completed,
a plan to provide            a framework for integrating                     image data with         the
new GIS.


We believe        that     selection     of the GIS prior              to the development            of
these specificat&ons             unnecessarily          increases        the risk         of procuring
a system that            does not provide       all     of the functionality                needed,
including      image processing,            at the lowest            cost.




To reduce the risk            that     the currently         proposed GIS procurement
strategy      may not be cost-beneficial                   or satisfy-mission             needs, we
are recommending in a report                 to be issued shortly                 that,     before
procooding        with     the produrement,           the Forest        Service


-- evaluate        the feasibility,         costs,         benefits,         and organizational
    impact of alternatives,               including         selective         placement      of GIS
    Capabilities          and associated       analytical            resources,       to achieve
    mission     based objectives.             The Service            should demonstrate,             in
    accordance with           OWBguidance,          that     the benefits          of the selected
    alternative          exceed projected        costs:        and


-- develop      a comprehensive           functional         requirements          analysis       that
    includes       sources,      flow,    timing,       accuracy        levels,     validation,           and
     performance          requirements         for processing           a complete range of data
     that      include     planned as well             a# existing          data bources.'


Mr. Chairman,            a8 you know, the Forest                  Service     differs     strongly        with
both our conclusions              and recommendations.                  I would now like            to
discuss        the Forest      Sentice       reaction           to our position.


                                    S ANDOUREV~


In commenting on our draft                   report,       the Forest         Service     stated     that
it   is well        prepared     to proceed with the acquisition                        and
implementation           of GIS capability.                It     said that      its    documentation
meets all          requirements      and has been approved by the Department of
Agriculture          and the General Services                   Administration,          and that
additional          studies    would make the technology                    appear more
advantageous          and would-not          yield       information        useful      to potential
offerors.




The Senrice          disagreed     with      our recommendation to evaluate                    the
feasibility,          costs,     benefits,       and organizational               impact of
alternative          systems before          proceeding          with the procurement.               It
stated      that     a new study would cost between $1 and $2 million                                and


     *                                               9
    .
I




        re8ult       in the same recommendation to management--that.G IS                                     is a
        co&-effective           technology.


        The Formt        Service said        it     developed alternatives                     that   focused on
        the noed to manage information                      and get work done on site                       at forest
        locations,       and analyzed several                alternatives.               It    said it       will
        conduct two additional              studies         as it    implements the planned G IS to
        determine whether it              is reasonable to-place                     G IS technology          at all
        site8.       The Service will             study employee productivity                      and the costs
        to implement full          G IS capability,              m inimal G IS capability,                   or no CIS
        capability      at various        sites.       It     said it         will     also study
        combinations         of conditions,          such as office              size,        work load,        and
        levrl     of public     pressure,          in order to determine a break-even point
        below which G IS technology may not be economically                                     justified.               The
        Service      expects the results             of these two studies                     to be especially
        useful     in deciding      whether the G IS is economically                            justified           for
        those offices         having 25 or fewer employees.                           (More than half               of
        Forest     Service     offices     have 25 or fewer              employees.)              The Service
        added that      these studies         will     become supplements to the
        cost/benefit         analysis,     as well as policy                  documents to guide the
        implementation         decisions      of all        offices.


        The Service's         intention     to conduct further                  studies         during the
        implementation         phase to determine the reasonableness of placing                                           G IS
        capability      at    small offices          indicates         that     the assumption that                      all
        880 offices      need G IS hardware and software                        needs to be reassessed.

            r)
                                                            10
 Although         this        addresses       one of our concerns about the feasibility
 study,        these studies              would amount to a reevaluation                            of the
Service's             preferred          alternative          after      the agency issues a request
,for        proposals         and perhaps         after       awarding a contract                   on the basis         of
its         preferred         alternative.             We believe            that     selectively           placing
GIS at the agency.'6 880 offices                              is a new alternative                   and should be
analyzed          as such before              moving the project                     into     the procurement
stage.


We also remain concerned that                               the Service             did not adequately
address the organizational                           impact of GIS in its                     feasibility        study
and cost/benefit                  analysis..           Service        officials             acknowledged that           GIS
technology             will      lead to significant                  changes in the organization
resulting             from transformations                   in how and where data are collected
and analyzed,                 staffing       level         changes among offices,                  changes in
personnel             skill     mix, and restructuring                       of jobs.          They agreed that
the feasibility                 study did not address the organizational                                     changes
they expect to result                      from the introduction                     of a GIS and new
computers.


                                         ts on Need to Reestimat&


The Senrice              disagreed         that      its    benefits         estimate          is invalid       and
stated         that      it    followed      traditional          methods of analyzing                      benefits
associated             with     computer technology.                    It     said it will           review     cost
data,         especially         management and overhead costs,                              to determine

       Ei
                                                             11
whether              they have been correctly                     presented.          The Senrice         also said
it      will         use a bontraotor               to develop a methodology                   and periodically               .
as6es8 the benefits                       realized         from    the GIS implementation.                   This
information              will      be     used to adjust            the Service's             implementation
8trateg-y.


After          further      review of the Service's                     methodology            and consideration
of      its         comments, we continue                  to believe       that     the Service          did not
use valid              economic methods to estimate                        the quantifiable             benefits         of
the GIS and that                   its         estimate      is invalid.           The Service         did not
estimate              the value of increased                   productivity          or how it would use
the increased               productivity,                 the value of increased                quality      or
quantity              of information              products,        or the value of improved program
delivery.               Instead,          it     assumed that        the current         value of handling
spatial             data and producing                informatign          is equal to the personnel
costs          it    now incurs           for such work.             The Service         estimated         the total
quantifiable              benefits              of the GIS by in&easing                 the estimated
current             personnel       costs         fourfold        because it        expects the GIS to
increase             productivity               400 percent,.        In effect,         this     methodology
assumes that              the value of GIS products                        would be equal to the
current             value of spatial               data handling           (current      estimated
personnel             costs)      increased           fourfold.         The increase           assumes it would
take four times as many staff                              to perform         spatial     data handling
using current,                  largely          manual, methods as might be achieved                         by a
fully          implemented GIS operated                      at the current           staff     year      level     of
about 6,530.
Since the quantifiable                benefits           of GIS were not properly                          estimated,
the co8t/benefit           basis      for selecting                  the agency's              preferred
alternative        wa8 not sound.                The Service',8               delegation            of procurement
authority       is al8o based on this                    analysis.


                                   ts on Need &r                    mive                       m



The Service        disagreed         with our recommendation to develop a more
comprehensive           functional        requirements                analysis          that       includes      data
sources,      -flow,     timing,      accuracy           levels,           validation,             and performance
requirements           for processing            a complete range of data that                              include
planned as well           as existing            data sources.


The Service        believes        that    its         information           requirements             have been
adequately       specified,          stating       that        it     has been developing                  and
refining       data,     information,            and functional                requirements             for 5
years.        The Service      said its           functional               requirements             have been
distributed        in two requests               for     information            with      little        adverse
comment from potential                vendors.            It        said that      it     has followed           the
direction       of Departmental           regulations,                 used Federal                Information
Processing       Standards         Publication            38 as a guideline,                       and considered
the factors       referenced          in the Federal                  Information          Resources
Management Regulation.                 The Service              also said that                 the planned
performance       evaluation          of offered             systems will           be based on the
    u
                                                        13
        c
*   .




            actual      data    of a Fore&             Service     office        that     will     br realistically
            waled       to a small-           and large-sized           field      unit;


            The Service         said that         it     is developing           an attachment             to the
            request      for proposals            that     describes        the Service@s current                    and
            anticipated         usee,of        remotely      sensed      data,          and how the GIS is
            expected      to use remotely                sensed data in the future.                        The Service
            will      also require           each office         to analyze        its     specific         data.
            requirements,          as part        of the implementation                  process,          to    identify
            the type and configuration                     of equipment that               is needed to meet its
            specific      needs.


            While- we believe               the Service's        plan to describe                 its     expected use
            of remotely         sensed data is an improvement,                          we believe          that     its     plan
            to further         study its        data and performance                requirements                during      the
            implementation          phase is inadequate.                    The functional                requirements
            analysis      provides           only some of the          data      and none of the performance
            specifications          required           by Agriculture           regulations              which
            incorporate         Federal        Information         Processing           Standards          Publication            38
            (as guidance).              The Service's         plan to test              the performance              of
            offered      systems is not a substitute                    for defining               its     data and
            performance        specifications.               The performance               test         is a one-time
            event and does not establish                     the continuing              data and'performance
            requirements         that       must*be met as the system is implemented                                and used
            by hundreds of offices.                    We believe       that       by failing             to adequately
            describe      the required           data and performance                   specifications,             the
               P
                                        -                         14
Service          is unnecessarily               increasing      its     risks      of acquiring            a system
that      will      not meet its            functionality,            processing,        and performance
needs.




If      the Service        doe8 not address these concerns and implement our
recommendations,                 it will        unnecessarily          increase       the      risk    that    the
proposed GIS will                 not result        in an effective,               efficient          system-to
support          the Service's            management and protection                  of the public             lands
and natural             resources         entrusted     to it.          We have found repeatedly
,that     the lack of clear                 identification        of needs, inadequate
consideration             of alternatives             and costs,         and inadequate               definition
of      requirements         are primary           causes of problems with                   government
civilian          and military            computer systems.              We have no reason to
believe          that   the GIS procurement              would be immune from such
problems.


Wr. Chairman,             this     concludes        my statement.               I will      be happy to
answer any questions                     that   you or other members of the Subcommittee
may have at this                 time.




                                                       15