oversight

Computer Acquisition: Navy's Aviation Logistics System Not Ready for Deployment

Published by the Government Accountability Office on 1990-02-09.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

I'thI)rllilry   I!)!)()
                                     COMPUTER
                                     ACQUISITION
                                     Navy’s Aviation
                                     Logistics System Not
                                     Ready for Deployment




_-______ -~~                  ----    -   -
(;AO/IM’I‘l1:(:-!)O-1     1
Information Management and
Technology Division

B-237899

February 9,199O

The Honorable John P. Murtha
Chairman, Subcommittee on Defense
Committee on Appropriations
House of Representatives

Dear Mr. Chairman:

This report is in response to your predecessor’s October 19,1988, request to review the
Naval Aviation Logistics Command Management Information System (NALWMIS). As agreed
with your office, we responded orally on September 19,1989, to questions concerning the
accuracy and disclosure of NALCOMIS cost estimates and the appropriateness of fiscal year
1990/1991 budget requests. This report addresses the question of whether the risks and
benefits of NALCOMIS warrant its scheduled deployment. We conclude that the second phase of
NAILTOME is not ready for deployment because the Navy has not adequately performed some
key software development steps. As a result, chapter 4 of the report contains
recommendations to the Secretary of the Defense outlining certain conditions that should be
met before deployment approval is granted.

As arranged with your office, we are providing copies of this report to the Secretary of
Defense and the Secretary of the Navy, select congressional committees and members, and to
other interested parties upon request.

Sincerely yours,




Ralph V. Carlone
Assistant Comptroller General
EjxecutiveSummary


                   Naval aviation maintenance and supply represents a formidable man-
&pose              agement challenge. The Navy and Marine Corps are responsible for
                   maintaining roughly 6000 aircraft worth $76 billion. These aircraft
                   average about 1.6 million maintenance actions each year, and the
                   replacement parts for these aircraft are valued at over $18 billion.

                   To improve management of these operations, the Navy is acquiring a
                   large management information system. This system, called Navy Avia-
                   tion Logistics Command Management Information System (NALCOMIS),
                   has an estimated life cycle cost of $1.34 billion.

                   Because of concern about the system’s cost, development, and testing,
                   the former Chairman, Subcommittee on Defense, House Committee on
                   Appropriations, asked GAO to determine if NALCOMIS' risks and benefits
                   warrant its scheduled deployment.


                   The Navy plans to use NALCOMIS at 88 intermediate maintenance activi-
Background         ties (e.g., Naval air stations, Marine air groups, aircraft carriers, etc.)
                   and about 400 organizational maintenance activities (i.e., squadrons). It
                   is currently in the second of a three-phased approach to acquiring the
  /                system. Because Phase I quickly provided some limited functions, it was
                   adopted as an interim system with the understanding that phase II
                   would replace it. Phase I is currently operating at 29 intermediate-level
                   sites. Phase II offers considerably more capability, and is operating at
                   nine intermediate-level maintenance sites. Phase III includes phase II
                   functions and some additional functions, and will be deployed to the
                   organizational-level sites.


                   To minimize the cost and performance risks associated with building
Results in Brief   information systems like NALCOMIS, Navy and Defense guidance requires
                   the performance of certain software development steps. While omitting
                   or only partially performing one or more of these steps does not categor-
                   ically condemn systems to failure, such actions increase the risk that
                   they will not adequately perform and will cost more to develop and
                   maintain than necessary.

                   Phase II NALCOMIS is not ready for deployment because the Navy has not
                   adequately performed some key software development steps. Specifi-
                   cally, the Navy confined phase II operational testing to sites that are not
                   representative of the heavy processing workloads and stringent on-line
                   system availability requirements of some sites scheduled to receive


                   Page 2                               GAO/IMTEGSO-11   Navy NALCOMIS   Acquisition
    ,                  Executive Summary




                       phase II. Additionally, NALCOMIS security features were not based on the
                       required assessment of system risks and cost-effective countermeasures,
                       and its security certification is questionable because security test results
                       do not indicate that important system controls were tested. Further, the
                       Navy either did not develop or did not maintain certain documentation
                       (e.g., program maintenance manual) necessary to efficiently maintain
                       the system. The Navy also has not evaluated whether the operational
                       readiness benefits expected from phase II are accruing. To the Navy’s
                       credit, it has taken steps to address problems GAO raised with system
                       regression testing and portability to new hardware, and it, has demon-
                       strated that phase II can operate at the small to medium-size sites. Fur-
                       ther, it appears that NALCOMIS users are pleased with the system.



Principal Findings

Further Testing and    Navy instructions require that information systems be (1) thoroughly
Documentation Needed   tested, (2) certified as secure, and (3) supported by adequate documen-
                       tation. However, GAO found that the Navy has not fully satisfied each of
                       these requirements. For example, the Navy has not field tested phase II
                       at representative sites. The larger of the two phase II test sites, Naval
                       Air Station-Norfolk, processes only about one-third the number of items
                       that the largest site scheduled to receive the system does. Further, both
                       test sites’ requirement for on-line system availability is less stringent
                       than the requirement aboard aircraft carriers.

                       The Navy has also not performed stress testing to determine how sys-
                       tem resources (e.g., processors, channels, primary storage) will perform
                       under more demanding workload and system availability conditions
                       than those encountered at the two test sites. Thus, the Navy does not
                       know if the system will perform effectively in the more taxing environ-
                       ments found at large air stations and aboard aircraft carriers. After GAO
                       alerted Navy officials to this concern, the officials stated that they are
                       acquiring the tools necessary to predict and monitor system require-
                       ments for a given site.

                       Additionally, the Navy has not followed Navy and Defense guidance
                       requiring that risk assessments be developed and used to determine the
                       type and extent of security measures that should be built into a system
                       during development. As a result, the Navy does not know if it has
                       designed the most appropriate security features into NALCOMIS. Further,


                       Page 3                               GAO/IMTEC90-11   Navy NALCOMIS   Acquisition
                            Executive   Summary




                            the Navy’s testing of the security features it did build into NALCOMIS
                            omitted requisite tests, such as controls to prevent users from executing
                            operating system commands/privileged instructions. Such omissions
                            raise serious doubt about the basis for its recent certification of phase I
                            and II software as secure.

                            Further, the Navy has yet to develop a NALCOMIS program maintenance
                            manual, as required by Navy instructions, and it has failed to either
                            develop or maintain certain testing documentation, such as the specific
                            results of the tests performed. According to NALCOMIS officials, the main-
                            tenance manual was not contractually required because they planned to
                            use system specifications and source code in lieu of the manual, and
                            they have maintained the system for over a year with this documenta-
                            tion. Without this manual and the testing documents, system mainte-
                            nance will be more difficult and costly than necessary.


User Reaction to Phase II   Navy users at the phase II sites GAO visited stated that they are pleased
Is Positive                 with the system, and officials with the Navy’s Atlantic and Pacific Com-
                            mands echoed these sentiments. Additionally, NALCOMIS oversight offi-
                            cials told us that their impressions of phase II user reactions are
                            positive. GAO'S observation of phase II operations at these two sites
                            showed user familiarity and comfort with the system, and disclosed no
                            apparent system performance problems.


Demonstration of Phase II   The Navy’s principal justification for NALCOMIS is to improve aviation
Bqnefits Needed             maintenance and supply readiness. However, the Navy has yet to
                            demonstrate through analysis of operational performance at a phase II
                            site that expected readiness improvements are actually being realized.
                            Specifically, Defense and Navy instructions require evaluations of ongo-
                            ing programs to ensure that expected benefits are actually being
                            attained in the most cost-effective manner. Although the Navy has oper-
                            ated phase II at four sites for over 1 year and one site for over 2 years,
                            the Navy’s analyses of phase II benefits identify only administrative/
                            clerical benefits (e.g., labor reductions associated with using and main-
                            taining manual records and documentation). They do not link these ben-
                            efits to the operational readiness improvements initially used to justify
                            NALCOMIS or to personnel reductions. GAO performed a limited examina-
                            tion of some indicators of operational performance at four phase II sites
                            to see if readiness benefits were actually accruing. Based on these
                            indicators, GAO found that phase II was not consistently producing



                            Page 4                               GAO/IMTEG90-11   Navy NALCOMIS   Acquisition
                    J%xwut.Ive Summary




                    improvements in aircraft readiness. While GAO'S results do not demon-
                    strate conclusively whether or not expected phase II benefits are accru-
                    ing, they do reinforce the need for the type of in-depth program
                    evaluation required by Navy and Defense instructions.


                    GAO recommends that the Secretary of Defense direct the Secretary of
Rec(mmendations     the Navy to defer any further phase II deployment until the NALCOMIS
                    program office (1) fully stress tests the system, (2) successfully field
                    tests the system aboard a carrier and at a large Naval air station, (3)
                    clearly demonstrates that expected system benefits are being achieved,
                    (4) fully assessessystem security requirements and completely tests
                    security features, and (6) develops a system maintenance manual. GAO
                    further recommends that the Secretary of Defense direct the Major
                    Automated Information System Review Committee to withhold its phase
                    II deployment approval until the committee reviews the Navy’s efforts
                    to expeditiously satisfy the above conditions.


                    As requested by the Chairman’s office, GAO did not obtain official
AgerIcy conunents   agency comments on a draft of the report. However, GAO discussed the
                    report’s findings with Navy and Office of the Secretary of Defense offi-
                    cials and has incorporated their comments where appropriate.




                    Page 5                              GAO/IMTEG90-11   Navy NALCOM’IS Acquisition
contents


E$ecutive Summary                                                                                           2

Chapter 1                                                                                                   8
Inboduction             Aircraft Maintenance and Supply: A Brief Description
                        What Is NALCOMIS?
                                                                                                            8
                                                                                                            9

C ’apter 2
F $ rther Testing and   Testing Has Not Demonstrated That Phase II Will
                             Effectively Perform at All Sites
Mbnagement Actions      Lack of Phase II Maintenance Manual and Test                                    19
Needed to Reduce             Documentation Will Make System Maintenance
Phase II Deployment          Unnecessarily Difficult and Costly
                        Navy Currently Taking Steps to Address Transition to                            20
Risks                        New Hardware
                        User Reaction to Phase II Is Positive                                           21

Chapter 3                                                                                               23
Navy Needs to           Navy Evaluations of Phase II Benefits Are Limited                               23
                        Indicators of Phase II Effect on Aircraft Readiness Are                         26
Demonstrate Phase II         Mixed
Benefits
Chapter 4                                                                                               27
C+nclusions and         Conclusions
                        Recommendations
                                                                                                        27
                                                                                                        28
Rbcommendations
Abpendixes              Appendix I: Objectives, Scope and Methodology                                   30
                        Appendix II: Major Contributors to This Report                                  33


                        Abbreviations

                        GAO        General Accounting Office
                        IMTEC      Information Management and Technology Division
                        MAG        Marine Air Group
                        MAISRC     Major Automated Information System Review Committee
                        NALCOMIS   Naval Aviation Logistics Command Management Information
                                       System
                        NAS        Naval Air Station


                        Page 6                              GAO/IMTEC-90-11   Navy NALCOMIS   Acquisition
*




    Page 7   GAO/lMTEC90-11   Navy NALCOMIS   Acquisition
Chkpter 1

titroduction


                       The Navy and Marine Corps’ ability to fulfill their missions depends in
                       part on how effectively and efficiently they repair, maintain, and sup-
                       ply parts for their aircraft. These services are responsible for the pre-
                       ventive and remedial maintenance on about 6000 aircraft, with an
                       estimated value of $75 billion, These aircraft average about 1.6 million
                       repair and maintenance actions each year. The Navy and Marine Corps
                       are also responsible for managing an aircraft parts inventory valued at
                       over $18 billion.

                       The Navy is currently in the second of a three-phase project designed to
                       improve management of aircraft maintenance and supply support oper-
                       ations. The system, called Naval Aviation Logistics Command Manage-
                       ment Information System (NALCOMIS), will automate the record keeping
                       and management reporting associated with these operations. The esti-
                       mated life cycle cost of NALCOMIS is $1.34 billion.

                       On October 19, 1988, the former Chairman, Subcommittee on Defense,
                       House Committee on Appropriations, expressed interest in NALCXIMIS'
                       costs, budgetary disclosure, benefits, development and testing, and
                       hardware procurement schedules, and asked us to review the program.
                       On the basis of this request and subsequent discussions with the Chair-
                       man’s office, we agreed to determine whether (1) the risks and benefits
                       associated with NALCOMIS phase II warrant its scheduled deployment, (2)
                       the NALCOMIS life cycle cost estimate is accurate and complete, (3) the
                       Navy has fully disclosed the NALCOMIS life cycle cost estimate to Con-
                       gress, and (4) the fiscal year 1990/1991 NALCOMIS operations and mainte-
                       nance and procurement budget requests are appropriate in light of the
                       number of installations either operating or scheduled to operate
                       NALCOMIS during this period.

                       This report addresses the first objective (i.e., do the risks and benefits of
                       phase II warrant its scheduled deployment). Our results on the other
                       three objectives were provided to the Chairman’s office during an oral
                       briefing on September 19, 1989. A detailed explanation of our scope and
                       methodology for the first objective is in appendix I.


                       Aircraft maintenance and repair in the Navy and Marine Corps is per-
Aircraft Maintenance
             .-.   A   formed at three levels-organizational,   intermediate, and depot--
and Supply: A BrIeI    depending on the complexity of the maintenance being performed. In
Descriptiori           general, the organizational level performs less complex maintenance
                       tasks while the depot level performs the most demanding repairs. The



                       Page 8                                GAO/IMTECW-11   Navy NALCOMIS   Acquisition
                    Chapter 1
                    Introduction




                    supplies and parts required for each maintenance activity are managed
                    by the supply centers that support each activity.

                    The organizational maintenance activities (i.e., organizational level) are
                    the squadrons that possess the aircraft. These activities perform routine
                    servicing, inspections, and replacement of parts. They also fix aircraft
                    components that remain on the aircraft while repairs are being made.

                    The intermediate maintenance activities (i.e., intermediate level) include
                    Naval air stations (NAS), Marine air groups (MAG), and aircraft carriers.
                    These activities perform maintenance that is beyond the squadron’s
                    capabilities, such as equipment tuning or adjusting and repairs requiring
                    special equipment or training. These activities also repair or replace
                    damaged aircraft components that must be removed from the aircraft to
                    be worked on, and they manufacture certain parts that are not
                    available.

                    The Naval aviation depots (i.e., depot level) perform the most complex
                    repairs, such as major overhauls and rework that are beyond the capa-
                    bilities of the organizational and intermediate levels.

                    In brief, the aviation maintenance and supply support process begins
                    when a maintenance activity requests a replacement part from supply
                    and is completed when either a new part or the repaired part is deliv-
                    ered to the requesting maintenance activity. This process includes track-
                    ing the defective part from the time it is turned in for repair until it is
                    returned to the supply shelf for immediate or subsequent issue.


                    NALCOMIS is an automated management information       system for Navy and
What Is NALCOMIS?   Marine Corps organizational and intermediate maintenance activities
                    and supply support centers. It will automate a variety of record keeping
                    and reporting requirements for these activities. For example, NALCOMIS
                    will maintain a repair history for aircraft, track inventory levels at sup-
                    ply centers, track the status of parts under repair, and automate avia-
                    tion repair and maintenance manuals. Prior to NALCOMIS, these functions
                    were mostly performed manually.

                    The ultimate objective of NALCOMIS is to improve aircraft mission readi-
                    ness through (1) improved availability of real-time information to sup-
                    port day-to-day aircraft maintenance and supply management decisions,
                    (2) reduced work force levels needed to manually collect and validate



                    Page 9                               GAO/IMTJSG90-11   Navy NALCOMIS   Acquisition
                      Chapter 1
                      Introduction




                      data during the maintenance and supply process, and (3) improved qual-
                      ity of information on parts repair and replacement actions reported to
                      other Navy organizations for use in engineering, budgeting, parts provi-
                      sioning, and other related decisions.

                      The Navy is following a three-phase approach to developing and
                      deploying NALCOMIS. According to the Navy, the phased approach allows
                      them to take advantage of some automated benefits while the system is
                      still being developed. Phase I provides an automated capability at the
                      intermediate maintenance activities for managing the extensive record
                      keeping associated with the repair of aircraft parts.

                      Because phase I’s functions are limited, it was adopted as an interim
                      system with the understanding that phase II would replace it. Phase II
                      will fully automate aviation maintenance functions at 88 intermediate
                      maintenance activities and supply support centers.’ Phase III includes
                      certain phase II capabilities and a few new system functions, and will be
                      deployed to about 400 additional sites (i.e., the organizational mainte-
                      nance activities).


PrOgram History and   Since approval of the NALCOMIS concept in February 1977 by the Assis-
Sta@ls                tant Secretary of the Navy (Financial Management), NALCOMIS has expe-
                      rienced a lengthy development process caused by (1) failures in
                      software design and (2) delays in awarding a Navy-wide hardware con-
                      tract? , which the NALCOMIS program was required to use. As stated by
                      one program official, NALCOMIS did not get on track until after the Assis-
                      tant Secretary of the Navy (Financial Management) June 1984 decision
                      to adopt a three-phase approach to deploy an existing system, Status
                      and Inventory Data Management System II,3 on an interim basis until the
                      NALCOMIS software was developed and tested. Following this decision,
                      the Navy converted this existing system’s software to operate on the
                      NALCOMIS hardware, and in November 1984 the Assistant Secretary of
                      the Navy (Financial Management) approved its deployment as phase I


                      ‘These 88 activities include all but one of the 33 activities that either has operated or is now operat-
                      ing phase I. In addition to the 88 activities, phase II will be implemented at 1.5more sites used for
                      training and contingency operations. The one phase I activity not scheduled for phase II is an aircraft
                      carrier to be decommissioned.
                      2The contract is the Shipboard Non-Tactical Automated Data Processing Program, a Navy-wide con-
                      tract intended to provide compatibility among the Navy’s non-tactical information systems.
                      “This system was developed by the Naval Air Forces Atlantic Fleet to automate aircraft repair record
                      keeping on aircraft carriers during deployment.



                      Page 10                                             GAO/IMTEGSO-11       Navy N4LCOMIS      Acquisition
                             Chapter 1
                             Intraduction




                             NALCCMIS.  The Navy then competitively awarded a contract for phase II
                             and III software development. Between 1985 and 1988, the Navy
                             deployed phase I to 33 intermediate maintenance activities.

                             In September 1986, the Navy began operating phase II at one MAG on a
                             prototype basis. After 6 months of operation, including a lo-day system
                             performance test, the program office requested full fleet deployment
                             approval. In June 1987, the Assistant Secretary of the Navy (Financial
                             Management) approved phase II’s deployment to four additional shore
                             sites and one aircraft carrier. However, full fleet deployment approval
                             was withheld pending the results of testing at an NAS. Also in June 1987,
                             the Navy halted phase III after 90 percent of the software was devel-
                             oped. According to a program official, funds allocated to phase III com-
                             pletion were shifted to phase II to correct functional deficiencies. In
                             December 1988, the Navy completed testing at the air station, and in
                             July 1989, was granted conditional full fleet deployment (see next sec-
                             tion for discussion of conditions).

                             Currently, phase I is operating at 29 intermediate activities, and phase II
                             is operating at nine intermediate activities (seven MAGS, one NAS, and one
                             non-deployed aircraft carriep ). During fiscal years 1990 and 1991, the
                             Navy plans to deploy phase II to 17 more intermediate sites, and it plans
                             t.o have deployed phase II to all 88 intermediate sites by fiscal year
                             1995. Additionally, the Navy plans to begin deploying phase III to its
                             organizational sites in fiscal year 1992 and to complete phase III deploy-
                             ment in fiscal year 1997. Through fiscal year 1989, the Navy has
                             expended $261 million on the three phases.


Office of the Secretary of   The Office of the Secretary of Defense exercises its oversight responsi-
                             bilities for major-6 automated information systems through its Major
Defense Program              Automated Information Systems Review Committee (MAISRC).~ How-
Ovei-sight                   ever, the MAISRC did not regularly review NALCCMIS until July 1986. The
                             system was exempted from review because it had progressed past its


                             4According to the program manager, this carrier has yet to be deployed and does not have a full air
                             wing aboard. As a result, phase II is not yet operating on this carrier under normal workload
                             conditions.

                             “Defense Directive 7920.1, Life Cycle Management of Automated Information Systems, defines major
                             systems as those with e&mated project costs over $100 million, those with estimated costs in any one
                             year over $26 million, or those designated as special interest.

                             sMAISRC is a senior-level Defense review board responsible for guiding and directing msjor resource
                             investment3 in general-purpose computer systems.




                             Page 11                                           GAO/IMTEGSO-11      Navy NALCOMIS      Acquisition
    Chapter I
    Introduction




    initial planning stages when the MAISRC was established in the late
    1970s.

    During the fiscal year 1986 appropriations process, the Committee on
    Appropriations expressed concern with NALCOMIS and directed the
    MAISRC to review the system. The review was held in July 1986, and it
    concluded that the Navy had proper management controls in place and
    that development was on schedule. As a result, NALWMIS oversight
    authority was returned to the Navy. In February 1988, the Office of the
    Secretary of Defense found that NALCOMIS was experiencing schedule
    delays and conducted an on-site review in March 1988. As a result of the
    review’s findings, the Navy’s oversight authority was reclaimed by the
    MAISRC in June 1988.

    In July 1989, the MAISRC reviewed NALCOMIS and conditionally approved
    phase II full fleet deployment. The conditions included the following
    actions and reports, which the Navy was directed to complete and sub-
    mit to the MAISRC.

. Full configuration management and capacity management programs
  must be operational and their procedures followed.
l NALCOMIS testing program should proceed and full documentation of test
  results accomplished. This program should include capacity testing, per-
  formance testing, and available on-board carrier test results.
l A report on the alternatives for accelerating phases II and III implemen-
  tation, including options for completing deployment by fiscal year 1993
  and identifying the resource and benefit impacts of each alternative.
. A report clarifying the strategies for hardware acquisition after the cur-
  rent contract expires in 1992. This report should include transition and
  open competition considerations and these considerations should be
  coordinated with the implementation schedule.

    In October 1989, the Navy submitted its response to the MAISRC. The
    MAISRC is reviewing the response and has yet to rule on whether it satis-
    fies the conditions.




    Page 12                             GAO/IMTEC90-11   Navy NALCOMIS   Acquisition
Chapter 2

F’ujrtherTesting and ManagementActions
N&ded to ReducePhaseII DeploymentRisks

                         The Navy’s approach to managing the development of NALCOMIS has not
                         met Navy and Defense system development requirements and has not
                         included the prudent management actions needed to minimize system
                         cost and performance risks. We found several deficiencies with phase II
                         testing that cast serious doubt on whether the system is ready to be
                         deployed. For example, the Navy tested the system at a MAG and an NAS
                         that are not representative of the larger air stations and the aircraft
                         carriers scheduled to receive it. Also, the Navy has not fully tested sys-
                         tem security features, and it did not develop and use a risk assessment
                         to determine if the security features it built into NALCOMIS were the most
                         cost effective and appropriate. Further, the Navy has operated both
                         phase I and II without obtaining the required security accreditations for
                         the sites where phase II has been installed. In addition, the Navy has not
                         developed and maintained certain system documentation (e.g., program
                         maintenance manual) that is necessary for the cost effective mainte-
                         nance of the system.

                         The Navy has, however, taken positive steps to address concerns we
                         raised about phase II stress testing and portability to different hardware
                         environments, and has demonstrated that the system can perform at a
                         MAG and a medium size NAS. Additionally, users at these sites where
                         phase II is performing stated that they are generally pleased with the
                         system. In spite of these positive steps and reactions, more actions are
                         needed to justify any further deployment.


                         According to Navy Instruction 5232.1, Quality Assurance Program for
Testing Has Not          Information System Projects, testing validates that an information sys-
Dekonstrated That        tem satisfies the functional and technical requirements and that the sys-
Phase II Will            tem can be used effectively. Accordingly, the instruction requires that
                         information systems be thoroughly reviewed, tested, and evaluated
Effectively Perform at   before being deployed and that all deployed systems be accredited in
All ‘Sites               accordance with Navy security requirements. The testing process is pro-
                         gressive, with each subsequent series of tests building on prior tests. It
                         is designed to determine if the system performs as required under nor-
                         mal operational conditions as well as how the system performs under
                         maximum workloads, We found several deficiencies in the Navy’s test-
                         ing of phase II NALCOMIS.


Test Sites Are Not       Defense Directive 7920.1, Life Cycle Management of Automated Infor-
                         mation Systems, and Navy Instruction 5231. lB, Life Cycle Management
Representative           Policy and Approval Requirements for Information System Projects,


                         Page 13                             GAO/IMTEGSO-11   Navy NALCOMTS Acquisition
Chapter 2
Further Testing and Management Actions
Needed to Reduce Phase II Deployment Risks




state that information systems, including those like NALCOMIS that will be
deployed to multiple sites, should be field tested at one or more repre-
sentative sites prior to deployment. However, neither MAG-14 nor NAS-
Norfolk is representative of all sites scheduled for phase II deployment
in terms of (1) expected workloads or (2) requirements for on-line sys-
tem availability.

The workload at NAs-Norfolk, the larger of the two test sites, is only
about one-third that of NAs-Oceana and NM-Cecil Field, the two largest
sites scheduled to receive phase II. Specifically, the number of items
processed per year at NAs-Norfolk is about 33,000. In contrast, the
number processed at NAs-Oceana is about 96,000 and the number at NAS-
Cecil Field is about 95,000. Six other air stations also have workloads
greater than N.As-Norfolk’s.L NALCOMIS officials agree that NAS-Norfolk is
not representative of the larger air stations, In fact, the officials respon-
sible for software development stated that they are certain that the
hardware configuration at NAS-Norfolk will not work at NAS-Oceana.
According to the NALCOMIS officials, NAs-Norfolk was selected as the first
air station to receive phase II because of its physical proximity to the
Navy office responsible for software development, not because it is rep-
resentative of all phase II sites.

Additionally, the requirement for on-line system availability to users at
NM-Norfolk and ~~-14 is less demanding than the on-line system avail-
ability requirement aboard aircraft carriers. NALCOMIS system require-
ments state that the system be up and on-line aboard an aircraft carrier
22 hours a day, 26 days a month. In contrast, the requirement for shore
sites, like NAs-Norfolk, is 22 hours a day, but for only 21 days a month.
According to officials in the Commander Naval Air Force, Atlantic Fleet,
the difference reflects the fact that shore sites operate 6 days per week
while carriers operate 7 days per week.

Moreover, the Navy has experienced difficulty in meeting the less
demanding on-line availability requirement at shore sites. NAS-Norfolk
system availability statistics show that on-line availability to users dur-
ing the S-day work week has averaged about 20 hours a day. However,
this average availability has been achieved by performing some process-
ing and system maintenance on the weekends, and thereby improving
the daily average for on-line availability. This option would not be possi-
ble aboard a carrier because the system must be on-line every day of the
week. When NAS-Norfolk’s system availability is viewed over the 7-day

‘These workloads range from about 41,000 to 78,000 items processed per year.



Page 14                                         GAO/IMTEGSO-11      Navy NALCOMIS   Acquisition
                            chap.er 2
                            Further Testing and Management Actions
                            Needed to Bcduce Phaee II Deployment Rleka




                            period, as it would be aboard a carrier, on-line availability drops to
                            about 19 hours a day. Officials in the Office of the Secretary of
                            Defense’s Directorate for Operational Test and Evaluation and Comp-
                            troller’s Office as well as NALCOMIS program officials agreed that NAS-
                            Norfolk is not representative of a carrier in terms of on-line system
                            availability.


Testhg to Measure Systenn   Stress testing is an integral part of capacity management.2 Its purpose is
Performance Under           to (1) ensure that the total system will successfully process workloads
                            expected during peak production periods and other extreme conditions
Max/mum Workloads Not       and (2) determine the point at which major system resources (e.g.,
Adequate but                processor, channels, primary storage, etc.) will be exhausted. Volume II
Improvements Underway       of Defense Software Test and Evaluation Manual and Federal Informa-
                            tion Processing Standard Publication 102 advocate stress testing. Addi-
                            tionally, the official in the Office of the Director for Operational Test
                            and Evaluation responsible for NALCOMIS oversight told us that system
                            testing should have included tests for extreme and abnormal conditions,
                            but did not.

                            Early in our review, NALCOMIS officials stated that the system configura-
                            tion needed by each phase II site would be determined as the system was
                            implemented at a given site. Additionally, NALCOMIS officials told us that
                            they had no plans to stress test the system before deploying it to a large
                            NAS. According to the official responsible for MAG-14 testing, stress test-
                            ing was not possible because they did not have the capability to generate
                            the heavy workloads needed to do so. After we expressed concern about
                            using such a trial and error approach to configuring systems, the Navy
                            took steps to acquire a commercially available capacity management
                            tool that can emulate a specified transaction work load on a proposed
                            system configuration and monitor the system’s performance. The Navy
                            has also analyzed and described the work loads expected at large air
                            stations. According to NALCOMIS officials, they “hope” to have acquired
                            the full emulation package necessary to simulate a large NAS before
                            deploying phase II to NAs-Lemoore, currently scheduled for early 1990.
                            In our opinion, acquiring the emulation package is a positive step, and it
                            should be used to stress test the system before operationally testing it at
                            a large site such as NAs-Lemoore.


                            %apacity management ensures that computer systems (1) are properly designed and configured to
                            give efficient performance and (2) have sufficient resources to support operating work loads. As part
                            of this process, future work loads and required user service levels (e.g., system availability) are fore-
                            casted, and system configurations to meet the demands are proposed, modeled, and tested.



                            Page 16                                             GAO/lMTEC9O-11       Navy NALCOMIS       Acquisition
                             Chapter 2
                             Further Testing and Management Actions
                             Needed to Reduce Phase II Deployment Risks




Ideotification and Testing   The Navy has not complied with key provisions of Navy and Defense
                             security requirements in (1) determining what security features should
of decurity Features Not     have been included in NALCOMIS and (2) testing those security features.
Ad&quate                     Further, the Navy has operated, and in light of deficiencies in recent
                             security testing, may still be operating NALCOMIS without satisfying its
                             own requirements for system certification and site accreditation.

                             Navy Instruction 5239.1A, Department of the Navy Automatic Data
                             Processing Security Program, states that information system security is
                             intended to (1) protect data against accidental or intentional destruc-
                             tion, modification, or disclosure, and (2) protect users against denial of
                             service that may result from such events as fraud, misuse, and sabotage.
                             Additionally, this instruction requires the development and use of risk
                             assessments in the system development process to systematically exam-
                             ine system threats and vulnerabilities and determine cost-effective
                             countermeasures to use.

                             The Navy did not do the required risk assessment and, therefore, could
                             not base the security features included in NALCOMIS on such an assess-
                             ment. The reason for not doing so, according to the NALCOMIS deputy pro-
                             gram manager, was that the security features built into NALCOMIS and
                             described in the NALCOMIS functional description were devised before any
                             requirement for such an assessment. However, that part of Chapter 5 of
                             Navy Instruction 5239.1A addressing risk assessments is dated August
                             19823 , while NALCOMIS officials stated that the NALCOMIS functional
                             requirements were not “baselined” until 1986 and the latest version of
                             the functional description is dated September 1988. The deputy program
                             manager later agreed that the timing of NALCOMIS does not exempt the
                             system from Navy Instruction 5239.1A. In our opinion, without a risk
                             assessment, the Navy does not know whether the security features in
                             NALCOMIS are either sufficient or cost effective.

                             According to Navy Instruction 5239.1A, a Navy activity cannot operate
                             a computer system or network without first obtaining either site accred-
                             itation or a written interim authority to operate the system. Accredita-
                             tion is approval by the designated approval authority4 to operate based
                             on a review of the site’s total security posture for its computer systems




                             3Navy Instruction 6239. IA updated parts of Navy Instruction 5239.1 on April 1,1985.

                             4Varies depending on the level of security required.



                             Page 16                                            GAO/IMTFE-SO-11   Navy NALCOMIS     Acquisition
    Chapter 2
    Further Testing and Management Actions
    Needed to Reduce Phase II Deployment Risks




    and networks. An interim authority to operate can be viewed as tempo-
    rary accreditation for a fixed time period, usually one year. The instruc-
    tion also states that systems to be deployed to multiple sites, like
    NALCOMIS, must be certified before site accreditation can be given to any
    site. System certification refers to a determination that the system’s
    software is secure (i.e., system security features are functioning
    properly).

    Key NALCOMIS security features were not tested as part of NALCOMIS
    security testing. As a result, the Navy’s recent certification of NALCOMIS
    software as secure is questionable. In March 1989, the Navy developed a
    NALCOMIS security test plan and in June 1989 executed part of this plan.
    The remainder of the test plan, according to the NAL~OMIS program man-
    ager, is to be completed by each phase II site. Based on the testing per-
    formed, the Navy certified the NAIXOMIS phase I and II software as
    secure. However, we found instances where key security controls were
    not tested as part of the system security test. For example, the plan
    states that “users will not have access to the operating system” (i.e.,
    users cannot execute operating system commands/privileged instruc-
    tions). However, the results of security testing do not show that this
    restriction was tested prior to security certification. According to the
    NALCOMIS program manager, this requirement was to be tested by each
    site. In our opinion, not only does this approach represent a significant
    duplication of effort, but more importantly it raises serious doubt about
    whether the NALCOMIS security certification is based on adequate testing.
    NALCOMIS officials agreed with our opinion, stating that all software test-
    ing, including restrictions on execution of privileged instructions, should
    have been performed centrally before certification was given. However,
    they also stated that although certain security features were not tested
    as part of formal security testing, the features are exercised daily as a
    consequence of working with the system. The officials said that the
    problem is that they have not documented their efforts to exercise these
    features. Additional examples of security features relating to the oper-
    ating system that were not identified as being tested in the security test
    results report, despite being cited in the test plan, include:

9 all processor instructions/operation codes (e.g., load, add, subtract, etc.)
  will produce known responses by the computer;
l read, write, and execute access rights of the user will be verified each
  time a computer instruction is executed;
. unauthorized attempts to change, circumvent or otherwise violate sys-
  tem security features will be detectable and will abort or suspend the
  operation running; and


    Page 17                                      GAO/IMTEG90-11   Navy NALCOMIS   Acquisition
                               Chapter 2
                               F’urther Testing and Management Actions
                               Needed to Reduce Phase II Deployment Riska




                             . an audit log or file will be maintained as a history of system use to per-
                               mit regular security reviews.

                               The Navy began deploying NALCOMIS phase I in 1985 and phase II was
                               installed at MAG-14 on a prototype basis in 1986; however, it did not
                               certify phases I and II software, which is a prerequisite to NALCOMIS site
                               accreditation, until July 1989. As a result, the Navy operated both
                               NALCOMIS phase I and phase II for at least 3 years at various sites with-
                               out meeting the conditions for site security accreditations. Additionally,
                               the NAIEOMIS program manager did not know whether any of the sites
                               had an interim authority to operate and was unable to provide any evi-
                               dence that they did. Thus, we believe that the Navy has operated
                               NALCOMIS and, in light of our above-cited concern that NAU=OMIS security
                               testing may not provide sufficient basis for software certification, may
                               still be operating it without satisfying the requisite conditions for doing
                               so.


Regression Testing Limited     Regression testing is testing following a program change to (1) ensure
but Improvements Planned       that the change has corrected the problem and (2) demonstrate that no
                               new problems have been introduced in any part of the system as a result
                               of the change. Regression testing is important because new errors are
                               often introduced when software is modified.

                               The Navy’s current approach to NALCOMIS regression testing is to do it
                               manually. According to NALCOMIS officials, 6-8 people manually enter
                               test transactions at terminals for a 45- to 60-day period for each new
                               software release. These officials stated that the tests cover “100 percent
                               of the code directly affected by a change and a random sample of the
                               remainder of the system.” However, they admitted that they suspect
                               that their people entering test transactions skip pages in the test proce-
                               dures and sometimes misinterpret test results because of waning atten-
                               tion. In our opinion, such an approach to regression testing is
                               inadequate. NAIXOMIS officials agreed that this approach is limited, and
                               since we first raised concerns about regression testing, the officials told
                               us that they plan to use an emulation tool being acquired to develop and
                               run an automated and repeatable regression test suite. On the basis of
                               our discussions with representatives of the company selling the pro-
                               posed emulation tool and a review of literature describing its features
                               and functions, we believe that it will aid in correcting the Navy’s regres-
                               sion testing problems.




                               Page 18                                      GAO/IMTEG90-11   Navy NALCOMB   Acquisition
                       chapter 2
                       FurtherTesting and Management Actions
                       Needed to Reduce PhaseII Deployment ~ICB




                       Navy Instruction 5231. lB, Life Cycle Management Policy and Approval
      of Phase II      Requirements for Information System Projects, requires that all compo-
Maintenance Manual     nents of an information system be identified and documented, and that
and Test               changes to these components be controlled, recorded, and reported.
                       Among the many components subject to these requirements are (1) sys-
Do4urnentation Will    tem test plans and the associated test results and (2) system manuals
Make System            (e.g., users manual, operators manual, and maintenance manual). These
                       requirements are designed to help ensure that the system meets users
Maintenance            requirements and can be operated and maintained efficiently and
Unpecessarily          effectively.
Difficult and Costly
                       We found that some key phase II documentation does not exist. As a
                       consequence, the Navy has seriously amplified the risks associated with
                       maintaining NALCOMIS. To illustrate, Navy Instruction 5232.1 as well as
                       the NALCOMIS Configuration Management Plan and the Quality Assurance
                       Plan require a program maintenance manual. According to Defense
                       Standard 7935, Automated Data Systems Documentation, a program
                       maintenance manual provides the maintenance programmers with the
                       information necessary to effectively maintain a system. However,
                       NALCOMIS officials stated that no such manual exists because a conscious
                       decision was made not to require one under the terms of the NALCOMIS
                       system development contract. The officials stated that maintenance per-
                       sonnel will rely on various system specifications and “heavily com-
                       mented” application source code in lieu of such a manual, and given
                       their knowledge of the system and the modular structure of the applica-
                       tion code, they added that this is all the documentation they have
                       needed to maintain the system for over 1 year. Additionally, the pro-
                       gram manager told us that existing documentation will be augmented by
                       word-of-mouth and on-the-job training. In our opinion, this approach is
                       not adequate for a system as large and important as NALCOMIS. Although
                       system maintenance may be possible without a manual, it will undoubt-
                       edly be more difficult and costly than necessary.

                       Limited phase II test documentation compounds the Navy’s maintenance
                       difficulties for NALCOMIS. Specifically, we found that only a report sum-
                       marizing the results of phase II testing at MAG-14 exists. The detailed
                       test results, according to the Navy official responsible for MAG-14 test-
                       ing, were discarded. As stated in Defense Software Test and Evaluation
                       Manual, Volume II, creating and maintaining the complete test documen-
                       tation chain is important. Specifically, future events may produce ques-
                       tions that can be answered using existing test results, supplemented
                       possibly with only limited new testing. If software test results are
                       improperly recorded or lost, the benefits of such economies cannot be


                       Page 19                                    GAO/JMTEG90-11   Navy NALCOMIS   Acquisition
                        Chapter2
                        Further Testing and Management Actions
                        Needed to Reduce Phase II Deployment Risks




                        realized. The manual further states that a lack of proper testing docu-
                        mentation can call into question the adequacy of testing in total.

                        Also, the test plan for MAG-14 includes only ambiguous and general guid-
                        ance, leaving the choice of specific test procedures and test data to the
                        discretion of the individuals performing the tests. Documented, detailed
                        test procedures for MAG-14do not exist, according to the Navy official
                        responsible for MAG-14 testing. As stated in Defense Standard 2167A,
                        Defense System Software Development, test procedures should specify
                        exactly what test inputs to provide, what steps to follow, what outputs
                        to expect, and what criteria to use in evaluating the outputs. If any of
                        these elements are absent, the test procedures are to be considered
                        inadequate.


                        Navy Instruction 5232.1, Quality Assurance Program for Information
Navy Currently          System Projects, states that quality requirements will be defined for
Taking Steps to         each information system project and progress against these require-
Address Transition to   ments will be assured. One quality factor the instruction identifies is
                        portability, which is the effort required to transfer software from one
Neh Hardware            hardware configuration or system software environment to another.
                        According to Defense Guidelines for Software Test and Evaluation, port-
                        ability is important to software that is expected to outlive its hardware.

                        Portability is a relevant and important issue for NALCOMIS. We found that
                        the current NALCOMIS processors are old, and that the contract from
                        which this hardware is being purchased expires in 1992. Specifically,
                        NALCOMIS   processors are bought off of a Navy-wide hardware and sys-
                        tems software contract. Awarded in 1982, this fixed-price contract
                        spans 10 years with an option to extend for 10 additional years. How-
                        ever, the contractor does not plan to continue marketing the processors
                        available under the contract, and Navy officials said that the processors
                        were obsolete even before the NALCOMIS software was developed.

                        Although the contract includes a lo-year renewal option, a resolicitation
                        appears likely. Under the terms of the contract, the Navy is to state
                        technology improvement requirements by contract year 8 (i.e., 1990). If
                        the contractor responds to the Navy’s requirement at a reasonable cost
                        to the government, then the lo-year option may be negotiated and
                        awarded. If the contractor does not respond or if the response is not
                        accepted by the Navy, a competitive procurement for phase II replace-
                        ment hardware and phase III initial hardware will ensue. According to
                        Automatic Data Processing Selection Office officials responsible for this


                        Page 20                                      GAO/IMTEGSO-11   Navy NALCOMIS   Acquisition
                       Chapter 2
                       Further Testing and Management Actions
                       Needed to Reduce Phase II Deployment Risks




                       process, a competitive resolicitation and contract award is likely because
                       of Competition in Contracting Act considerations.

                       Despite the importance and relevance of software portability to
                       NALCOMIS, the Navy had not addressed this issue until our inquiries. We
                       found early in our review that the Navy was not using an available soft-
                       ware monitoring utility that flags deviations from standards for the pro-
                       gramming language being used for all applications. Further, we found
                       that they had not examined the extent to which software contractor’s
                       compiler complied with standards. Any deviations from standards will
                       make the software more difficult to “port” to different hardware.
                       According to the NALCOMIS program manager, portability was heretofore
                       not examined because they did not plan to transfer the system to any
                       other hardware in the near future. In our opinion, the Navy should have
                       considered portability before it developed NALCOMIS, and we suggested
                       that the Navy immediately develop a systematic approach to the sys-
                       tems inevitable migration to new hardware.

                       Since then, NALCOMIS officials told us that they have examined portions
                       of the application code and estimate that about 30 percent is non-stand-
                       ard. They also stated that they plan to evaluate all of the code’s compli-
                       ance with the standards in the near future. Additionally, they recently
                       began a study addressing, among other things, the impact of converting
                       NALCOMIS to operate on new hardware.


    /
                       During our review, we visited two phase II sites, NAS-Norfolk and MAG-
User Reaction to       39, to obtain users’ reactions to the system and to observe the system in
Phase II Is Positive   operation. We also interviewed NALCOMIS user representatives with the
                       Navy’s Atlantic and Pacific Commands as well as Office of the Secretary
                       of Defense officials responsible for NALCOMIS oversight to obtain their
                       impressions of user reaction to phase II. In general, we found that users
                       are reacting positively. For example, users at NAs-Norfolk stated that
                       phase II has improved the efficiency of aviation maintenance and sup-
                       ply at the air station and that since phase II, the air station has moved
                       from a reactive to a proactive operation. Other users at NAs-Norfolk told
                       us that phase II has greatly facilitated their work. Users at MAG-39 also
                       offered favorable comments about phase II. For example, one user
                       stated that he “feels that overall the staff are more productive” since
                       phase II, and another summed up MAG-39 user reactions by stating that
                       the general feeling among MAG-39 users is that operations are improving
                       because of phase II.



                       Page 21                                      GAO/IMTEG90-11   Navy NALCOMIS   Acquisition
Chapter 2
Further Testing and Management Actions
Needed to Reduce Phase II Deployment Risks




Our interviews with user representatives and NALCOMIS oversight offi-
cials also revealed user satisfaction with phase II. Specifically, user rep-
resentatives with both the Atlantic and Pacific Commands stated that
users “like” phase II. Similarly, the official in the Office of the Secretary
of Defense’s Office of Operational Test and Evaluation who is responsi-
ble for oversight of testing stated that users “like” phase II. Likewise,
the official in the Assistant Secretary of Defense (Program Analysis and
Evaluation) responsible for oversight of system costs told us that the
phase II users he talked to are “very happy” with the system.

In concert with our interviews of NM-Norfolk and MAG-39 users, we wit-
nessed the input, processing, and output of various ad hoc transactions
in both the maintenance and supply areas. On the basis of this limited
observation of the system, we found that users were generally comforta-
ble with the system, and we observed no apparent problems in system
performance.




Page 22                                      GAO/IMTEG90-11   Navy NALCOMIS   Acquisition
Chapter 3

Naky Needsto DemonstratePhaseII Benefits


                        The primary objective for NALCOMIS is to improve aircraft readiness, and
                        the Navy has largely justified the system on this basis. Additionally, the
                        Navy has been operating phase II at five sites for periods ranging from 1
                        year to over 2 years. Despite this, the Navy has yet to demonstrate
                        through actual experience at any of these sites, as required by Defense
                        and Navy instructions, that expected operational readiness benefits are
                        being realized. Moreover, our quick look at certain indicators of phase
                        II’s operational impact at four sites provided mixed results, and thus we
                        believe it is unclear whether readiness improvements are actually occur-
                        ring. Without thoroughly evaluating and validating expected benefits,
                        the Navy does not have sufficient information to assure Office of the
                        Secretary of Defense and congressional decision makers that phase II is
                        cost effective.


                        Although the principal goal for NALCOMIS is to improve aircraft readi-
Navy Evaluations of     ness, the Navy has yet to demonstrate through actual experience at a
Phase II Benefits Are   phase II site that expected readiness benefits are accruing. Further,
Limited                 while the Navy has analyzed phase II administrative and clerical bene-
                        fits and has identified the potential for significant savings, we do not
                        believe that these analyses adequately justify deployment of phase II.

                        The primary goal of aviation maintenance and supply is ensuring that
                        the maximum number of aircraft in the Navy’s inventory are mission
                        ready and safe. Achieving this goal is the Navy’s principal justification
                        for NALCOMIS. Specifically, the Navy’s fiscal year 1990/1991 budget sub-
                        missions to the Congress state that the Navy is acquiring NALCOMIS to
                        improve aircraft operational readiness. Similarly, Navy life cycle man-
                        agement documents state that a principal objective of NALCOMIS is to
                        implement a system that will measurably improve aircraft readiness.

                        Defense Instruction 7041.3, Economic Analysis and Program Evaluation
                        for Resource Management, and Navy Instruction 7000,14B, Economic
                        Analysis and Program Evaluation for Navy Resource Management,
                        require that evaluations of ongoing programs be conducted to (1) ensure
                        that expected benefits are being attained in the most cost effective man-
                        ner and (2) determine how best to improve the programs. The instruc-
                        tions further require that these program evaluations be conducted as
                        early in the acquisition process as practical and they state that the eval-
                        uations should, among other things, compare actual performance data
                        against expected performance data.




                        Page 23                              GAO/EMTEGSO-11   Navy NALCOMIS   Acquisition
Chapter 3
Navy Needs to Demonstrate     Phase II Benefits




The Navy’s most recent evaluations of Phase II benefits are (1) the Ben-
efit Analysis included in the life cycle management documentation dated
February 1989 and reviewed by the MAISRC in July 1989, and (2) an
Independent Benefit Analysis conducted by the Naval Regional Data
Automation Command and dated February 1989. However, the former
Benefit Analysis is the same one used in justifying the Navy’s 1987 lim-
ited deployment decision, and thus is over 2 years old. As a result, this
analysis only addresses phase II administrative and clerical savings. It
does not address expected benefit areas such as increased productivity,
efficiency, and readiness. According to the Benefit Analysis, the Navy
deferred evaluating these productivity, efficiency, and readiness bene-
fits until after it deployed the system because any such evaluation
would require pre- and post-phase II mission effectiveness data for a
given site. At the time the analysis was performed (i.e., over 2 years
ago), phase II was operating for only a few months at one site and thus
evaluating operational benefits was not viewed as feasible. However,
since that time, phase II has operated at four sites for over 1 year and
one site for over 2 years.

Although phase II’s administrative and clerical benefits identified by the
Benefit Analysis are significant, they represent potential benefits that
the Navy has yet to demonstrate as actually accruing. Specifically, the
analysis identifies benefits totalling about $113 million a year.’ How-
ever, about $107 million or 96 percent of these benefits are attributable
to more efficient use of supply and maintenance personnel (i.e., freeing
supply and maintenance specialists from performing administrative and
clerical tasks). Further, the analysis states that this $107 million is not a
savings but a quantification of the potential value of technical labor pro-
ductivity increases, and that reducing the work force cannot be justified
by the productivity increases. In our opinion, the true measure of
whether any benefits are accruing from this redirection of technical
labor is whether aircraft maintenance and supply readiness effective-
ness has increased and/or work force reductions are possible. However,
the Navy has yet to analyze phase II’s effect on maintenance and supply
readiness, and the analysis states that it is not a basis for work force
reductions. Thus, the analysis stops short of demonstrating actual
benefits.

The more recent Independent Benefit Analysis also demonstrates only
potential benefits and thus does not justify full fleet deployment of

‘Primarily labor and other costs associated with the maintenance of manual records and input of data
from forms.



Page 24                                           GAO/lMTECsO-11     Navy NALCOMIS     Acquisition
 .

                            Chapter 3
                            Navy Needs to Demonstrate     Phase II BenefIta




                           phase II. Like the above discussed analysis, the Independent Benefit
                           Analysis identifies significant administrative and related savings. How-
                           ever, this independent analysis also does not address readiness improve-
                           ments. In our opinion, the only way to demonstrate any real benefit
                           from the reported productivity increases is to show either (1) improve-
                           ments in readiness, which have not been shown, or (2) planned reduc-
                           tions in work force levels, which are not planned based on the benefit
                           analyses.

                           Office of the Secretary of Defense and Navy program officials agreed
                           that program evaluations are needed to determine the actual operational
                           impact of phase II on aircraft maintenance and supply readiness, and
                           according to the deputy program manager, the evaluations will be per-
                           formed. However, this official stated that no efforts are underway to
                           plan for them, and no dates have been set for such actions to begin.


                           We performed a limited examination of several operational performance
Indicators of Phase II     indicators at four sites to see whether phase II was achieving its goal of
Effect on Aircraft         improved aviation readiness. However, we did not attempt to perform
ReadinessAre Mixed         the type of in-depth, thorough program evaluation required by Navy
                           and Defense instructions and directives. Specifically, for each of the
                           four sites, we looked at pre- and post-phase II statistics on:

                           Turnaround Time (i.e., the amount of time a part spends in the repair
                           cycle). This indicator should decrease with phase II.
                           Ready For Issue Material (i.e., parts that are ready for issue). This indi-
                           cator should increase with phase II.
                           Awaiting Parts (i.e., the condition that exists when a part under repair
                           must wait for needed material(s)). This indicator should decrease with
                           phase II.
                         . Expeditious Repair (i.e., the processing of repairs requiring expeditious
                           action because a replacement part is not in supply). This indicator
                           should decrease under phase 11.

                           We chose these statistics because Navy benefit analyses for phase I and
                           II and Navy officials cited them as indicators of NALCOMIS' effect on site
                           performance. The statistics covered the period beginning 5-6 months
                           before phase II implementation at each site to the period 4-12 months
                           following implementation2

                           “Post-phase II time periods vary because site implementation dates differ as do the latest dates that
                           statistics are available for each site.



                           Page 26                                            GAO/IMTEG90-11       Navy NALCOMIS      Acquisition
Chapter 3
Navy Needs to Demonstrate    Phase II Benefits




Our comparison3 of pre- and post-phase II statistics provided mixed
results, and does not conclusively demonstrate the effect of phase II on
sites’ operational readiness posture. However, it does raise questions
about the benefits of phase II and reinforces the need for a thorough
program evaluation of phase II’s impact on aircraft readiness. Specifi-
cally, we found that:

Average Turnaround Time decreased about 3 and 5 percent at two sites,
but increased about 9 percent and 18 percent at the other two sites;
Ready For Issue Material increased about 1 percent at two sites, while it
decreased about 2 percent and 7 percent at the other two sites;
Average Awaiting Parts time decreased about 41 percent and 27 percent
at two sites, while it increased about 13 and 14 percent at the other two
sites;

Finally, our analysis of the number of Expeditious Repairs, although
constrained by limited data, also provided inconsistent results. Specifi-
cally, at one site where only post-phase II data was available, the
number of Expeditious Repairs consistently increased for 4 months and
then consistently decreased for the next 3 months. At another site
where only post-phase II data was available, the number of Expeditious
Repairs fluctuated monthly. Last, at a site where both pre- and post-
phase II data were available, the number of Expeditious Repairs
increased by an average of about 3 percent after phase II implementa-
tion. No monthly statistics on Expeditious Repairs were available for the
fourth site.




“We compared the percentage change between the average monthly pre-phase II and post-phase II
statistics.



Page 26                                          GAO/lMTEC9O-11    Navy NALCOMIS     Acquisition
Chapter 4

Cohclusionsand Recommendations


              The Navy is attempting to deploy phase II NALCOMIS before it satisfies its
Con$lusions   own software development requirements and takes the prudent and nec-
              essary management steps to (1) minimize system cost and performance
              risks and (2) validate expected system benefits. Specifically, the Navy
              has limited phase II field testing to intermediate maintenance activities
              that are only representative of the small to medium size shore sites
              scheduled to receive the system. As a result, the Navy does not have
              reasonable assurance that the system can perform successfully on air-
              craft carriers or at large phase II sites. Further, although the Navy is
              acquiring a capacity management tool to assist in configuring the system
              at all sites, the entire tool may not be acquired before phase II is sched-
              uled for deployment to the first large air station. Even if it is, use of the
              tool does not substitute for the need to operationally test phase II at
              representative sites. Additionally, although the Navy has performed
              system maintenance for over 1 year, it has not developed a system
              maintenance manual nor maintained certain system documentation
              which is essential to effective and economical system maintenance. Also,
              it has not conducted a security risk assessment, and its security test did
              not address important system controls. Finally, the Navy has not con-
              ducted the requisite program evaluations to determine whether readi-
              ness improvements in aviation maintenance and supply operations,
              which it expected phase II to deliver, are actually accruing.

              In July 1989, the MAISRC reviewed NALCOMIS and established several con-
              ditions that the Navy must meet before phase II deployment would be
              granted. In October 1989, the Navy advised the MAISRC of the actions it
              took to address each condition. The Navy also advised the MAISRC that it
              was proceeding with phase II deployment. As of December 1989, the
              MAISRC was reviewing the Navy’s actions and had yet to rule on whether
              it satisfied the conditions.

              To deploy NAL~OMIS phase II, the Navy plans to spend about $173 million
              over the next 5 years. In light of this substantial investment, it is essen-
              tial to ensure before proceeding that the system will work as intended at
              all sites and that the expected benefits will be achieved. While we recog-
              nize that phase II is operating at nine sites, users at the sites we visited
              voiced their satisfaction with the system, and the Navy has taken some
              steps to reduce NALCOMIS deployment risks, more needs to be done. In our
              opinion, phase II should not be deployed until it has been operationally
              field tested at the more demanding intermediate maintenance activities.
              Additionally, the Navy must thoroughly evaluate its operational phase
              II system to ensure that the readiness benefits it expects from the sys-
              tem are occurring. To proceed with deployment without resolving these


              Page 27                              GAO/IMTEGSO-11   Navy NALCOMIS   Acquisition
                  Chapter 4
                  Conclusions   and Recommendations




                  shortcomings raises the risk that phase II NALCOMIS will not perform as
                  expected and will take longer and cost considerably more to develop and
                  maintain than is necessary.


                  We recommend that the Secretary of Defense direct the Secretary of the
Rckommendations   Navy to defer any further phase II deployment until the NALCOMIS pro-
                  gram office (1) fully stress tests the system, (2) successfully completes
 I
                  operational testing aboard a carrier and at a large Naval air station, (3)
 ,                clearly demonstrates that system benefits are actually being achieved,
 I                (4) fully assessessystem security requirements and completely tests
                  system security features, and (5) develops a system maintenance man-
                  ual. We further recommend that the Secretary of Defense direct the
                  MAISRC to withhold its phase II deployment approval until the council
                  reviews the Navy’s efforts to expeditiously satisfy the above conditions.




                  Page 28                             GAO/EWFEG90-11   Navy NALCOMJS Acquisition
L




    w




        Page 29   G A O /IM T E G S O - 1 1 Navy N A L C O M IS Acquisition
Apbendix I

Objectives,Scopeand Methodology


                 Concern about cost increases on the NALCOMIS program prompted the for-
                 mer Chairman, Defense Subcommittee, House Committee on Appropria-
                 tions to ask us for an update on the program. The Chairman specifically
                 expressed interest in NALCOMIS' costs, budgetary disclosure, benefits,
                 development and testing, and hardware procurement schedules. On the
                 basis of the October 19,1988, request and subsequent discussions with
                 the Chairman’s office, we agreed to determine whether

             . the benefits and risks associated with the second phase of NALCOMIS war-
               rant its full deployment,
             l the NALCOMIS life cycle cost estimate is accurate and complete,
             . the Navy has fully disclosed the NALCOMIS life cycle cost estimate to Con-
               gress, and
             . the fiscal year 1990-1991 NALCOMIS operations and maintenance and pro-
               curement budget requests are appropriate in light of the number of
               activities either operating or scheduled to operate NALCOMIS during this
               period.

                 This report addresses the first objective. We provided our results on the
                 other three objectives during a September 19,1989, briefing to the
                 Chairman’s office.

                 With respect to the risks associated with NALCOMIS phase II, we
                 examined the type and extent of testing performed on the system,
                 including the results of testing and how test findings were addressed.
                 We also examined the Navy’s approach for sizing the system (i.e., deter-
                 mining what hardware configuration was needed for each site) as well
                 as its adherence to system life cycle management requirements. Con-
                 cerning system benefits, we examined the Navy’s justifications for
                 NAICOMIS (i.e., statements of expected benefits) as well as its efforts to
                 validate benefits from phase I and phase II NALCOMIS.

                 In accomplishing these tasks, we reviewed relevant Defense and Navy
                 instructions and directives as well as federal requirements and generally
                 accepted industry practices concerning system testing, sizing, and docu-
                 mentation. We also reviewed Defense and Navy requirements for defin-
                 ing and validating expected system benefits. Additionally, we reviewed
                 NALCOMIS test plans and procedures, test results, and test reports as well
                 as NALCOMIS benefit assessments, system sizing analysis, and other perti-
                 nent life cycle management and system documentation. We also col-
                 lected and performed some analysis on aircraft maintenance and supply
                 performance data from four phase II sites for the 5-6 month period



                 Page 30                              GAO/lMTECSO-11   Navy NALCOMIS   Acquisition
Appendix I
Objectives, Scope and Methodology




before system implementation and the 4-12 month period following sys-
tem implementation to quickly see whether phase II benefits were actu-
ally accruing.

Our accomplishment of these tasks also included interviewing officials
responsible for program management, contract management, system
testing, and system development and maintenance as well as interview-
ing system users at four Naval air stations and one Marine air group and
user representatives with the Navy’s Atlantic and Pacific Commands.
We also interviewed various officials within the Navy and the Office of
the Secretary of Defense having system oversight and approval roles.
Additionally, we interviewed representatives from Honeywell Federal
Systems, Inc. and Neal Nelson & Associates, whose products are being
used.

We performed our work from November 1988 to September 1989, at
Navy and Office of the Secretary of Defense headquarters offices in
Washington, D.C., and at selected Navy field activities. The principal
Navy headquarters offices include the NALCOMIS program office, under
the Naval Air Systems Command; the Navy Management Services Sup-
port Office, the central design agency for NALCOMIS; the Naval Data
Automation Command; the Naval Aviation Maintenance Program Divi-
sion within the Assistant Chief of Naval Operations for Air Warfare, the
functional sponsor; and the Automated Data Processing Selection Office.
Field activities visited include the Commander Naval Air Forces-Atlantic
Fleet, Commander Naval Air Forces-Pacific Fleet, Naval Air Station-
Jacksonville (no NALCOMIS), Naval Air Station- Cecil Field and Naval Air
Station-Oceana (phase I NALCOMIS), Naval Air Station-Norfolk and
Marine Air Group-39 (phase II NAWMIS). The principal offices within
the Office of the Secretary of Defense include the Office of the Deputy
Assistant Secretary (Information Resources Management) within the
Assistant Secretary of Defense (Comptroller), the Directorate for Opera-
tional Test and Evaluation, the Directorate for Forces Structure and
Support Cost Analyses Division within the Assistant Secretary of
Defense (Program Analysis and Evaluation), and the Directorate for
Plans and System Implementation within the Assistant Secretary of
Defense (Production and Logistics).

We discussed the facts in this report with Navy and Office of the Secre-
tary of Defense officials and have incorporated their comments where
appropriate. However, in accordance with the requester’s wishes, we did




Page 31                             GAO/lMTJK-SO-11   Navy NALCOMIS   Acquisition
Appendix I
Objectives, Scope and Methodology




not obtain official agency comments on a draft of the report. We per-
formed our review in accordance with generally accepted government
auditing standards.




Page 32                             GAO/IMTEGSO-11   Navy NALCOMIS   Acquisition
Appen&x II

M$or Contributors to This Report


                         James R. Watts, Associate Director
Information              John B. Stephenson, Assistant Director
Matiagement and          Randolph C. Hite, Evaluator-in-Charge
Technology Division,     Karlin I. Richardson, Technical Advisor
                         Robin M. Nazzaro, Evaluator
Washington, D.C.         Alice E. Morris, Evaluator
                         Erum N. Welling, Evaluator


                         Robert E. Erdman, Evaluator
Boston Regional Office




(51034H)                 Page 33                             GAO/E%lTEG(N)-11 Navy NALCOMIS   Acquisition