oversight

Navy's Evaluation Process in Ship Donation

Published by the Government Accountability Office on 1997-06-03.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

      United States
GAO   General Accounting  Office
      Washington, D.C. 20548

      National Security and
      International Affairs Division


      B-277117

      June 3, 1997

      The Honorable Norman D. Dicks
      House of Representatives

      Subject: Navy’s Evaluation Process in Ship Donation

      Dear Mr. Dicks:

      On August 21, 1996, the Secretary of the Navy announced his decision to donate
      the USS Missouri, a ship of historical significance, to the USS Missouri
      Memorial Association in Hawaii. At your request, we reviewed the facts
      surrounding the donation process. Specifically, we obtained information on the
      (1) process of applying for the ship, (2) evaluation criteria and weighting used
      to evaluate the applications, and (3) use of the criteria and weighting in the
      selection process. On May 22, 1997, we briefed you on the results of our work,
      which is summarized below. Additional details on our results are contained in
      the enclosed briefing charts.

      BACKGROUND

      The Secretary of the Navy has legal authority (10 U.S.C. 7306) to transfer title of
      ships no longer needed for the Navy’s purposes to not-for-profit entities and
      others. However, the law requires that (1) such a donation be made at no cost
      to the government, (2) the recipient must maintain the ship, and (3) Congress
      be allowed 60 days to review the Secretary’s decision.

      The Navy’s ship donation process is designed to help the Secretary determine
      whether those seeking a donation of a ship meet the Navy’s minimum
      requirements for financial and technical capabilities. In the past, with one
      exception, only one application was received for each of 43 donations and the
      qualified applicant received the donation. However, for the USS Missouri, the
      Navy received five applications.

      RESULTS IN BRIEF

      The Navy began the donation process for the USS Missouri in the same manner
      as prior donations, by requesting financial and technical information from the
      applicants and working with applicants to help ensure that their applications

                                                     GAO/NSIAD-97-171R USS Missouri
B-277117

           would satisfy the Navy’s financial and technical requirements. Subsequently,
           the Navy decided that, with respect to the USS Missouri, additional evaluation
           criteria, ‘historical significance” and “public affairs benefits to the Navy,” were
           needed to assist the Secretary in making the donation decision among four of
           five applicants that met the Navy’s financial and technical requirements. This
           was the first time such additional criteria were used in any donation selection
           process.

           While the Navy’s donation process appears to have been impartially applied, and
           all applicants were provided the same information on the additional criteria at
           the same time, the Navy did not do a good job in communicating its additional
           requirements to the applicants. Specifically, applicants were not told (1) what
           the relative importance of the evaluation criteria was in the process (the added
           criteria actually represented 75 percent of the donation award weight), (2) what
           the added evaluation criteria meant, or (3) how well already submitted
           applications met the added criteria (a procedure routinely used in the financial
           and technical evaluation process). These factors are particularly important
           because the Navy’s evaluation teams were told to base their scoring only on the
           information contained in the applications. As a result, to varying degrees, the
           evaluation teams found all applications lacking in information when measured
           against the added criteria. According to some applicants, had they lmown that
           the additional criteria carried so much weight, they would have revised their
            applications.

           As a result of our review, we are preparing a separate report to the Secretary of
           the Navy recommending changes to the Navy’s donation procedures to better
           handle future situations where there may be multiple applicants.

           AGENCY COMMENTS

           The Department of Defense reviewed a draft of this report and provided oral
           comments. It concurred with our report.

           SCOPE AND METHODOLOGY

           To obtain information for this report, we interviewed officials and reviewed
           files at the Naval Sea Systems Command, the Naval Historical Center, the Office
           of Chief of Naval Information, and the Office of the Deputy Assistant Secretary
           of the Navy for Ship Programs. We also interviewed representatives of four of
           the top five applicants; the fifth applicant has disbanded.

            We conducted our review during April and May 1997 in accordance with
            generally accepted government auditing standards.




            2                                                GAO/NSIAD-97-171R USS Missouri
B-2771 17


            As arranged with your office, we plan no further distribution of this report until
            1 day after its issuance. At that time, we will send copies of this report to the
            Secretary of Defense; the Secretary of the Navy; the Director, Office of
            Management and Budget; and other interested congressional committees and
            members. We will also make copies available to others upon request.

            Please contact me at (202) 5124587 if you or your staff have any questions
            concerning this report. Major contributors to this report were Charles W.
            Thompson and John P. Ting.

            Sincerely yours,



                                            I
            David E. Cooper
            Associate Director
            Defense Acquisitions Issues


            Enclosure




            3                                              GAO/-NSIAD-97-171RUSS Missouri
Enclosure                                                      Enclosure




     MCI Background

            l   IO U.S.C. 7306 confers authority on
                Secretary of Navy to transfer ships
                stricken from Naval Vessel Register to,
                among others, any not-for-profit entity.
            l   Statute requires that transfer be made’at
                no cost to the government, transferee
                maintain ship in conditions satisfactory to
                Navy, a notice be sent to Congress, and
                60 days of continuous session of
                Congress have expired from notice date.




                                              GAO/NSIAD-171   R USS Missouri
Enclosure                                                          Enclosure




    GA(3 Past Ship Donations

            l   43 past donations. All but one involved single
                applicant. Navy procedures require evaluation
                to determine if an applicant has financial and
                technical capabilities to move and sustain
                ship. Navy practice is to work with applicants
                to attain acceptable applications.
            l   1992:USS Lexinaton had three applicants.
                Navy scored and ranked applicants’ financial
                and technical capabilities. Since one applicant
                was clearly above the rest, no additional
                criteria used as a tiebreaker.




                                                 GAO/NSIAD-171   R USS Missouri
Enclosure                                                         Enclosure




     GAD Unique Circumstances Surrounding
         USS Missouri
            * USS Missouri is a ship of historical
              significance and commands considerable
              congressional and public interest.
                l    Instruments of Japanese surrender
                     signed aboard the ship on September 2,
                     1945, ending World War II.
            l       USS Missouri stricken from Naval Vessel
                    Register in January 1995, but donation
                    status held in abeyance until March 1996
                    due to conflicting legislative language.




                                                 GAO/NSIAD-171   R USS Missouri
Enclosure                                                                  Enclosure




    MCI Guidance to Applicants

            l       Navy’s initial guidance to applicants directed at
                    meeting minimum financial and technical
                    requirements. These dealt with
                l    funding needed to move and support the ship,
                l    mooring design to maintain the ship,
                l    environmental   requirements,
                l    towing plan to move ship to its permanent
                     location,
                l    continuing maintenance plan outlining how the
                     ship will be maintained in the future, and
                l    continuous security plan.




                                                         GAO/NSIAD-171   R USS Missouri
Enclosure                                                                       Enclosure




     MO             Five Applications for USS Missouri
                    (Date of Application in Parenthesis)
            l       Honolulu, Hawaii
                    l       USS Missouri Memorial Association    (Mar. 1995)
            l           Bremerton, Washington
                    l       Save the Missouri Committee (Oct. 1995)
                l       Long Beach, California
                    l       Battleship Missouri Foundation (Dec. 1995)
                l       San Francisco, California
                        l   USS Missouri Allied Forces Memorial (Oct. 1995)
                        l   San Francisco Operation Missouri (Mar. 1996)




                                                                GAO/NSIAD-171   R USS Missouri
Enclosure                                                                   Enclosure




     MO           Evaluation Criteria Added

            l   As applications were received, Navy conducted
                financial and technical reviews and requested
                clarification or additional information.
            l   With four applicants meeting Navy’s financial and
                technical requirements and given the historical
                significance of the ship, the Navy concluded there
                was a need for additional criteria in making an award.
            l   In May 1996, the Navy added “historical significance”
                and “public benefits to the Navy” to evaluation criteria.
            l   The Navy also assigned 50-percent weight to public
                benefits, 25 percent to historical significance, and 25
                percent to financial and technical capabilities.




                                                          GAO/NSIAD-171   R USS Missouri
Enclosure                                                                  Enclosure




     w      Applicants Notified of Added
            Evaluation Criteria
            l       On June 5, 1996, each of the five applicants was
                    notified for the first time that “In addition to the
                    financial and technical information that you have
                    provided..., your application will also be evaluated
                    in terms of its overall public benefit to the Navy
                    and the historical significance associated with
                    each location (to include the manner in which the
                    ship will be used as a naval museum or
                    memorial).” Notification was made in writing
                     (fax), with telephone confirmation.

                l   A June 21, 1996, deadline for submitting
                    additional information was provided in the letter.




                                           10             GAO/NSIAD-171   R USS Missouri
Enclosure                                                                  Enclosure




     GAO June 5th Notification A Major Source of
         Discontent Among Applicants
            l   No applicant told of specific or relative evaluation
                weightings, meaning of added criteria, or how well
                previously submitted applications met the added
                criteria.
                l   Two applicants said that had they known that the
                    historical and public benefits criteria carried 75
                    percent of the weight, they would have devoted
                    much more attention to them in their applications.
                l   They also stated they had to guess the meaning of
                    the term “public benefits to the Navy”. One took it to
                    mean “benefits to the Navy personnel”. The other
                    took it to mean “benefits to US. citizens”. As it
                    turned out, the Navy was looking for “benefits to
                    Navv as an institution”.




                                                         GAO/NSIAD-171   R USS Missouri
Enclosure                                                                Enclosure




     w          No Clarification Was Requested

            l   None of the applicants requested clarification of the
                June 5 letter or expressed concern about the
                additional requirements at the time. All responded
                to the letter.
                l   According to two applicants, the brevity of the
                    notification to them of the additional criteria, as
                    well as the short time frame to respond relative to
                    the more extensive and protracted previous
                    discussions on the financial/technical criteria, gave
                    them the mistaken impression that the
                    financial/technical criteria carried the most weight
                    and that these additional requirements were not
                    that significant.




                                          12             GAO/NSIAD-171   R USS Missouri
Enclosure                                                         Enclosure




     GM         Navy Conducts Evaluation

            l   In June/July 1996, Navy evaluation
                teams scored and ranked applications
                based on (1) financial and technical
                capabilities, (2) historical significance,
                and (3) public benefits to the Navy.
            l   Navy review group and higher level
                management essentially accepted
                evaluation teams’ results.
            l   Aug 21, 1996, the Secretary of Navy
                announced Hawaii as winner.




                                                GAO/NSIAD-171   R USS Missouri
Enclosure                                                     Enclosure




     GKI Three Evaluation Teams To Review
         Applications
            l   Public Affairs Benefits: Performed by the
                Naval Office of information
            l   Historical Significance: Performed by the
                Naval Historical Center
            l   Financial and Technical: Performed by
                Naval Sea Systems Command




                                  14         GAO/NSIAD-171   R USS Missouri
Enclosure                                                                                   Enclosure




     GM         Comments on Evaluation

            l   The Navy’s evaluation teams for historical significance and public
                affairs benefit had difficulty applying the criteria to the applications.
                l   According to the historical evaluation team, “None of the
                    applications were very strong on the historical or curatorial
                    aspects of the donation... .” The evaluator recommended “...if
                    possible, that each group be requested to provide more details on
                    this matter.”
                l   This team also commented that “no presentation includes any
                    historical site discussion; all simply make brief reference(s) to
                    history.” Historical site discussion carried the most weight among
                    the historical significance criteria. All applications received “low”
                    evaluation results.
                l   The Navy’s Information Office told us it could not evaluate
                    applications based on the initially established public affairs
                    benefits subcriteria. As a result, the team revised the evaluation
                    subcriteria to better fit the applications.




                                                 15                  GAO/NSIAD-171    R USS Missouri
Enclosure                                                                                Enclosure




    MO         Overall Evaluation Results (Highest
               Score Given Full Percentage Weight)
                         Public           Historical       Financial/
                         affairs        significance       technical          Total
                         (50%)             (25%)             (25%)

      Pearl
      Harbor                                  .25               .25              1 .o


      Bremerton            .39                .23a              .25               .87”

      San                   .41               .I2               .20               .73
      Francisco

      Long Beach            .37               .I1               .I7               .65




      a The Navy made a mathematical   error. The correct score should be 24 and the total .88.




                                                    16                GAO/NSIAD-171 R USS Missouri
Enclosure                                                            Enclosure




     GM       Public Affairs Scores (Highest Possible
              Points: IO)

      Subcriteria      Pearl                  San         Long
      (Weight)        Harbor   Bremerton   Francisco     Beach

      Message (0.5)     IO            8        7               6
      Guest
      Projections       IO            7        8               6
      (0.3)

      Demographics      6             5        8               IO
      (O-2)




                                 17            GAO/NSIAD-171       R USS Missouri
Enclosure                                                                                    Enclosure




    MO          Public Affairs--Weighted Scores (Raw
                Score Times Weight)
                               Pearl                              San
                              Harbor         Bremerton         Francisco           Long Beach

     Message                     5                4                3.5                 3
     Guest
     projections                 3               2.1               2.4                 1.8

     Demographics               1.2                1               1.6                 2

                                                  7.1              7.5                 6.8
     Weighted
     scorea                     30                                             1       .37
                                                                               L




      a Highest score given full weight; remainder given proportionate   weight.




                                                   18                    GAO/NSIAD-171 R USS Missouri
Enclosure                                                           Enclosure




    MI        Historical Significance Scores (Highest
              Possible Points: IO)

     Subcriteria        Pearl                  San          Long
     (Weight)          Harbor   Bremerton   Francisco      Beach
     Association o
     ship with site      2             9        2               4
     Kw
     Relationship of
     site With Navy      3             1        1               1
     (0 .5)
     Museum
     aspects (0.3)       IO            8        5               3




                                  19            GAO/NSIAD-171   R USS Missouri
Enclosure                                                                                    Enclosure




     w          Historical Significance--Weighted
                Scores (Raw Scores Times Weight)
                             Pearl                               San
                             Harbor           Bremerton       Francisco           Long Beach
      Association of
      ship with site            0.4               1.8               0.4               0.8
      Relationship of
      site with Navy             1.5              0.5               0.5               0.5

      Museum
      aspects                    3.0              2.4               I .5              0.9

      Total                      5.1a             4.7               2.4               2.2

      Weighted                   .25              .23b              .I2               .I1
      scorec



       a The Navy made an addition error.      The total should be 4.9.
       b As a result of the addition error, the weighted score should be .24.
       c Highest score given full weight; remainder given proportionate weight.




                                                   20                      GAOINSIAD-171    R USS Missouri
Enclosure                                                                                        Enclosure




     w              Financial/Technical Scores
                    (Highest Possible Points: IO)
                                            Pearl                          San         Long
                                            Harbor             Bremerton   Francisco   Beach

     Acquisition     costs                      10                  10          5           7
     Sources of income                          9                   10          a           6
     Operating/support         costs            10                  9           a           a
     Cash flow analysis                         6                   5           5           5
     Average       Financial    (.50)          8.75                8.5         6.5         6.5
     Towing plan acceptability                 Yes                 Yes         Yes         Yes
     Mooring plan                               9                   9           a           5
     Ship maintenance          plan             9                   9           7           6
     Environmental                              a                   9           a           5
    1 Average      Technical    (.50)   1      8.67        ]        9          7.67        5.33
     Total weighted score (avg                                                 7.0         5.9
     fin+avg tech)/2




                                                      21                   GAO/NSIAD-171   R USS Missouri
Enclosure                                                                       Enclosure




    MCI Recap of Key Events

            l       January 1995: USS Missouri stricken from the Naval Vessel
                    Register.
            l       March 1995 to March 1996: Five applications received
                    requesting donation of ship. Applications focused on meeting
                    Navy financial and technical requirements.
            l       February 1996 to May 1996: Navy developed evaluation
                    methodology and weighting scheme and adds historical
                    significance and public benefits to Navy as criteria. SECNAV
                    approved evaluation methodology and criteria on May 3, 1996.
                l   March 1996 to May 1996: Navy conducted preliminary financial
                    and technical reviews and forwarded questions to applicants
                    for clarification or additional information.
                l   June 3, 1996: Navy completed financial and technical
                    assessment. Four of the five applicants met minimum
                    requirements to advance to phase II scoring and ranking.




                                               22               GAO/NSIAD-171 R USS Missouri
Enclosure                                                                      Enclosure




     WCI Recap of Key Events

            l   June 5, 1996: Navy notified applicants by letter of added
                criteria and asked for additional information by June 21.
            l   June 14-21 ,I 996: Applicants responded to June 5 request.
            l   June-July 1996: NAVSEA scored applications on
                financial/technical requirements, Naval Historical Center
                scored applications on historical significance, and Naval Office
                of Information scored applications on public benefits to Navy.
            l   July 18-30, 1996: Evaluation teams briefed results to Navy
                review panel, and ASN(RDA). All essentially concurred with
                teams’ results.
            . August 1, 1996: SECNAV briefed on evaluation results.
            l   August 21,1996:   SECNAV announced      Hawaii as winner.
            l   September 1996: Navy debriefed applicants on results and
                weighting scheme.




                                            23               GAO/NSIAD-171   R USS Missouri
Enclosure                                                                     Enclosure




    GA0 Conclusion

            l   The Navy’s ship donation process is designed to provide the
                Secretary of the Navy with applicants that satisfies the Navy’s
                requirements for financial and technical capabilities. In the
                past, with one exception, only one application was received for
                each of 43 donations, and the qualified applicant received the
                donation.
            l   The Navy began the donation process for the USS Missouri in
                the same manner as prior donations, requesting financial and
                technical information from the applicants and working with
                them to make their applications satisfy the Navy’s financial and
                technical requirements. Subsequently, the Navy decided that,
                with respect to the USS Missouri, additional criteria were
                needed to assist the Secretary in making the donation decision
                among four of five applicants that met the Navy’s financial and
                technical requirements.




                                            24               GAO/NSlAD-171   R USS Missouri
Enclosure                                                                    Enclosure




    GAo Conclusion

        l   While the Navy’s donation process appears to have been
            impartially applied, the Navy did not do a good job in
            communicating its additional requirements to the applicants.
            Specifically, applicants were not told (1) what the relative
            importance of the evaluation criteria was in the process (the added
            criteria represented 75 percent of the donation award weight), (2)
            what the added evaluation criteria meant, and (3) how well already
            submitted applications met the added criteria (a procedure routinely
            used in the financial and technical evaluation process).
            Communication is particularly important because the Navy’s
            evaluation teams were told to base their scoring only on the
            information contained in the applications.
        l   As a result, Navy evaluation teams found the applications lacking in
            information when measured against the added criteria. According to
            some applicants, had they known that the additional criteria carried
            so much weight, they would have improved their applications




            (707255)




                                           25               GAO/NSIAD-171 R USS Missouri
.:.
Ordering    Information

The first copy of each GAO report and testimony is free.
Additional   copies are $2 each. Orders should be sent to the
following address, accompanied by a check or money order
made out to the Superintendent     of Documents, when
necessary. VISA and Mastercard      credit cards are accepted, also.
Orders for 100 or more copies to be mailed to a single address
are discounted 25 percent.

Orders by mail:

U.S. General Accounting Office
P.O. Box 6015
Gaithersburg, MD 20884-6015

or visit:

Room 1100
700 4th St. NW (corner of 4th and G Sts. NW)
U.S. General Accounting  Office
Washington, DC

Orders may also be placed by calling (202) 512-6000
or by using fax number (301) 258-4066, or TDD (301)        413-0006.

Each day, GAO issues a list of newly available reports and
testimony.    To receive facsimile copies of the daily Iist or any
list fi-om the past 30 days, please caIl(202)   512-6000 using a
touchtone phone. A recorded menu will provide information          on
how to obtain these Ii&s,

For information on how to access GAO reports on the INTERNET,
send an e-mail message with “info” in the body to:

info@www.gao.gov
United States
General Accounting  Office
Washington, D.C. 20548-0001


Official   Business
Penalty    for Private   Use $300

Address    Correction    Requested




                                     ‘.