oversight

Tactical Intelligence: Joint STARS Full-Rate Production Decision Was Premature and Risky

Published by the Government Accountability Office on 1997-04-25.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                  United States General Accounting Office

GAO               Report to Congressional Committees




April 1997
                  TACTICAL
                  INTELLIGENCE
                  Joint STARS Full-Rate
                  Production Decision
                  Was Premature and
                  Risky




GAO/NSIAD-97-68
             United States
GAO          General Accounting Office
             Washington, D.C. 20548

             National Security and
             International Affairs Division

             B-275858

             April 25, 1997

             Congressional Committees

             We have reviewed the Department of Defense’s (DOD) recent decision to
             commit to the full-rate production of the Joint Surveillance Target Attack
             Radar System (Joint STARS). More specifically, we analyzed whether (1) the
             system had demonstrated a level of maturity through testing to justify a
             full-rate production commitment, (2) DOD considered and resolved
             important cost and performance issues prior to making its decision, and
             (3) there are future actions that could reduce program risk. This review
             was performed under our basic legislative responsibility and we are
             addressing it to you because it falls under your committees’ jurisdiction.


             Joint STARS is a joint Air Force and Army wide-area surveillance and target
Background   attack radar system designed to detect, track, classify, and support the
             attack of moving and stationary ground targets. This $11 billion major
             defense acquisition program consists of air and ground segments—
             refurbished 707 aircraft (designated the E-8) equipped with radar,
             operation and control, data processing, and communications subsystems,
             together with ground stations equipped with communications and data
             processing subsystems.

             Low-rate initial production (LRIP)1 of the Joint STARS aircraft began in fiscal
             year 1993. In line with 10 U.S.C. 2399, DOD’s final decision to proceed
             beyond LRIP first required the DOD Director of Operational Test and
             Evaluation (DOT&E) to submit a report to Congress, referred to as the
             Beyond LRIP report, stating whether (1) the test and evaluation performed
             was adequate and (2) testing demonstrated that the system is effective and
             suitable for combat, that is, operationally effective and suitable.2

             The Joint STARS aircraft was scheduled to begin its initial operational test
             and evaluation—referred to as the Joint STARS multi-service operational



             1
              Low-rate initial production of systems is to produce the minimum quantity necessary to (1) provide
             production-configured or representative articles for operational test and evaluation, (2) establish an
             initial production base for the system, and (3) permit an orderly increase in the production rate for the
             system sufficient to lead to full-rate production upon the successful completion of operational test and
             evaluation.
             2
              Operational effectiveness refers to the ability of a system to accomplish its mission in the planned
             operational environment. Operational suitability is the degree to which a system can be placed
             satisfactorily in field use considering such factors as reliability and maintainability.



             Page 1                                        GAO/NSIAD-97-68 Joint STARS Production Decision
                   B-275858




                   test and evaluation3—in November 1995. That testing was delayed and
                   then changed because of the deployment of Joint STARS assets to the
                   European theater to support Operation Joint Endeavor in Bosnia. The Air
                   Force Operational Test and Evaluation Center (AFOTEC) and the U.S. Army
                   Operational Test and Evaluation Command conducted a combined
                   development and operational test of Joint STARS from July through
                   September 1995 and an operational evaluation of the system during
                   Operation Joint Endeavor from January through March 1996.

                   Two Air Force Joint STARS aircraft and 13 Army Joint STARS ground station
                   modules were deployed to support Operation Joint Endeavor and
                   operationally evaluated from January through March 1996. After analyzing
                   the data from the combined development and operational test and the
                   operational evaluation performed during Operation Joint Endeavor,
                   AFOTEC issued its Joint STARS multi-service operational test and evaluation
                   final report on June 14, 1996. DOT&E staff analyzed the same and additional
                   data and the Director issued his Beyond LRIP report to Congress on
                   September 20, 1996.4 On September 25, 1996, the Under Secretary of
                   Defense for Acquisition and Technology signed an acquisition decision
                   memorandum approving the Joint STARS program’s entry into full-rate
                   production with a total planned quantity of 19 aircraft.


                   The Joint Surveillance Target Attack Radar System’s performance during
Results in Brief   its combined development and operational test and the operational
                   evaluation done in Bosnia do not support a decision to commit the system
                   to full-rate production. The system’s operational effectiveness and
                   suitability were not demonstrated during the operational testing. For
                   example, the DOD Director of Operational Test and Evaluation could only
                   state that the system had demonstrated effectiveness for “operations other
                   than war” and found that the system “as tested is unsuitable.” He further
                   reported that only 18 (25 percent) of 71 performance criteria tested were
                   demonstrated met by the system and that further testing is required for the
                   remaining 53.



                   3
                    The multi-service operational test and evaluation was to consist of a combined development and
                   operational test and a dedicated operational test.
                   4
                    Our analysis focused principally on the details in the “Beyond LRIP” report rather than just the
                   conclusions in the letter transmitting that report to Congress. Those details provide a clearer picture of
                   Joint STARS’ performance. For example, while the letter states “as far as suitability is concerned, [the
                   system] did not meet its requirements in [Operation Joint Endeavor], a problem which would be
                   exacerbated in a higher intensity conflict,” the report states that Joint STARS “as tested is unsuitable”
                   and provides detailed examples of its suitability problems.



                   Page 2                                        GAO/NSIAD-97-68 Joint STARS Production Decision
B-275858




DOD’s decision to move Joint STARS into full-rate production was premature
and raised the program’s level of risk. The program could have continued
under LRIP until operational effectiveness and suitability for combat were
demonstrated and plans to address identified deficiencies and reduce
program costs were completed. Instead, DOD decided in favor of Joint
STARS full-rate production without the benefit of that information. During
the period that the full-rate production decision was being considered, the
Assistant to the President for National Security was promoting the sale of
the system to the North Atlantic Treaty Organization (NATO). In an
August 10, 1996, memorandum to the Secretaries of State, Defense, and
Commerce and to the Chairman of the Joint Chiefs of Staff, the Assistant
to the President for National Security Affairs stated that: “We have been
working through various military, diplomatic, and political channels to
secure NATO support for a Fall 1996 decision in principle by the Conference
of Armament Directors. . .to designate [Joint STARS] as NATO’s common
system.” A DOD official informed us that in November 1996, the NATO
armament directors delayed their decision on Joint STARS for a year.

Before DOD approved the full-rate production of Joint STARS, DOT&E
provided Congress with a Joint STARS Beyond LRIP report, as required by
law. The report clearly indicates that (1) further operational testing is
needed, (2) DOT&E could only declare effectiveness for operations other
than war, and (3) the system was unsuitable as tested. Having issued this
report, DOT&E is under no further obligation to report to Congress at the
Beyond LRIP report level of detail on the adequacy of the operational
testing or on whether the system has demonstrated effectiveness and
suitability for combat. However, DOD plans follow-on test and evaluation to
address the deficiencies identified during the earlier testing.

There is an opportunity not currently under consideration that could
reduce the Joint STARS’s program cost and result in an improved system.
Since the Joint STARS was approved for LRIP, the procurement cost
objective of the Air Force’s share of the Joint STARS has increased by about
$1 billion. This is primarily due to the fact that it is taking greater effort
and more resources to refurbish the 25-30 year old 707 airframes than
previously anticipated. The estimated cost of procuring, refurbishing, and
modifying each used 707 airframe to receive the system’s electronics is
now about $110 million. As early as 1992, the Boeing Company proposed
putting the system on newer Boeing 767-200 Extended Range aircraft, but
this proposal was not accepted at that time as cost-effective. Given the
current 707 airframe procurement, refurbishment, and modification cost
and a 1996 price for a commercial version Boeing 767-200 Extended Range



Page 3                            GAO/NSIAD-97-68 Joint STARS Production Decision
                            B-275858




                            aircraft of between $82 million and $93 million, it may now be more
                            cost-effective for the Air Force to buy that or some other new, more
                            capable aircraft. Such an aircraft could provide a longer life, greater room
                            for growth, greater flight range, greater fuel efficiency, higher operational
                            availability, and lower program life-cycle costs.


                            LRIP of the Joint STARS aircraft began in fiscal year 1993. By statute, 10
Test Results Do Not         U.S.C. 2399, the “Secretary of Defense shall provide that a major defense
Support Full-Rate           acquisition program may not proceed beyond low-rate initial production
Production                  until initial operational test and evaluation of the program is completed.”

                            Operational test and evaluation is the primary means of assessing weapon
                            system performance in a combat-representative environment. It is defined
                            as the (1) field test, conducted under realistic combat conditions, to
                            determine an item’s effectiveness and suitability for use in combat by
                            typical military users and (2) evaluation of the results of such a test. If
                            used effectively, operational test and evaluation is a key internal control
                            measure to ensure that decisionmakers have objective information
                            available on a weapon system’s performance, thereby minimizing risks of
                            procuring costly and ineffective systems.

                            Joint STARS was moved from low-rate to full-rate production even though
                            (1) it performed poorly during both the combined development and
                            operational test and the operational evaluation in Bosnia, (2) excessive
                            contractor effort was needed to support Operation Joint Endeavor, (3) the
                            suitability and sustainability of the system is questionable since it uses
                            refurbished 25-30 year old airframes, and (4) operational software is
                            considered significantly immature.


Test Results Were           In DOT&E’s Beyond LRIP report, the DOT&E stated that Joint STARS had only
Reported as Disappointing   demonstrated effectiveness for operations other than war. The report
                            indicated that of three critical operational issues5 to judge effectiveness,
                            only one had been demonstrated as met “. . . with limitations.” Those
                            critical operational issues related to (1) performance of the tactical
                            battlefield surveillance mission, that is, surveillance—“met with


                            5
                             DOD regulation 5000.2-R states that “critical operational issues are the operational effectiveness and
                            operational suitability issues (not parameters, objectives or thresholds) that must be examined in
                            operational test and evaluation to evaluate/assess the system’s capability to perform its mission.” It
                            also states that “if every critical operational issue is resolved favorably, the system should be
                            operationally effective and operationally suitable when employed in its intended environment by
                            typical users.”



                            Page 4                                       GAO/NSIAD-97-68 Joint STARS Production Decision
B-275858




limitations”; (2) support of the execution of attacks against detected
targets, that is, target attack support; and (3) the provision of information
to support battlefield management and target selection, that is, battle
management. The effectiveness critical operational issues were judged
based on seven supporting measures. In its report to Congress, DOT&E
listed four of those measures of effectiveness as “not met” during the
system’s combined development and operational test and did not list any
as having been demonstrated during the Operation Joint Endeavor
operational evaluation.

However, of greater concern, according to DOT&E, is the fact that the
system did not meet its overall suitability requirements during Operation
Joint Endeavor. In his executive summary, the Director stated that most of
DOT&E’s Joint STARS concerns relate to operational suitability. He went on
to say that

“In the current configuration, the [Joint STARS] aircraft has not demonstrated the ability to
operate at the required maximum altitude; adequate tactics, techniques, or procedures to
integrate [Joint STARS] into operational theaters have not been developed; [Joint STARS]
exceeded the break rate and failed the mission reliability rate during [Operation Joint
Endeavor]. During [Operation Joint Endeavor], [Joint STARS] did not achieve the effective
time-on-station requirement.”


He concluded that without corrective actions, “[Joint STARS] would not be
suitable in higher intensity conflict” and later in the report judged that the
system “as tested is unsuitable.”

Analysis of DOT&E’s Beyond LRIP report indicates that not only did Joint
STARS have disappointing test results but also that extensive follow-on
operational testing of Joint STARS is needed. In its Beyond LRIP report,
DOT&E presented a table that reported its findings of the combined
development and operational test and Joint STARS Operation Joint
Endeavor operational evaluation and indicates where further testing is
required. Our analysis of that table indicates that at most only 25 of 71 test
criteria could be judged met. DOT&E considers 18 of those 25 to require no
further testing, that is, DOT&E judges them clearly met. However, our
analysis also indicates that 19 test criteria were clearly not met and that as
many as 26 might not have been met. Twenty-seven of the criteria could
not be determined in either the combined development and operational
test or the Operational Joint Endeavor operational evaluation. Of the 71
Joint STARS operational test and evaluation criteria listed, DOT&E indicates
that 53, or about 75 percent, require further testing.




Page 5                                   GAO/NSIAD-97-68 Joint STARS Production Decision
                         B-275858




                         In addition to the above, DOT&E also noted that there were several
                         operational features present during Joint STARS Operation Joint Endeavor
                         deployment that were essential to its mission accomplishment but were
                         not included in the recent production decision. It provided two specific
                         examples—satellite communications and a deployable ground support
                         station. DOT&E believes these features “will be a necessary part of the
                         production decision to achieve a capable [Joint STARS] system.” It also
                         noted the need for other features—moving target indicator clutter
                         suppression, communications improvements, terrain masking tools for
                         ground station module operators, and linkage to operational theater
                         intelligence networks. Since at least two of the features present during
                         Operation Joint Endeavor were “essential” to its mission accomplishment
                         have already been developed, and may be needed “to achieve a capable
                         Joint STARS system,” those features should also be tested during the
                         planned Joint STARS follow-on test and evaluation.


Significant Contractor   The degree of contractor involvement required during the operational
Involvement              evaluation indicates increased program risk and makes the reported Joint
                         STARS performance appear better than it would have otherwise. The
                         multi-service operational test and evaluation plan, in discussing contractor
                         involvement during the testing, stated:

                         “[Multi-service operational test and evaluation] must yield the most credible and objective
                         results possible. All facets of the test effort must operate under the rules that support total
                         objectivity and prevents improper data manipulation.”


                         The test plan also states that interim contractor support “will be limited to
                         perform ground maintenance only; no in-flight support.” Regarding the
                         Army’s ground station modules, it states that “the Army maintenance
                         concept does not call for [contractor involvement] at any level . . . .”

                         However, during Operation Joint Endeavor there was significant
                         contractor support of the 2 aircraft and 13 ground station modules
                         deployed. According to the AFOTEC report,

                         “Approximately 80 contractors were deployed to support the E-8C. However, three or four
                         [contractor] systems engineers flew on each flight to ensure they could provide system
                         stability and troubleshooting expertise during missions. Additionally, three or four
                         [contractor] software engineers were on the ground full time, researching and developing
                         fixes to software problems identified during the deployment.”




                         Page 6                                    GAO/NSIAD-97-68 Joint STARS Production Decision
B-275858




AFOTEC also reported that “Each of the [ground station modules] had one
contractor representative on site and on call with additional help available
as necessary. Five contractor representatives remained at [Rhein-Main Air
Base] and functioned as a depot.” The AFOTEC report stated that the “test
director agreed to contractor participation in the [Operational Evaluation]
to a greater extent [than] permitted under US Public Law, Title 10, Section
2399.” [Emphasis added.]

When we formally expressed our concerns about the significant contractor
involvement in Operation Joint Endeavor,6 DOD did not directly
acknowledge that contractors were utilized beyond the constraints of the
law governing operational test and evaluations. It stated that “were this
solely an [initial operational test and evaluation], contractors would not
have been utilized beyond the constraints of 10 U.S.C. §2399,” and noted
that the contractors were involved in the Joint STARS operation to support
the mission. It further stated that employing Joint STARS in Operation Joint
Endeavor “allowed the system to be operated and tested at a greater
operational tempo than the system would have undergone in traditional
testing.” DOD also stated that “because of the developmental nature of the
aircraft, we needed to have more contractor personnel involved than we
would otherwise have had.”

It is understandable that DOD wanted to provide the best support possible
in Operation Joint Endeavor. However, such significant contractor use
neither supports a conclusion that the system is operationally effective or
suitable for combat, nor is it indicative of a level of system maturity that
justifies full-rate production.

Joint STARS failure to meet its maintainability criteria during an operation
less demanding than combat, even with such significant contractor
involvement beyond that planned for in combat, also raises the question of
the Air Force’s ability to develop a cost-effective maintenance plan for the
system. This issue is recognized in the Under Secretary’s acquisition
decision memorandum approving Joint STARS full-rate production. In that
memorandum, the Under Secretary called for the Air Force to fully
examine Joint STARS affordability, sustainability, and life-cycle costs,
including the scope of contractor support.




6
 Joint STARS Production Decision (GAO/NSIAD-96-242R, Sept. 20, 1996).



Page 7                                     GAO/NSIAD-97-68 Joint STARS Production Decision
                              B-275858




System Sustainability and     In discussing the sustainability of the Joint STARS system, DOT&E noted in its
Suitability Is Questionable   Beyond LRIP report that “It is not yet known what the operational tempo
                              will be for Joint STARS.” It concluded that

                              “If it is determined that the system will be operated at rates similar to AWACS [Airborne
                              Warning And Control System], it is questionable whether the [Joint STARS aircraft] can be
                              sustained over time. Airframe problems have already been experienced on the existing
                              [Joint STARS airframes], including a hydraulics failure and a cracked strut in the fuselage
                              between the wings.”


                              In discussing the Joint STARS aircraft engines, DOT&E noted that they “are
                              1950s technology and may not be reliable” and cited AFOTEC’s reporting
                              that engine failures were among the principal reasons that the aircraft
                              failed to meet the break rate criteria7 during Operation Joint Endeavor.

                              In discussing Joint STARS suitability, DOT&E also noted that the limited
                              power of the engines “made it difficult to reach the aircraft’s normal
                              operating altitude of 36,000 feet, much less the 42,000 feet maximum
                              altitude it is required to reach.” It further reported that during Operation
                              Joint Endeavor, the aircraft required approximately 11,000 feet of runway
                              when taking off with 140,000 pounds of fuel and concluded that “this may
                              pose a significant challenge to operational commanders because the
                              [North Atlantic Treaty Organization] standard runway length is 8,000 feet.”
                              It noted that operational challenges would be great in other theaters as
                              well and cited Korea as an example. It reported that Joint STARS

                              “. . . would face operational challenges taking off from five runways in Korea, each
                              approximately 9,000 feet long. Operations out of Korea would likely require taking off with
                              less fuel and subsequent aerial refueling or shortening the time on station.”


                              Another area of Joint STARS suitability concern is the system’s growth
                              potential. DOT&E has reported that it is not clear that the remanufactured
                              707 platforms will be capable of incorporating all of the planned upgrades,
                              noting that the airframe limits the system’s growth potential both in weight
                              and volume. It reported that as the current mission equipment already fills
                              much of the fuselage, there is little room for expansion. DOT&E also noted
                              that increasing the payload weight would require longer takeoff runways
                              or taking off with less fuel, thus increasing the aerial refueling requirement
                              or decreasing mission duration.



                              7
                                Break rate is one measure of system suitability. For Joint STARS operational testing it was defined as
                              the percentage of missions flown for a specific period of time in which a previously mission-capable
                              essential subsystem was inoperable once the aircraft landed.



                              Page 8                                        GAO/NSIAD-97-68 Joint STARS Production Decision
                           B-275858




                           DOT&E also noted that the system’s current computers limited its growth
                           potential due to their having very little reserve processor time or memory.
                           It stated that the Air Force requires that no more than 50 percent of central
                           processor unit cycles or memory be utilized by a new system. DOT&E
                           reported that “None of the E-8C computer subsystems meet these
                           requirements.” It provided an example of the problem, stating that “the
                           memory reserve of the operator workstations still does not meet the
                           requirement, even after being increased from 128 megabytes to
                           512 megabytes just prior to [Operation Joint Endeavor].” This assessment
                           is another indicator of the program’s elevated risk. As DOT&E noted “Future
                           software enhancements and modifications may require significant
                           hardware upgrades. . . .”


Operational Software       The AFOTEC report specifically pointed to the lack of maturity in Joint STARS
Rated Significantly        software. For example, AFOTEC reported that
Immature
                       •   “during Joint STARS [multi-service operational test and evaluation],
                           software deficiencies were noted on every E-8C subsystem;”
                       •   the software “does not adequately support [the] operator in executing the
                           mission;” and
                       •   “Joint STARS software does not show the expected maturity trends of a
                           system at the end of development.”

                           In discussing Joint STARS software maturity, DOD advised us that the AFOTEC
                           report judged the system overall operationally effective and suitable.
                           Specifically, in reference to software problems, DOD stated that “the
                           majority of software faults that occurred during Operation Joint Endeavor
                           were resolved while airborne in less than 10 minutes.” However, both
                           AFOTEC and DOT&E had some critical concerns regarding how Joint STARS
                           software functioned. For example, according to AFOTEC, the “Joint STARS
                           software is immature and significantly impedes the system’s reliability and
                           effectiveness,” and according to DOT&E

                       •   “Immature software was clearly a problem during [Operation Joint
                           Endeavor]. . .”
                       •   “. . .the prime contractor had to be called in to assist and correct 69
                           software-specific problems during the 41 E-8C missions . . . .an average of
                           1.4 critical failures per flight. . .”
                       •   “Communications control was lost on 69 percent of the flights.”
                       •   “The system management and control processor failed and had to be
                           manually reset on half of the flights.”



                           Page 9                           GAO/NSIAD-97-68 Joint STARS Production Decision
                       B-275858




                       DOD  has stated that the Air Force “plans several actions to mature the
                       software and provide the required support resources” and that “an interim
                       software release in April 1997 will correct some software deficiencies
                       identified during the operational evaluation.” DOD also noted that software
                       updates will be loaded each year thereafter and that software changes are
                       easily incorporated. How easily these software changes are incorporated
                       remains to be seen because much of this software, according to AFOTEC
                       and DOT&E, is poorly documented. For example, AFOTEC has reported that
                       there are 395 deficiency reports open against the Joint STARS program, 318
                       of which are software related. DOT&E also stated that the more than 750,000
                       lines of Joint STARS software code are “poorly documented” and later
                       commented that “Software problems with the communications and
                       navigation systems were never fully corrected, even after extensive efforts
                       by the system contractor.”8 These facts in combination with DOD’s
                       comments raise the serious question as to which software deficiencies are
                       to be addressed in the planned April software update.



                       There is an opportunity not currently under consideration that could
Alternative Aircraft   reduce the Joint STARS program cost and result in an improved system.
Should Be Considered   Since the Joint STARS was approved for LRIP, the procurement cost
                       objective of the Air Force’s share of the Joint STARS has increased by about
                       $1 billion. Program costs escalated from approximately $5.2 billion to
                       approximately $6.2 billion in then-year dollars. A DOD official informed us
                       that of the $1 billion cost growth, $760 million is attributed to the
                       increased cost to buy, refurbish, and modify the used 707 airframes to
                       receive the Joint STARS electronics. The remaining cost growth is attributed
                       to other support requirements and growth in required spare parts.

                       At least as early as 1992, the Boeing Company proposed putting Joint STARS
                       on newer Boeing 767-200 Extended Range aircraft, but this proposal was
                       not accepted as cost-effective. According to the 1996 Boeing price list, the
                       commercial version of this aircraft can be bought for between $82 million
                       and $93 million depending on options chosen (this is flyaway cost—the
                       cost of a plane ready to be flown in its intended use). Furthermore, the
                       flyaway cost of a commercial Boeing 757, which a Boeing representative
                       informed us is in many respects more comparable to the 707s being used,
                       is listed at between $61 million to $68 million. The actual cost of procuring
                       either of these aircraft could be lowered by volume discounts and by the

                       8
                        AFOTEC stated that “Safety of flight is jeopardized due to invalid navigational commands sent from
                       the FMS-800 [flight management system] and conflicting and/or invalid navigational data displayed to
                       the flight crew.”



                       Page 10                                     GAO/NSIAD-97-68 Joint STARS Production Decision
B-275858




cost of the commercial amenities not required. On the other hand, these
aircraft would require modifications to receive Joint STARS equipment,
which would raise their cost.

DOD  informed us that the cost of procuring, refurbishing, and modifying the
current 707 aircraft to receive Joint STARS equipment is now estimated to
be about $110 million per airframe. The cost of procuring and preparing
new aircraft might be comparable or even less than the current cost. In
addition, the Air Force would acquire a new platform that could have
(1) greater room for growth (both volume and weight), (2) take off
capability from a shorter runway, (3) greater time-on-station capability,
(4) significantly improved fuel efficiency, (5) extended aircraft life over
the 707 currently used, and (6) reduced operational and support cost.

In commenting on a draft of this report, DOD stated that it considered
alternatives to the current air platform, both before LRIP started and at the
full-rate production decision point. It also stated that the cost of moving
the Joint STARS mission to an alternative platform would outweigh the
benefits. We note, however, that at a meeting with DOD and service officials
to discuss that draft, we asked about the reported DOD and service
analyses. One Air Force official stated that the Air Force’s platform choice
was not revisited prior to the full-rate production decision. None of the
other 13 DOD and service officials present objected to that statement.
Furthermore, when we asked for copies of the air platform analyses that
were done in support of either the low-rate or the full-rate production
decision, DOD was unable to supply those analyses. Finally, DOD officials
have informed us that a Command, Control, Communications, Computers,
Intelligence, Surveillance, and Reconnaissance Mission Assessment9 has
been performed that indicates that the Air Force could acquire a more
effective system while saving $3 billion through the year 2010 by moving
the Joint STARS mission to either a business jet or an unmanned aerial
vehicle following the procurement of the twelfth current version Joint
STARS aircraft.




9
 The results of this mission assessment are being considered in the Quadrennial Defense Review, a
process in which DOD is taking a fundamental look at its strategy, force structure, modernization,
infrastructure, and readiness to meet future mission requirements.



Page 11                                     GAO/NSIAD-97-68 Joint STARS Production Decision
                  B-275858




                  We have previously informed DOD of our concerns about the decision to
Production        move to full-rate production in spite of the numerous testing deficiencies
Commitment        reported by both AFOTEC and DOT&E. DOD responded that in making the
Unnecessary and   decision to move to full-rate production, it “considered the test reports
                  (both the services’ and the Director, Operational Test and Evaluation’s),
Risky             the plans to address the deficiencies identified during developmental and
                  operational testing, cost estimates, operational requirements, and other
                  program information.”

                  Although DOD believes that “none of the deficiencies identified are of a
                  scope that warrants interrupting production,” the production decision
                  memorandum clearly reflects a recognition that this program carries
                  significant risk. In his memorandum, the Under Secretary of Defense for
                  Acquisition and Technology directed (1) an update of the Joint STARS Test
                  and Evaluation Master Plan to “address multi-service [operational test and
                  evaluation] deficiencies (regression testing);” (2) acceleration of the
                  objective and threshold dates for the planned Follow-on Operational Test
                  and Evaluation; and (3) the Air Force to “fully examine [Joint STARS]
                  affordability, sustainability, and life cycle costs10 including the scope of
                  contractor use for field-level system support.”

                  Factors other than system performance may have influenced the decision
                  to move Joint STARS forward into full-rate production. DOD’s full-rate
                  production decision for this program occurred during the same time frame
                  the Joint STARS system was actively being promoted as the U.S.
                  government’s candidate for meeting NATO’s military requirement for a
                  ground surveillance system. For example, in an August 10, 1996,
                  memorandum to the Secretaries of State, Defense, and Commerce and to
                  the Chairman of the Joint Chiefs of Staff, the Assistant to the President for
                  National Security Affairs stated that “We have been working through
                  various military, diplomatic and political channels to secure NATO support
                  for a Fall 1996 decision in principle by the Conference of Armament
                  Directors . . . to designate [Joint STARS] as NATO’s common system.” He
                  went on to state that

                  “I am writing to be sure you know that the President is personally committed to [Joint
                  STARS], has engaged Chancellor Kohl on this issue and will continue his personal
                  involvement with key allies to ensure our goal is achieved. I would ask that you underscore
                  your personal support for our collective efforts on behalf of [Joint STARS] when you meet
                  with your NATO and European counterparts.”




                  10
                    We believe this analysis should include examining the use of new airframes.



                  Page 12                                      GAO/NSIAD-97-68 Joint STARS Production Decision
                        B-275858




                        Notwithstanding DOD’s September 1996 commitment to full-rate Joint STARS
                        production, a DOD official informed us that the NATO armament directors in
                        their November 1996 meeting delayed for 1 year any decision on
                        designating Joint STARS as NATO’s common system or pursuing an alternate
                        system to be developed.


                        In the process of moving the Joint STARS program forward into full-rate
Follow-on Operational   production, DOD produced a Beyond LRIP report for Congress and thus
Test and Evaluation     moved past a key congressional reporting requirement that serves as an
Planned                 important risk management mechanism. The Beyond LRIP report to
                        Congress that is required before major defense acquisition programs can
                        proceed into full-rate production serves to inform Congress of the
                        adequacy of the operational testing done on the system and to provide it
                        with a determination of whether the system has demonstrated
                        effectiveness and suitability. Having issued this report, DOT&E is under no
                        further obligation to report to Congress at the Beyond LRIP report level of
                        detail on the adequacy of the operational testing or on whether the system
                        has demonstrated effectiveness and suitability for combat. However, DOD
                        plans follow-on test and evaluation of the system to address the
                        deficiencies identified during the system’s earlier testing.

                        On September 20, 1996, DOT&E sent to Congress a Joint STARS “Beyond LRIP”
                        report that (1) clearly indicates that further operational testing is needed,
                        (2) could only declare effectiveness for operations other than war, and
                        (3) stated that Joint STARS is unsuitable as tested. On September 25, 1996,
                        DOD approved the full-rate production of Joint STARS. In the acquisition
                        memorandum approving Joint STARS full-rate production, the Under
                        Secretary of Defense for Acquisition and Technology called for an
                        accelerated follow-on operational test and evaluation of Joint STARS that is
                        to address the deficiencies identified in the initial operational test and
                        evaluation DOT&E reported on in the Beyond LRIP report to Congress. The
                        planned follow-on operational test and evaluation will provide an
                        opportunity to judge the Joint STARS program’s progress in resolving the
                        issues identified in earlier testing.


                        Notwithstanding any concurrent efforts to have Joint STARS designated as a
Conclusions             NATO common system, Joint STARS test performance and the clearly
                        unresolved questions about its operational suitability and affordability
                        should have, in our opinion, caused DOD to delay the full-rate production
                        decision until (1) the system had, through the planned follow-on



                        Page 13                          GAO/NSIAD-97-68 Joint STARS Production Decision
                     B-275858




                     operational test and evaluation, demonstrated operational effectiveness
                     and suitability; (2) the Air Force had completed an updated analysis of
                     alternatives for the Joint STARS to address the identified aircraft suitability
                     and cost issues; and (3) the Air Force had developed an analysis to
                     determine whether a cost-effective maintenance concept could be
                     designed for the system. Furthermore, as they were judged “essential” to
                     mission accomplishment and needed “to achieve a capable Joint STARS
                     system,” the satellite communications and deployable ground support
                     station features (present, but untested, during Operation Joint Endeavor)
                     should also be tested during the planned Joint STARS follow-on operational
                     test and evaluation.

                     Concerns of the magnitude discussed in this report are not indicative of a
                     system ready for full-rate production. The program should have continued
                     under LRIP until the issues identified by AFOTEC and DOT&E were resolved
                     and the system was shown to be effective and suitable for combat.
                     Furthermore, the recent cost growth related to refurbishing and modifying
                     the old airframes being used for Joint STARS and questions regarding the
                     suitability of those platforms indicate an opportunity to reduce the
                     program’s cost and improve the systems acquired. We believe, therefore,
                     that an updated study of the cost effectiveness of placing Joint STARS on
                     new, more capable aircraft is warranted.


                     We recommend that the Secretary of Defense direct the Air Force to
Recommendation       perform an analysis of possible alternatives to the current Joint STARS air
                     platform, to include placing this system on a new airframe.


                     Because of (1) DOD’s decision to commit to full-rate production in the face
Matters for          of the test results discussed in this report and (2) its subsequent decision
Congressional        to do additional tests while in production to address previous test
Consideration        deficiencies, we are convinced that DOD plans to proceed with the
                     program. However, if Congress agrees that there is unnecessarily high risk
                     in this program and believes the risk should be reduced, it may wish to
                     require that:

                 •   The Air Force obtain DOT&E approval of a revised test and evaluation
                     master plan (and all plans for the tests called for in that master plan) for
                     follow-on operational testing to include adequate coverage of gaps left by
                     prior testing and include testing of any added features considered part of




                     Page 14                            GAO/NSIAD-97-68 Joint STARS Production Decision
                         B-275858




                         the standard production configuration and that DOT&E considers key
                         system components.
                     •   DOT&E provide a follow-on test and evaluation report to Congress
                         evaluating the adequacy of all testing performed to judge operational
                         effectiveness and suitability for combat and a definitive statement stating
                         whether the system has demonstrated operational effectiveness and
                         suitability.
                     •   DOD develop and provide Congress an analysis of alternatives report on the
                         Joint STARS air platform that considers the suitability of the current
                         platform and other cost-effective alternatives, and the life-cycle costs of
                         the current platform and best alternatives.


                         In commenting on a draft of this report, DOD disagreed with our
Agency Comments          recommendation that the Air Force be directed to perform an analysis of
and Our Evaluation       possible alternatives to the current Joint STARS air platform. It also
                         disagreed with our suggestion that Congress may wish to require DOD to
                         develop and provide Congress a report on that analysis. DOD stated that
                         alternative platforms were considered prior to both the start of LRIP and
                         the full-rate production decision. DOD stated that based on (1) the fact that
                         over half the fleet is already in the remanufacturing process or delivered to
                         the user; (2) the large nonrecurring costs that would be associated with
                         moving the Joint STARS mission to a different platform; (3) the additional
                         cost to operate and maintain a split fleet of Joint STARS airframes; and
                         (4) the expected 4-year gap in deliveries, such a strategy would force the
                         costs of moving the Joint STARS mission to a different platform outweigh
                         the benefits.

                         DOD’s comment about having previously considered alternative platforms
                         is inconsistent with the information we developed during our review and
                         with Air Force comments provided at our exit conference. In an effort to
                         reconcile this inconsistency, we requested copies of the prior analyses of
                         alternative platforms, but DOD was not able to provide them. DOD’s
                         statement that the costs of moving the Joint STARS mission to another
                         platform would outweigh the benefits contradicts Command, Control,
                         Communications, Computers, Intelligence, Surveillance, and
                         Reconnaissance Mission Assessment briefings given the Quadrennial
                         Defense Review. Those briefings recommend (1) limiting Joint STARS
                         production to 12 aircraft, (2) moving the Joint STARS mission to either
                         corporate jets or Unmanned Aerial Vehicles, and (3) phasing out Joint
                         STARS 707 variants as quickly as the new platform acquisitions will allow.
                         According to those briefings, implementation of this recommendation



                         Page 15                          GAO/NSIAD-97-68 Joint STARS Production Decision
B-275858




could result in a more effective system and save over $3 billion through
fiscal year 2010. We believe that the issue clearly warrants further
consideration. Furthermore, given DOD’s resistance to the concept, we are
more convinced of the merits of our suggestion that Congress might wish
to require a report on such an analysis.

In commenting on our draft report, DOD also indicated that congressional
direction was unneeded on our suggestions that Congress might wish to
require (1) DOT&E approval of a revised test and evaluation master plan for
the planned Joint STARS follow-on operational test and evaluation and
(2) DOT&E to provide Congress with a follow-on operational test and
evaluation report on the adequacy of Joint STARS testing and stating
whether Joint STARS has demonstrated operational effectiveness and
suitability. DOD stated that congressional direction on the first point was
unneeded because the Joint STARS full-rate production decision
memorandum required that the test and evaluation master plan be updated
for Office of the Secretary of Defense approval and current DOD policy is
that DOT&E will review, approve, and report on oversight systems in
follow-on operational test and evaluation. DOD also stated that
congressional direction on the second point is unneeded because DOT&E
has retained Joint STARS on its list of programs for oversight and is to
report on the system in its annual report to Congress as appropriate.

DOD’s response did not directly address our point since, as DOD pointed out,
the acquisition decision memorandum that approved full-rate production
required Office of the Secretary of Defense approval, not DOT&E approval,
of the follow-on operational test and evaluation master plan. During the
course of our review, DOD officials informed us that there was significant
disagreement between the Air Force and DOT&E as to what follow-on
testing was needed. It was indicated that the issue would probably have to
be resolved at higher levels within the department, an indication of greater
flexibility than DOD implies. Furthermore, while DOD stated there were
some improvements and enhancements “that could benefit the warfighter”
and acknowledged that those features were not tested, it did not respond
to our comments that DOT&E judged those features “essential” to mission
accomplishment or commit to their operational test and evaluation. Given
these facts, we have not only maintained our suggestion that Congress
may wish to require the Air Force to obtain DOT&E approval of a revised
test and evaluation master plan, but also strengthened it to include DOT&E
approval of supporting test plans.




Page 16                          GAO/NSIAD-97-68 Joint STARS Production Decision
              B-275858




              In its response to our suggestion that Congress may wish to require that
              DOT&E provide it a detailed, follow-on test and evaluation report, DOD states
              congressional direction is unnecessary as DOT&E will report on the system,
              among many others, in its annual report to Congress. DOD’s comment fails
              to recognize, however, that we are suggesting that, given the already
              reported test results, Congress may wish a more detailed report outlining
              the adequacy of and the system’s performance during follow-on
              operational testing to help in its oversight and provide it assurance that
              the system’s problems have been substantially resolved. Given that
              (1) Congress felt such reporting to be beneficial enough to require it
              before a system can proceed beyond LRIP and (2) the fact that DOT&E, in the
              required report provided for Joint STARS, could not certify effectiveness for
              war and found the system unsuitable as tested, we continue to believe that
              Congress may wish to require a similar report based on the follow-on
              operational test and evaluation planned.

              DOD’s comments are reprinted in their entirety in appendix I, along with
              our evaluation.


              To determine whether Joint STARS test performance indicates a maturity
Scope and     justifying full-rate production, we interviewed officials and reviewed
Methodology   documents in Washington, D.C., from the DOD Office of the Director of
              Operational Test and Evaluation and the Joint STARS Integrated Product
              Team. We reviewed the Air Force Operational Test and Evaluation
              Center’s multi-service operational test and evaluation plan and its final
              report on that testing and the DOD Director of Operational Test and
              Evaluation’s Beyond LRIP report. To determine whether DOD considered
              and resolved important cost and performance issues prior to making its
              full-rate production decision, we reviewed Joint STARS program budget
              documents and program-related memoranda issued by the Under
              Secretary of Defense for Acquisition and Technology. To determine
              whether it is possible that a more useful operational test and evaluation
              report can be provided Congress, we reviewed the statute governing
              operational testing and evaluation, examined DOT&E’s Beyond LRIP report,
              and considered other relevant program information. We considered and
              incorporated where appropriate DOD’s response to our September 20, 1996,
              letter of inquiry11 and its response to a draft of this report. We conducted
              this review from October 1996 through April 1997 in accordance with
              generally accepted government auditing standards.


              11
                Joint STARS Production Decision (GAO/NSIAD-96-242R, Sept. 20, 1996).



              Page 17                                    GAO/NSIAD-97-68 Joint STARS Production Decision
B-275858




We are sending copies of this letter to other appropriate congressional
committees; the Director, Office of Management and Budget; and the
Secretaries of Defense, the Army, and the Air Force. Copies will also be
made available to others upon request.

If you or your staff have any questions, please contact me, Mr. Charles F.
Rey, Assistant Director, or Mr. Bruce Thomas, Evaluator-in-Charge, at
(202) 512-4841.




Thomas J. Schulz
Associate Director,
Defense Acquisitions Issues




Page 18                          GAO/NSIAD-97-68 Joint STARS Production Decision
B-275858




List of Congressional Committees

The Honorable Strom Thurmond
Chairman
The Honorable Carl Levin
Ranking Minority Member
Committee on Armed Services
United States Senate

The Honorable Ted Stevens
Chairman
The Honorable Daniel K. Inouye
Ranking Minority Member
Subcommittee on Defense
Committee on Appropriations
United States Senate

The Honorable Floyd D. Spence
Chairman
The Honorable Ronald V. Dellums
Ranking Minority Member
Committee on National Security
House of Representatives

The Honorable C.W. Bill Young
Chairman
The Honorable John P. Murtha
Ranking Minority Member
Subcommittee on National Security
Committee on Appropriations
House of Representatives

The Honorable Richard C. Shelby
Chairman
The Honorable J. Robert Kerrey
Vice Chairman
Select Committee on Intelligence
United States Senate

The Honorable Porter J. Goss
Chairman
The Honorable Norman D. Dicks
Ranking Minority Member


Page 19                          GAO/NSIAD-97-68 Joint STARS Production Decision
B-275858




Select Committee on Intelligence
House of Representatives




Page 20                        GAO/NSIAD-97-68 Joint STARS Production Decision
Page 21   GAO/NSIAD-97-68 Joint STARS Production Decision
Appendix I

Comments From the Department of Defense


Note: GAO comments
supplementing those in the
report text appear at the
end of this appendix.




See pp. 10-11 & 15-16.




See comment 1.



See pp. 10-11 & 15-16.




                             Page 22   GAO/NSIAD-97-68 Joint STARS Production Decision
                 Appendix I
                 Comments From the Department of Defense




Now on p. 1.

Now on p. 2.




See comment 2.


Now on p. 2.

Now on p. 3.




See comment 3.




Now on p. 3.




                 Page 23                            GAO/NSIAD-97-68 Joint STARS Production Decision
                 Appendix I
                 Comments From the Department of Defense




See comment 4.




Now on p. 4.




Now on p. 4.




See comment 5.


See comment 6.




                 Page 24                            GAO/NSIAD-97-68 Joint STARS Production Decision
                 Appendix I
                 Comments From the Department of Defense




See comment 3.




See comment 7.




See comment 8.




See comment 8.




                 Page 25                            GAO/NSIAD-97-68 Joint STARS Production Decision
                  Appendix I
                  Comments From the Department of Defense




See comment 8.




See comment 8.




See comment 4.




Now on pp. 4-6.



See comment 3.




                  Page 26                            GAO/NSIAD-97-68 Joint STARS Production Decision
                  Appendix I
                  Comments From the Department of Defense




Now on p. 6.




See comment 9.




Now on pp. 6-7.




See comment 10.




Now on p. 7.




See comment 7.




                  Page 27                            GAO/NSIAD-97-68 Joint STARS Production Decision
                  Appendix I
                  Comments From the Department of Defense




Now on p. 7.




See comment 11.




See comment 10.




See comment 6.



Now on p. 8.




See comment 12.




See comment 11.


See comment 12.




                  Page 28                            GAO/NSIAD-97-68 Joint STARS Production Decision
                  Appendix I
                  Comments From the Department of Defense




See comment 6.


Now on p. 8.




See comment 13.




Now on p. 8.




Now on p. 9.




Now on p. 9.




See comment 14.




                  Page 29                            GAO/NSIAD-97-68 Joint STARS Production Decision
                 Appendix I
                 Comments From the Department of Defense




Now on p. 10.




Now on p. 10.



See comment 8.


Now on p. 12.

Now on p. 12.




See comment 2.

Now on p. 12.




                 Page 30                            GAO/NSIAD-97-68 Joint STARS Production Decision
                         Appendix I
                         Comments From the Department of Defense




See comment 15.



See comment 16.




Now on p. 14.




See comment 8.




Now on pp. 14-15.




See comments 1 and 17.




                         Page 31                            GAO/NSIAD-97-68 Joint STARS Production Decision
                         Appendix I
                         Comments From the Department of Defense




Now on p. 15.




See comments 1 and 18.


Now on p. 15.




See comment 8.




                         Page 32                            GAO/NSIAD-97-68 Joint STARS Production Decision
               Appendix I
               Comments From the Department of Defense




               The following are GAO’s comments on the Department of Defense’s (DOD)
               letter dated March 31, 1997.


               1. We have not suggested or recommended that Joint STARS production be
GAO Comments   interrupted. We have, however, suggested actions that we believe (1) will
               help reduce the program’s risk; (2) could result in the acquisition of a
               more effective, less costly system; and (3) could help decisionmakers
               ensure that the Joint STARS program continues to make progress.

               2. The report has been modified in light of DOD’s comments.

               3. DOD’s indication that other factors were considered in deciding to
               proceed to full-rate production is a signal that DOD and the Air Force are
               willing to accept a high level of risk even when the Director, Operation,
               Test, and Evaluation (DOT&E) has concluded that the system was
               unsuitable as tested and operational effectiveness for war remains to be
               demonstrated. We believe, given the system’s test performance as reported
               by both the Air Force Operational Test and Evaluation Command (AFOTEC)
               and DOT&E and the program’s procurement cost growth of $1 billion
               between the low-rate and full-rate production decision points, that an
               informed full-rate production decision required the following information:
               (1) an approved test and evaluation master plan for follow-on operational
               testing and specific plans for the tests called for in that master plan,
               (2) the results of the already ongoing study of ways to reduce the
               program’s cost, and (3) an analysis of alternatives to the current platform.
               DOD did not have these items in hand when it made its decision. We must
               also note that DOD implies that our recommendations would require a
               break in production. This is inaccurate. As we stated in the body of our
               report, the program could have continued under low-rate initial
               production (LRIP) until operational effectiveness and suitability for combat
               were demonstrated and plans to address identified deficiencies and reduce
               program costs were completed.

               4. In its report on the Joint STARS multi-service operational test and
               evaluation, AFOTEC stated that “Joint STARS software is immature and
               significantly impedes the system’s reliability and effectiveness.” We do not
               believe that, given the software intensive nature of the system, this
               statement supports a conclusion that the system could be judged
               operationally effective.




               Page 33                            GAO/NSIAD-97-68 Joint STARS Production Decision
Appendix I
Comments From the Department of Defense




5. We must note that follow-on operational test and evaluation of the
system was planned before the full-rate production decision. The full-rate
production decision called for acceleration of that testing and for that
testing to address deficiencies identified in the earlier tests. Joint STARS
could have continued under LRIP pending a demonstration of operational
effectiveness and suitability.

6. This speaks to the number of aircraft missions planned and the number
for which an aircraft was provided. It does not address the quality or
quantity of the support provided during those missions. Furthermore,
DOD’s comment refers to the same—operation that is reported on in both
the Air Force and DOT&E reports and in this report.

7. U.S.-based contractor support was utilized during the first Operation
Joint Endeavor deployment. It is also our understanding that during the
second Operation Joint Endeavor deployment the Air Force may have
utilized a “reach-back” maintenance concept in which U.S. stationed
contractor staff were providing field support through satellite
communications. Moreover, DOD and Air Force officials told us that at least
at the beginning of the second Operation Joint Endeavor deployment,
contractor staff were flying on the deployed aircraft. This clearly raises the
question of what the overall level of contractor support was for both the
first and second deployments.

More importantly, a decrease in the level of contractor support between
the two Operation Joint Endeavor deployments does not speak to (1) the
poor test results during the first deployment with, and in spite of, the level
of contractor support or (2) the quality of the system’s performance during
the second deployment; that is, there was no independent—DOT&E—
measurement or observation of how the system performed against its
operational requirements.

8. Given that (1) the procurement cost growth of $760 million for 19 Joint
STARS aircraft since the low-rate production decision; (2) a current
707 airframe purchase, refurbishment, and modification cost of about
$110 million; (3) the age of the current airframes—25 to 30 years; and
(4) the $7 billion estimated operations and maintenance life-cycle cost of
those aircraft, we continue to believe that an analysis of alternatives to the
current air platform should be performed, a belief bolstered by DOD’s
inability to provide copies of its reported analyses and by the
recommendations of the Command, Control, Communications,
Computers, Intelligence, Surveillance, and Reconnaissance Mission



Page 34                            GAO/NSIAD-97-68 Joint STARS Production Decision
Appendix I
Comments From the Department of Defense




Assessment, as discussed on pages 10, 11, 15, and 16 of this report. We
believe that the issue clearly warrants further study. Furthermore, given
DOD’s resistance to the concept, we are more convinced of the merits of
our suggestion that Congress might wish to require a report on such an
analysis.

9. DOD stated that there were some improvements and enhancements “that
could benefit the warfighter” but acknowledges that those features were
not tested. It did not respond, however, to our comments that DOT&E
judged some of those features “essential” to mission accomplishment.
Furthermore, it did not state whether those features would be subjected to
operational testing and evaluation. As we stated in the body of our report,
since at least two of the features present during Operation Joint Endeavor
were “essential” to its mission accomplishment, have already been
developed, and may be needed “to achieve a capable Joint STARS system,”
those features should also be tested during the planned Joint STARS
follow-on test and evaluation.

10. Given the level of contractor support during the multi-service
operational test and evaluation, we are unable to understand how that
support could not have impacted upon the system’s test results. As we
stated in our report, AFOTEC reported that 80 contractors were deployed,
“three or four [contractor] systems engineers flew on each flight to ensure
they could provide system stability and troubleshooting expertise during
missions,” and “three or four . . . were on the ground full time, researching
and developing fixes to software problems . . . .” Furthermore, in its report
DOT&E states that “While these [contractor] needs show that [Joint STARS]
requires sophisticated support, they may also mask certain deficiencies.” It
also reported that

“As already discussed, extensive efforts by the system contractor were required to achieve
the demonstrated availability for the E-8C aircraft. Even with those efforts the system was
not able to meet the user criteria for several measures directly related to the maintenance
concept in place during [Operation Joint Endeavor]—a concept that involved considerably
more contractor support than previously envisioned.”


11. As we noted in the body of our report, Joint STARS failed to meet test
criteria during an operation less demanding than combat, even with such
significant contractor involvement beyond that planned for in combat. In
discussing operational tempo in its Beyond LRIP report, DOT&E stated that if
the system is operated at rates similar to the Airborne Warning and
Control System, “it is questionable whether the [Joint STARS aircraft] can




Page 35                                 GAO/NSIAD-97-68 Joint STARS Production Decision
Appendix I
Comments From the Department of Defense




be sustained over time.” DOD commented that an unbiased assessment of
the measure of Joint STARS’ ability to maintain the required tempo could
not be made and would be tested during the follow-on operational test and
evaluation. We believe that an informed full-rate production decision
requires knowledge of a system’s ability to satisfy the operational tempo
expected of it. DOD made its Joint STARS full-rate production decision
without this knowledge.

12. We understand that Joint STARS, like most systems, has limitations that
need to be planned around. At issue here is a question of how great those
limitations are and whether they are acceptable. DOD states that “the user
is satisfied that the system meets requirements.” However, we must note
that the Air Force’s own Operational Test and Evaluation Center reported
that the “two critical suitability [measures of performance, sortie
generation rate and mission reliability rate], were affected by [Operation
Joint Endeavor] contingency requirements and system stability problems.”

AFOTEC stated that the sortie generation rate performance was
undetermined and judged the other critical suitability measure of
performance—mission reliability rate—as not being met. In discussing the
later critical measure of performance AFOTEC reported that

“The high failure rate of aging aircraft components affected [mission reliability rates] as
critical failures were statistically determined to affect over 30 percent of the sorties flown.
Analysis revealed the elevated critical failure rate was steady and showed no potential for
improvement. Technical data and software immaturity affected the maintainability of the
aircraft, and contractor involvement further compromised clear insight into the Air Force
technicians’ ability to repair the system.”


AFOTEC  also reported on Joint STARS performance relative to 15 supporting
suitability criteria. It stated “Eight did not meet users’ criteria. One was not
tested. Only one . . . met the users’ criteria. The remaining five are reported
using narrative results.”

13. DOD discusses only the weight growth of funded activities, leaving open
the question of whether there are future, but currently unfunded,
improvements planned that will add weight growth. Air Force officials told
us that the Airborne Warning and Control System had experienced weight
growth over the life of its program. That growth was attributed to the
system’s being given added tasks over time. We believe it reasonable to
expect that the Joint STARS program experience might track that of the
Airborne Warning and Control Systems program, that is, be given added




Page 36                                   GAO/NSIAD-97-68 Joint STARS Production Decision
Appendix I
Comments From the Department of Defense




tasks and face weight growth as a result. Also, regarding Joint STARS room
for growth, DOD previously advised us that Joint STARS currently has about
455,000 cubic inches of space available. We must note that this equates to
a volume of under 7 feet cubed and that in commenting on the system’s
space limitation, DOT&E stated “There is little room available for additional
people or operator workstations.”

14. As we stated in the body of our report, how easily these software
changes are incorporated remains to be seen.

15. We requested and DOD provided additional information on this point.
DOD’s subsequent response indicates that this DOD comment was in error.
In its subsequent response, DOD stated that the follow-on test and
evaluation was accelerated “to reflect [Office of the Secretary of Defense]
desire for earlier [operational test and evaluation] to evaluate fixes to
[multi-service operational test and evaluation] deficiencies.” We believe
this statement reflects a recognition of increased program risk.

16. The acquisition decision memorandum approving Joint STARS
production clearly indicates that the Skantze study1 mentioned was not
completed at that time. We believe that the full-rate production decision
should have been made with the Skantze study in hand. Furthermore, we
do not understand why DOD felt the need to direct the Air Force to fund
and implement a plan that is to save it money, but felt no need to direct the
Air Force to examine alternative platforms that at least one other DOD
panel had stated would not only save $3 billion but also provide greater
effectiveness.

17. We believe that not only should DOT&E approval of the Joint STARS Test
and Evaluation Master Plan be required, but also of all supporting test
plans. We have changed the language of this matter for congressional
consideration accordingly.

18. We are suggesting that Congress may wish to request a more detailed
report, one at the Beyond LRIP report level of detail, a level of detail not
provided in DOT&E’s annual report. Given that DOT&E could only state
effectiveness for operations other than war—could not state a belief as to
whether the system would be effective in two of the three critical
operational roles it is expected to perform in war—and found the system
unsuitable as tested, we believe that such report would help Congress

1
  This study was an affordability review of Joint STARS that examined the affordability, sustainability,
and life-cycle costs to thoroughly evaluate and control program costs.



Page 37                                        GAO/NSIAD-97-68 Joint STARS Production Decision
           Appendix I
           Comments From the Department of Defense




           maintain program oversight. DOD’s comment of “other reports as
           appropriate” leaves the matter in DOD’s hand to decide if Congress would
           benefit from such a report.




(702226)   Page 38                            GAO/NSIAD-97-68 Joint STARS Production Decision
Ordering Information

The first copy of each GAO report and testimony is free.
Additional copies are $2 each. Orders should be sent to the
following address, accompanied by a check or money order
made out to the Superintendent of Documents, when
necessary. VISA and MasterCard credit cards are accepted, also.
Orders for 100 or more copies to be mailed to a single address
are discounted 25 percent.

Orders by mail:

U.S. General Accounting Office
P.O. Box 6015
Gaithersburg, MD 20884-6015

or visit:

Room 1100
700 4th St. NW (corner of 4th and G Sts. NW)
U.S. General Accounting Office
Washington, DC

Orders may also be placed by calling (202) 512-6000
or by using fax number (301) 258-4066, or TDD (301) 413-0006.

Each day, GAO issues a list of newly available reports and
testimony. To receive facsimile copies of the daily list or any
list from the past 30 days, please call (202) 512-6000 using a
touchtone phone. A recorded menu will provide information on
how to obtain these lists.

For information on how to access GAO reports on the INTERNET,
send an e-mail message with "info" in the body to:

info@www.gao.gov

or visit GAO’s World Wide Web Home Page at:

http://www.gao.gov




PRINTED ON    RECYCLED PAPER
United States                       Bulk Rate
General Accounting Office      Postage & Fees Paid
Washington, D.C. 20548-0001           GAO
                                 Permit No. G100
Official Business
Penalty for Private Use $300

Address Correction Requested