United States GAO General Accounting Office Washington, D.C. 20548 Information Management and Technology Division B-238826 April 19,199O The Honorable John Conyers, Jr. Chairman, Legislation and National Security Subcommittee Committee on Government Operations House of Representatives The Honorable Frank Horton Ranking Minority Member, Legislation and National Security Subcommittee Committee on Government Operations House of Representatives Your May 18,1989, letter expressed concern about the Department of Defense’s (DOD) management process for controlling the development and acquisition of embedded computer resources-computer software, hardware, and firmware,’ which are a physical part of a weapons sys- tem and necessary to perform the system’s mission-and major auto- mated information systems. In subsequent discussions with your offices, we agreed to determine (1) how important and prevalent embedded computer resources are to Defense weapons systems, how the Office of the Secretary of Defense (OSD) oversees the development of embedded computer resources, and how the oversight process has evolved since 1976; and (2) whether DOD is effectively controlling the acquisition of major automated information systems. As agreed, this report addresses the first objective and offers some observations on the oversight pro- cess. We will report separately on the second objective. Our work for this report was conducted from June 1989 through December 1989, pri- marily at OSD offices in Washington, D.C., and at selected weapons sys- tem program offices throughout the country. A detailed explanation of our objectives, scope, and methodology is contained in appendix I. Effective management oversight is imperative in the development of Results in Brief computer resources embedded in today’s weaponry. These resources control such critical tasks as navigation, enemy detection, and weapon firing on multimillion-dollar pieces of military armament. Further, while ‘A special type of computer program that is classified as neither hardware nor software. Firmware consists of instructions permanently stored in a special section of a computer’s memory that the com- puter can read from but cannot write into. It typically controls hardware or consists of commonly used computer programs. Page 1 GAO/IMTEG90-34 DOD Embedded Computer Technology I%238826 DOD’S investment in mission-critical computer resources2 is substantial- an estimated $30 billion in 1990 -this investment pales in comparison with the cost of the weapons systems that will not work effectively without this technology. Moreover, the prevalence and importance of computer technology to these weapons systems has increased tremen- dously in recent years, and performance problems due to computer soft- ware “bugs” are becoming all too commonplace. For example, we recently reported on software problems with the Air Force’s B-1B bomber’s defensive avionics system-problems that the program office estimates will cost about $1 billion to correct and will still only provide limited performance improvements.3 Given that the dependence of weapons systems on computer resources is expected to grow, the value of focused management attention becomes even more acute. OSD has in place a process to supervise the development of weapons sys- tems. This process permits oversight of systems’embedded computer resources through a high-level review board supported by various com- mittees and ad hoc working groups. The mission of these bodies is to identify and resolve weapons system issues before allowing systems to advance to the next stage of development. This process does not treat embedded computer resources as a discrete area of focus. According to OSD officials, reasons for this approach include (1) preferred concentra- tion on the entire weapons system rather than its separate components, (2) a lack of comfort by board and committee members with computer resource issues, and (3) the absence of a designated OSD entity specifi- cally responsible for overseeing systems’embedded computer resources. DOD is currently examining its approach to overseeing and managing the development of computer systems. However, DOD already completed a related study in 1982,* and its findings and recommendations on embed- ded computer resource oversight may still be appropriate today. According to that study, the scope of automation in weapons systems far outstrips OSD’S committee-management approach to overseeing it. As a result, the study called for (1) designation of a senior official to advise weapons system review and approval authorities on computer resource %ission-critical computer resources include embedded computer resources as well as computer resources used for such activities as intelligence, cryptography, and command and control. Cost data are not accumulated and available on embedded computer resources. 3Strategic Bombers: BIB Cost and Performance Remain Uncertain (GAO/NSIAD-89-55, Feb. 3, 1989). 4Final Report of the Defense ScienceBoard Task Force on Embedded Computer ResourcesAcquisition and Management, November 1982. Page 2 GAO/IMTEC90-34 DOD Embedded Computer Technology matters and (2) improvement in OSD’S oversight of embedded computer resource development activities. Although OSD took certain actions at the time that address these areas, over the years these initiatives have silently expired and have been gradually replaced by an oversight struc- ture that is fundamentally the same as the one DOD reviewed and criti- cized in 1982. Such an approach raises the question of whether embedded computer resources are receiving the level of OSD oversight that their role in today’s weapons systems suggests they should. What are embedded computer resources? Although a widely used term, Background it is not well defined. As a result, DOD officials differ in what they con- sider the term to mean. For the purposes of this report, we define embedded computer resources to include any computer hardware, soft- ware, or firmware that is physically part of and necessary for a weap- ons system to perform its full mission. Of these resources, software is usually the most difficult and costly to develop and maintain. Embedded computer resources continue to take on more significance with each new, more sophisticated weapons system being developed. For example, the Air Force’s F-4 fighter of the early 1970s had practi- cally no software, while today’s F-16D fighter has over 200,000 lines of code.”According to a 1987 Defense task force report,6 software-intensive systems have mushroomed in the past 5 years, with annual mission- critical computer resource costs rising from $9 billion in 1985 to $30 billion in 1990. Accompanying this growth in costs has been an increase in weapons sys- tem problems linked to embedded computer resource difficulties. According to the software and computer technology focal point in OSD'S Office of the Deputy Director for Defense Research and Engineering (Research and Advanced Technology), many weapons systems are behind schedule largely because of software problems. In addition, an official in OSD’S Office of the Deputy Director for Research and Engineer- ing (Test and Evaluation) estimated that 7 out of 10 major weapons sys- tems in development today are encountering software problems, and the rate is increasing. Similarly, the Deputy Secretary of Defense has said that sophisticated computer technology is a growing problem in fielding reliable weapons systems. Our work also confirms the importance of this 5Defiitions of a line of code vary. Generally, a line of code is considered a single computer program command, de&ration, or instruction. ‘Report of the Defense Science Board Task Force on Military Software, September 1987. Page 3 GAO/IMTEG9034 DOD Embedded Computer Technology B238826 technology. For example, schedule delays and cost overruns on the Air Force’s C-17A cargo aircraft and the Navy’s V-22 aircraft can be linked to embedded computer resource problems. Additional information on embedded computers is contained in appendix II. Over the last decade, OSD oversight of embedded computers has been Evolution of accomplished through various committees, groups, and councils-an Embedded Computer approach that has met with some internal criticism. Beginning in 1976, System Oversight the Management Steering Committee for Embedded Computer Resources was established to, among other things, advise the Defense System Acquisition Review Council7 on specific major system embedded com- puter resource issues. In 1982, a Defense task force examined whether this committee served a useful role and whether another approach to embedded computer oversight and policy guidance was warranted. In short, the task force found that (1) senior DOD management was clearly not comfortable in addressing computer resource issues, particularly software; (2) OSD had outgrown its committee management approach to addressing these issues; and (3) embedded computer resource manage- ment oversight was not at a high enough organizational level to signal management concern and provide control. It recommended that (1) a senior official be designated to advise weapons systems review and approval authorities on computer resource matters and (2) improve- ments occur in OSD'S oversight of embedded computer resources.8 In 1983 the Management Steering Committee for Embedded Computer Resources was replaced by the Defense Computer Resources Board. Also at this time, a senior Defense official for mission-critical computer resources was designated. Shortly thereafter, this official established the Computer Resources Council to temporarily oversee mission-critical computer resources until the board could establish formal procedures. Neither of these two review bodies currently exists, but it is not clear when they discontinued operation since they were never formally abolished. OSD'S current approach to overseeing embedded computer resources began in 1987. Under this approach, the Defense Acquisition Board and its committees and working groups are responsible for reviewing major weapons systems and deciding whether they are ready to proceed to the 7This council was the predecessor to the Defense Acquisition Board. 8Final Report of the Defense ScienceBoard Task Force on Embedded Computer ResourcesAcquisition Management, November 1982. Page 4 GAOj’IMTEG90-34 DOD Embedded Computer Technology , B-238826 next stage of development. While this review process allows for focused attention on the system’s embedded computer resources, our case stud- ies of three weapons systems and interviews with OSD oversight officials show that these resources are not typically a discrete area of focus at any level in the review process. This lack of focus is caused by (1) a preoccupation with the weapons system as a whole and its ability to satisfy overall mission requirements, (2) a lack of comfort by senior OSD officials with computer resource issues, and (3) an absence of any one OSD entity responsible for embedded computer resource issues. More detailed information concerning the history of OSD'S oversight pro- cess for embedded computer resources and its current approach is con- tained in appendix II. OSD'S process for overseeing embedded computer resources may change Future Changes in the near future as a result of several Defense initiatives currently Possible in OSD underway. Specifically, on October 4,1989, the Deputy Secretary of Oversight of Defense announced a plan to establish an executive-level group to review and make recommendations on the procedures used for oversee- Embedded Computers ing software development, which is clearly DOD'S most significant and problem-plagued computer resource. The Deputy Secretary also announced that, in the interim, the Major Automated Information Sys- tem Review Committee (MAERC), which has previously existed as a sepa- rate review body for major automated information systems, will become a Defense Acquisition Board committee. At this point it is not clear if MAISRC'S review authority will be expanded beyond information systems to include embedded computer resources. Further, it is unclear if MAERC'S organizational placement under the Defense Acquisition Board will be a permanent move. Also not clear is whether the Defense Acqui- sition Board will assume responsibility for directing major automated information system acquisitions, as proposed by the Secretary of Defense in the 1989 Defense Management Report to the President, or whether MAISRC will continue to be solely responsible for this direction. Another initiative that could affect OSD'S embedded computer resources oversight process is a revision of DOD Directive 5000.29, Management of Computer Resources in Major Defense Systems. The current draft directs the Under Secretary of Defense (Acquisition) to appoint a senior DOD official for mission-critical computer resources to advise the Under Secretary and the Defense Acquisition Board on acquisition of these resources for weapons systems. This senior DOD official would be respon- sible for oversight of weapons systems’ embedded computer resources. Page 5 GAO/lMTJSG90-34 DOD Embedded Computer Technology I B238826 Coordination of this draft is on hold pending revis ion of a related direc- tive, and the offic ial responsible for 5000.29 was uncertain when it will be made final. Last, a software working group under the Defense Acquisition Board’s Science and Technology Committee is developing a D O D Software Master Plan. The plan provides specific actions for D O D to follow over the next 5 years to address a variety of goals . Among the many items called for in the plan are (1) designation of an office in OSD and each D O D component with primary responsibility for identify ing, managing, integrating, and implementing software acquis ition and (2) increases in D O D management awareness and v is ibility of software and its impact on s y s tems. How- ever, the plan is currently only in its preliminary s tages and is being c irculated for public comment. The use of computer resources in D O D weaponry grew dramatically in Observations the 1980s; this trend is expected to continue. Compounding this s itua- tion is the fac t that defective software is being linked to weapons s y s - tem cost, schedule, and performance problems. How successful D O D will be in avoiding such problems in the future as it incorporates more and more computer technology in its weapons s y s tems will depend, in part, on how well it supervises the development of the embedded computer resources upon which the weapons s y s tems rely so heavily . DOD'S 1982 tas k force findings concerning the need for greater oversight of embedded computer resources may s till be appropriate today. OSD'S oversight approach in this area has remained fundamentally unchanged over the las t decade-at a time when the importance and prevalence of embedded computers has soared and when weapons s y s tem develop- ment problems have time and again been traced to embedded computer problems. The scope of automation in today’s weapons s y s tems far out- s trips OSD'S committee-management approach to oversee it. DOD is currently examining its process for developing and using com- puter resources, and changes may be forthcoming in OSD'S approach to overseeing the embedded computer area. G iven the extremely high costs of weapons s y s tems that are critically dependent on embedded com- puters, the need for increased management attention to this important technology becomes even more acute. Accordingly , D O D needs to explore better ways to ensure that embedded computer resources, particularly software, receive focused management attention throughout all phases of weapons s y s tem development. Page 6 GAO/IMTJZG90-34 DOD Embedded Computer Technology B-238826 As requested by the Chairman’s office, we did not obtain official agency comments on a draft of this report. However, we did discuss its contents with OSD and service officials, and have incorporated their comments where appropriate. Our work was performed in accordance with gener- ally accepted government auditing standards, between June and Decem- ber 1989. As arranged with your offices, unless you publicly announce its contents earlier, we plan no further distribution of this report until 30 days from the date of this letter. At that time, we will send copies to the Chairmen, Senate and House Committees on Appropriations; the Secretaries of Defense, Air Force, Army, and Navy; and to other interested parties. We will also make copies available to others upon request. This report was prepared under the direction of Samuel W . Bowlin, Director, Defense and Security Information Systems, who can be reached at (202) 275- 4649. Other major contributors are listed in appendix III. Ralph V. Carlone Assistant Comptroller General Page 7 Contents Letter 1 Appendix I 10 Objectives, Scope and Methodology Appendix II 13 DOD Embedded Embedded Computer Resources-An Ill-Defined Term 13 14 Embedded Computer Resources Are Critical to the Computer Resources Performance of Today’s Weapons Systems and OSD Oversight of Embedded Computer Resources Have Been a Recent 18 Cause of Weapons System Development Problems Them OSD’s Oversight of Embedded Computer Resources-An 19 Historical Perspective Current Approach for OSD Oversight of Embedded 21 Computer Resources-A Detailed Description Appendix III 26 Major Contributors to This Report Abbreviations DOD Department of Defense GAO General Accounting Office IMTEC Information Management and Technology Division MAISRC Major Automated Information System Review Committee OSD Office of the Secretary of Defense Page 8 GAO/MTEG90-34 DOD Embedded Computer Technology Page 9 GAO/IMTEG99-94 DOD Embedded Computer Technology Objectives, Scopeand Methodology On May 18,1989, the Chairman and Ranking Minority Member, Legisla- tion and National Security Subcommittee, House Committee on Govern- ment Operations, requested that we review OSD'S process for supervising and controlling embedded computer resources and automated informa- tion systems. On the basis of this request and subsequent discussions with the requesters’ offices, we agreed to determine and separately report on (1) how important and prevalent embedded computer resources are to Defense weapons systems, how OSD oversees the devel- opment of embedded computer resources, and how the oversight process has evolved since 1976; and (2) how effectively OSD oversees the devel- opment and acquisition of automated information systems. This report addresses the first set of questions and offers some observations on the oversight process. We will report separately on the second question. In developing information on the importance and prevalence of embed- ded computer resources, we interviewed knowledgeable OSD and service program officials and reviewed recent publications and reports on the subject. We also examined nine weapons system programs to determine the amount of computer resources being incorporated in each, the cost to develop these resources as compared with the cost to develop the entire weapons system, and the extent to which the weapons system relied on its embedded computer resources. Our criterion for system selection was to choose the three most expensive programs in each ser- vice that were subject to Defense Acquisition Board oversight. The nine weapons systems are: 1. Advanced Tactical Fighter Aircraft Air Force 2. C-17A Cargo Aircraft 3. B-1B Bomber Aircraft 1. LHX Light Helicopter 2. Forward-Area Air Defense System Line-of-Sight Forward Heavy Armored Track Vehicle 3. Single Channel Ground and Airborne Radio System Page 10 GAO/IMTEG90-34 DOD Embedded Computer Technology Appendix I Objectives, !3cope and Methodology 1. SSN-21 Class Submarine Navy 2. V-22 Advanced Vertical Lift Aircraft 3. Trident II (D-5) Missile To determine how OSD oversees embedded computer resources, we reviewed relevant DOD directives and instructions as well as DOD studies and related documentation addressing the subject. We also interviewed 0s~ officials identified for us as either players in the oversight process or persons knowledgeable about it. Additionally, we chose one of the three above-cited weapons system programs in each service as case illustrations of the oversight process. These three are (1) Air Force’s C- 17A program, (2) Army’s Forward-Area Air Defense System Line-of- Sight Forward Heavy program, and (3) Navy’s Trident II program. Our criterion for selecting the three was to pick one from each service that had at least reached the full-scale development decision milestone. In reviewing the three, we interviewed OSD action officers responsible for the day-to-day monitoring of each, as well as service program manage- ment officials. We also analyzed available documentation illustrating oversight events and direction. To document how the oversight process has evolved since 1976, we reviewed past DOD directives and instructions and OSD memoranda, as well as reports and studies addressing the subject. Additionally, we interviewed 0s~ officials identified for us as knowledgeable about the evolution. We performed our work from June 1989 through December 1989, pri- marily at 0s~ offices in Washington, D.C. The principal 06~ offices included the Office of the Deputy Director for Defense Research and Engineering (Research and Advanced Technology), the Office of the Deputy Director for Defense Research and Engineering (Test and Evalu- ation), the Office of the Deputy Director for Defense Research and Engi- neering (Tactical Warfare Programs), the Office of the Deputy Director for Defense Research and Engineering (Strategic and Theater Nuclear Forces), the Office of the Director for Program Integration, the Office of the Director for Operational Test and Evaluation, and the Office of the Assistant Secretary of Defense (Command, Control, Communications, and Intelligence). Other DOD organizations included service program offices for each of our nine weapons system programs located at various sites throughout the country, and the Defense Logistics Agency’s Defense Technology Analysis Office in Alexandria, Virginia. Page 11 GAO/IMTEG9034 DOD Embedded Computer Technology Appendix I Objectives, Scope and Methodology As requested by the Chairman’s office, we did not obtain official agency comments on a draft of this report. However, we discussed its content with OSD and service officials, and have incorporated their comments where appropriate. Our work was performed in accordance with gener- ally accepted government auditing standards. Page 12 GAO- DOD Embedded Computer Technology A&en& II DOD Embedded Computer Resourcesand OSD Oversight of Them This appendix provides background information on computer resources embedded in Defense weapons systems, including how DOD defines and views these resources, how important and prevalent the resources are, and how OSD’S oversight of them has progressed since 1976. Embedded computer resources are nebulously defined in DOD. A 1987 Ernbedded Computer Defense Science Board’ report broadly describes them as software sys- Resources-An tems that are functionally unique and embedded in larger systems. Simi- Ill-Defined Term larly, Defense Directive 5000.29, Management of Computer Resources in Major Defense Systems, defines embedded as “. . . adjective modifier; integral to, from the design, procurement, and operations point of view. . . .” A 1982 Defense Science Board report? more specifically defines embedded computer resources as computers incorporated as an integral part of, dedicated to, or required for the direct support of, or for the upgrading or modification of, major or less-than-major systems. Thus this term refers not only to those computing devices buried deeply within subsystems as radars, radios, missiles and the like but more generally to com- puters which are used to perform a portion of a larger task such as fire-control, automatic testing, navigation, and threat warning. The key discriminator is whether the application is computation alone or whether computation is merely a subtask to be performed as a part of a larger activity. . . . They perform specialized and dedi- cated tasks and are not, in general, available to support the general computational or data processing needs of the organization and hence are subject to a more special- ized selection process than classical Automated Data Processing Equipment. Officials in OSD oversight roles and in service program offices varied in how they viewed embedded computer resources. For example, an offi- cial in the Office of the Deputy Director for Defense Research and Engi- neering (Research and Advanced Technology) said that embedded computer resources are those computers that are integral to or part of a weapons system, as identified in the Warner Amendment3 In contrast, an official from the Office of the Deputy Director for Defense Research and Engineering (Test and Evaluation) stated that embedded computer *Report of the Defense Science Board Task Force on Military Software, September 1987. %inal Report of the Defense Science Board Task Force on Embedded Computer ResourcesAcquisition and Management, November 1982. 3Federal agencies’acquisition of general-purpose computer equipment is governed by the Brooks Act (40 U.S.C. 759), which gives oversight authority for such acquisitions to the General Services Admin- istration. The Warner Amendment (10 U.S.C. 2315) exempts from the Brooks Act computer systems that are (1) intelligence- and cryptologic-related, (2) command and control of military forces-related, (3) integral to a weapons system, or (4) critical to fulfilling military or intelligence missions and that are not used for routine administrative and business applications. Page 13 GAO/lMTEC90-34 DOD Embedded Computer Technology Appendix Jl DOD Embedded Computer Resources and OSD Oversight of Them resources include all mission-critical systems (i.e., all categories of sys- tems covered by the Warner Amendment). Other descriptions offered include a computer system that cannot be separated from the weapon without the weapon experiencing a serious decline in effectiveness, and a computer system that is integral to a weapons system. Additionally, officials with several of the major weapons system programs we sur- veyed told us that they consider embedded computer resources to be those that are critical to fulfilling the mission of their weapons systems. However, these officials differed as to whether this includes computer resources used for ground-based training and support equipment. According to an official for one program, the determining factor should be whether the computer resources in question are managed or budgeted for by the program office. If they are, then they are considered embed- ded in the weapons system. Embedded computer resources are playing a larger and more significant Embedded Computer role in the functioning of weapons systems. In fact, in the not-too-distant Resources Are Critical fu t ure, it is conceivable that virtually every subsystem in all major to the Performance of weapons systems will be computer-controlled. According to a Defense Marketing Service estimate, the year 1990 will see 250,000 computers Today’s Weapons installed in military systems, compared with 10,000 in 1980. One illus- Systems tration of weapons systems’ growing reliance on embedded computers is the F-16A fighter aircraft, which had about 125,000 lines of code and 50 processors. In contrast, one of its offspring, the F-16C, has an estimated 230,000 lines of code and 300 processors.4 Another illustration is the C- 5A cargo aircraft, which requires 25,000 lines of code, while its succes- sor, the C-17A, requires an estimated 700,000 lines of code. DOD’S budgetary investment in mission-critical computer resources, which includes embedded computer resources, is also growing. Accord- ing to a 1987 Defense Science Board report, software-intensive systems have grown exponentially since 1985, with software costs rising from $9 billion annually to an expected $30 billion by 1990. While significant in and of itself, these figures pale in comparison to the literally hundreds of billions of dollars being invested in the Defense weapons systems that so heavily depend on embedded computer resources. To illustrate, the program cost for the Army’s Light Helicopter, its next generation scout and attack helicopter, is estimated to be about $40 billion. However, 4A processor is the part of a computer that interprets and executes instructions. Page 14 GAO/lMTEG9O34 DOD Embedded Computer Technology Appendix II DOD Embedded Computer Resources and OSD Oversight of Them only about 4 percent of this cost is estimated at this time to relate to embedded computer resources.5 To better gauge the importance and prevalence of embedded computer resources in today’s weapons systems, we surveyed the three largest weapons system programs in each of the three services (i.e., nine weap- ons systems) that are under the purview of the Defense Acquisition Board. The nine systems are the Air Force’s Advanced Tactical Fighter, C-17A cargo transport, and the Bl-B bomber; the Army’s LHX Light Helicopter, Forward-Area Air Defense System Line-of-Sight Forward Heavy armored tracked vehicle, and Single Channel Ground and Air- borne Radio System; and the Navy’s SSN-21 class submarine, V-22 advanced vertical lift aircraft, and Trident II (D-5) missile. In sum, we found that data on the cost and size are not routinely collected and available because the work breakdown structure and cost accounting for weapons system projects is not structured this way. As a result, some program offices could not provide all the cost and size data requested. Our specific findings on the nine systems follow. .r Force l The Advanced Tactical Fighter, which is to become the Air Force’s pri- mary air superiority fighter, will require an estimated 4.5 to 6 million lines of code. Additionally, the projected development cost for just the aircraft’s avionics embedded computer resources is about $1 billion6 or about 13 percent of the total weapons system’s development cost. According to the program office software manager, this aircraft could not perform its mission without its embedded computer resources. . The C-17A is expected to have 93 separate processors and approxi- mately 700,000 lines of code, making it the most computerized cargo transport aircraft ever developed. The development cost of these embedded computer resources is estimated at $225 million, or about 5 percent of the total weapons system’s development cost. According to program officials, the C-17A is highly dependent on its embedded com- puter resources to perform its mission. . The B-1B is estimated to have 1.3 million lines of flight software code; the development cost of the bomber’s embedded computer resources is about $726 million, or approximately 19 percent of the total weapons 5According to the deputy program manager, the helicopter is in the early stages of development, making the full extent of embedded computer resources’use uncertain at this time. The 4 percent accounts for the two largest uses of computer technology on the aircraft. Although additional uses are likely, their size and cost were not yet known. ‘Cost estimates for other embedded computers were not available. Page 16 GAO/IMTEG9o-34 DOD Embedded Computer Technology , Appendix II DOD Embedded Computer Resources and OSD Oversight of Them system’s development cost. According to the program office, critical B-1B functions such as radar, navigation, and weapon delivery, cannot satisfy mission requirements without the embedded computer resources. As a result, the program office described the bomber as highly depen- dent on these resources. . The LHX helicopter program, which is in the early stages of develop- ment, is estimated so far to require 950,000 lines of software code for the embedded computer resources controlling navigation and fire con- trol-the two largest uses of computer technology on the helicopter. Additionally, the deputy program manager stated that more embedded computer resources will likely be required, but the aircraft’s design is not far enough along to know their size and cost. The early cost estimate to develop the two largest embedded computer resources is $115 million, or about 3 percent of the estimated cost to develop the entire aircraft. The deputy program manager stated that the aircraft is highly depen- dent on embedded computer resources for navigation and weapons fire control. l The armored track vehicle for forward-area, line-of-sight, air defense, although using embedded computer resources, is a nondevelopment item. It was developed with private funds for another purpose and later sold to the Army. Thus, while it relies on the execution of about 425,000 lines of code to perform its mission, there are no development costs asso- ciated with this code. According to the deputy program manager, with- out the system’s embedded computer resources to identify, set priorities, and fire at targets, the system would be useless. l The single channel radio system is not a complex weapons platform with large, sophisticated embedded computer resources like the other eight systems we examined. As stated by the deputy program manager, it is a radio system with five microprocessors, like those found in a commer- cial videocassette recorder. Further, these embedded computer resources are bought off-the-shelf, so there is no development cost. The estimated number of lines of code in each radio system is 15,000; accord- ing to the deputy program manager, the radios are totally dependent on their embedded computer resources to function as intended. Page 16 GAO/lMTEG9O-34 DOD Embedded Computer TecJmology Appendix II DOD Ehbedded Computer Ibsourcea and OSD Oversight of Them . The SSN-21 submarine relies on embedded computer resources to per- Navy form a large number of on-board functions. For example, the subma- rine’s AN/BSY-2 combat system7 has approximately 200 separate processors; according to the program office’s ship acquisition manager, the Enhanced Modular Signal Processor within the AN/BSY-2 is by far the most costly and important of these processors-having an estimated 193,000 lines of code.s The development cost of the Enhanced Modular Signal Processor is about $450 million, which is about 13 percent of the submarine’s development cost. However, the program manager for the Enhanced Modular Signal Processor stated that this processor is used in 16 different weapons system programs. Thus, allocating its cost among all 16 programs reduces its portion of the SSN-21’s development cost to less than 1 percent. Regardless, the SSN-21 acquisition program mana- ger stated that the submarine is highly reliant on embedded computer resources to meet its mission requirements. . The V-22 is a tilt rotor, vertical takeoff and landing aircraft for joint service application. Although the program was recently cancelled, it was to use approximately 50 microprocessors and 700,000 lines of code. The program office could not provide the cost to develop these embedded computer resources because developing these data would be difficult enough that the contractor would charge for it. However, the deputy program manager stated that the V-22 could not fly without its embed- ded computer resources. l Although each of the Trident II missile’s five subsystems rely on com- puter technology, the program manager stated that the navigation and fire control subsystems are the two principal areas using embedded com- puter resources. According to the program office, these resources entail about 1.1 million lines of software code, and the cost to develop them is approximately $280 million, or roughly 9 percent of the missile’s devel- opment cost. Further, the program manager stated that without the mis- sile’s embedded computer resources, there would not be a Trident II missile. 7An advanced combat system designed to detect, classify, track, and launch weapons at enemy sub- surface, surface, and land targets. *The AN/EC%-2 combat system has an estimated 3.6 million lines of code. Page 17 GAO/IMTJS90-34 DOD Embedded Computer Technology I Appendix II DOD Embedded Computer Resources and OSD Oversight of Them The potential exis ts for embedded computer resources to cause serious Embedded Computer weapons s y s tem problems, not only in terms of cost overruns and sched- ResourcesHave Been a ule delay s , but, more importantly , in terms of performance degradation. Recent Cause of According to an offic ial in the O ffice of the Deputy Direc tor for Defense Research and Engineering (Test and Evaluation), weapons s y s tems are W eapons Sy s tem encountering major performance defic ienc ies primarily due to software Development Problems problems. In fac t, this offic ial s tated that 7 out of 10 weapons s y s tems under the purview of the Defense Acquisition Board are encountering software problems and this rate is increasing. The Air Force’s Bl-B bomber aircraft is a case in point. W e recently reported9 that this s y s - tem’s computer-based defensive avionic9O cannot meet its requirements without a major redesign. Although program office offic ials believe that software revis ions costing an estimated $1 billion may allow limited per- formance improvements, a s ignificant drop-off in s y s tem capability nev- ertheless exis ts and cannot be corrected without a major redesign. W e also recently reported about other weapons s y s tem problems which are linked to software development, inc luding schedule delay s in the Air Force’s C-17AI and the Navy’s V-22 aircraft.12 The impact that embedded computer resource problems can have on weapons s y s tems’cost, schedule, and performance objec tives appears to be well recognized in DOD. In fac t, the Commander, Air Force Systems Command, has been quoted as characteriz ing software as the Achilles ’ heel of weapons development. Additionally , an offic ial in the O ffice of the Deputy Direc tor for Defense Research and Engineering (Research and Advanced Technology) s tated that many weapons s y s tems being developed are behind schedule, and more often than not the blame is on software. Similarly , the Deputy Secretary of Defense has been quoted as saying that sophis ticated software in weapons s y s tems is a growing problem in bringing reliable weapons s y s tems to the field. The Direc tor of the O ffice of Operational Test and Evaluation s tated that assessing software-intensive weapons s y s tems is the bigges t challenge over the next 10 years and that not enough attention is being paid to this issue. gStrategic Bombers: BlB Cost and Performance Remain Uncertain (GAO/NSIAD89-65, Feb. 3, . 1989). loThe radar warning receiver and processorfunction that initiates defensive action by receiving and identifying threat system s ignals. llMilitary Alrliftz C-17 FacesSchedule, Cost, and Performance Challenges (GAO/NSIAD-89-195, Aug. 18,1989). %efense Acquisition Programs: Status of Selected Systems (GAO/W&ID-90-30, Dec. 14,1989). Page 18 GAO/IMTEG90-34 DOD Embedded Computer Technology . I Appendix II DOD Embedded Computer Resources and OSD Oversight of Them Over the last decade, OSD’S approach to overseeing embedded computer OSD’s Oversight of resources has experienced little fundamental change. Basically, OSD has Ernbedded Computer relied on various committees to examine these computer resources, for- Resources-An mulate policies, and raise any pertinent issues to the OSD decision-mak- ing body responsible for reviewing the computers’ host weapons system. Historical Perspective A chronological description of the oversight committees and their m is- sions follows. Management Steering In April 1976, the Management Steering Committee for Embedded Com- puter Resources was established to oversee the management of com- Committee for Embedded puter resources in major weapons, communications, command and Computer Resources control, and intelligence systems. The steering committee was chartered (1976-1983) to (1) develop and oversee implementation of acquisition and manage- ment policies for major system embedded computer resources, (2) advise the Defense System Acquisition Review Council13on general policy mat- ters and on specific major system embedded computer resource issues, and (3) incorporate embedded computer resource considerations into the established major system acquisition process. The steering committee’s decision-making power was vested in an executive board, chaired by the Assistant Secretary of Defense (Installations and Logistics). This com- m ittee was replaced in 1983 by the Defense Computer Resources Board. Defense Computer In March 1983, the Under Secretary of Defense for Research and Engi- neering, as the Defense Acquisition Executive, delegated to the Deputy Resources Board (1.983-?) Under Secretary of Defense (Research and Advanced Technology) the role of senior official for m ission-critical computer resources. The Under Secretary also identified the Defense Computer Resources Board as the successor to the Management Steering Committee for Embedded Com- puter Resources. In August 1984, the Defense Computer Resources Board was formally chartered to oversee Warner Amendment-exempt computer resources. The charter did not specify any role for this board in advising the Defense Systems Acquisition Review Council on computer resource mat- ters. Officials from the Office of the Deputy Director, Defense Research and Engineering (Research and Advanced Technology) and the Office of the Director for Program Integration confirmed that the board did not advise the Defense Systems Acquisition Review Council. The first offi- cial also stated that this board, although no longer in existence, was 13This council was the predecessor to the Defense Acquisition Board. Page 19 GAO/lMTEG9@34 DOD Embedded Computer Technology Appendix II DOD Embedded Computer Resources and OSD Oversight of Them never formally abolished. We were unable to find any information or evidence describing actions this board took in response to its charter. Computer Resources Also during August 1984, the Deputy Under Secretary of Defense (Research and Advanced Technology) established the Computer Council (1.984-?) Resources Council to oversee mission-critical computer resources until the Defense Computer Resources Board established formal procedures for oversight. This council also did not have a specified role in support- ing the Defense Systems Acquisition Review Council. In February 1986, the Deputy Secretary of Defense directed that policy responsibility for general-purpose, mission-critical computer resources be moved under the IvIAISRC,‘~ but that the Under Secretary for Defense Research and Engineering continue to oversee computer resources embedded in weapons systems. The Deputy Secretary did not comment on the role of the Computer Resources Council. According to officials in the Office of the Deputy Director, Defense Research and Engineering (Research and Advanced Technology), although the Computer Resources Council no longer exists, it was never formally abolished. Defense Acquisition Board In February 1987, the Under Secretary of Defense (Acquisition) was designated to serve as the Defense Acquisition Executive, the principal (1987-Present) staff assistant and adviser to the Secretary of Defense on all matters relating to the acquisition system. In September 1987, the Defense Acquisition Board was established as the primary forum to resolve issues, provide and obtain guidance, and make recommendations to the Defense Acquisition Executive. Ten acquisition committees were also established to support the board by raising and resolving issues, making recommendations on weapons system milestones, and developing acqui- sition policy. The committee charters authorized the committee chairpersons to establish ad hoc working groups to address specific issues or weapons systems. Recently, MAISRC was made the Board’s elev- enth committee. Three of the eleven committees address specific weapons systems and account for almost all of the board’s activity. These are the Conven- tional Systems Committee; the Command, Control, Communications, and Intelligence Systems Committee; and the Strategic Systems Committee. “MAJSRC is responsible for review and approval of major automated information systems. It was created in the late 1970s; in 1989 it was moved under the Defense Acquisition Board as a commitke. Page 20 GAO/IMTEC9O-34 DOD Embedded Computer Technology Appendix JI DOD Embedded Computer Resources and OSD Oversight of Them The remaining eight committees address functional areas, such as sci- ence and technology, test and evaluation, and production and logistics. Although there is no specific committee for embedded computer resources, the process does not preclude any of the committees from addressing it. Embedded computer resource oversight can occur as part of the Defense Current Approach for Acquisition Board’s review process for major weapons system96 at OSD Oversight of either the board, committee, or working group level. However, embed- Embedded Computer ded computer resources are not a discrete area of focus. Instead, the reviews concentrate on whether a weapons system as a whole is ready Resources - A to proceed into the more advanced stages of development or production. Detailed Description Such an approach raises the question of whether embedded computer resources are being recognized for what they are-a very risky element of weapons system development that can largely determine whether the weapon system succeeds or fails. Responsibility for embedded computer resource oversight does not reside with one organization. Rather, it is shared by various OSD organi- zations, most of which are under the Under Secretary of Defense (Acqui- sition). These organizations participate in the Defense Acquisition Board review process as members of the board16or its committees and working groups. For example, the Deputy Director for Defense Research and Engineering (Test and Evaluation) is responsible for overseeing a weap- ons system’s developmental testing, including testing associated with the development of the system’s embedded computer resources. Action officers within this organization perform the day-to-day analysis and monitoring of testing, including that associated with the program’s embedded computer resources, and prepare position papers for the Dep- uty Director, who is a member of each of the three principal committees, and for the Director for Defense Research and Engineering, who is a member of the board. 15A major weapons system is one that is expected to cost $200 million for research, development, test, and evaluation, or a total of $1 billion for procurement. ‘%urrent membership includes the Under Secretary of Defense (Acquisition) as chairperson; the Vice Chairman, Joint Chiefs of Staff as vice chairperson; the Director, Defense Research and Engineering; the Acquisition Executives from the Army, Navy, and Air Force; the Assistant Secretary of Defense (Program Analysis and Evaluation); the Comptroller, Department of Defense; the Director, Opera- tional Test and Evaluation; and the chairperson of the cognizant committee. Page 21 GAO/lMTEG9O-24 DOD Embedded Computer Technology , Appendix II DOD Embedded Computer Resources and OSD Oversight of Them While the Defense Acquisition Board process allows for issues, including embedded computer resource issues, to be raised at any level, it encour- ages resolution of issues at the lowest level possible. Issues that cannot be resolved are forwarded to the next level for consideration. An official in the Office of the Deputy Director, Defense Research and Engineering (Strategic and Theater Nuclear Forces) told us that the Defense Acquisi- tion Board has consistently held that the board should not be burdened with issues that can be resolved at lower levels. According to this offi- cial, “the best board meeting is one that just ratifies committee positions.” Embedded computer resource issues are not a discrete area of focus at any of the levels. According to an official in the Office of the Director for Program Integration who regularly attends Defense Acquisition Board meetings, such issues are not generally discussed by the board because members tend to focus on matters with which they are comfort- able. On the basis of this official’s recollection of the over 200 board and its predecessor meetings attended over the last 11 years, embedded com- puter resource issues have been addressed only once. Office of the Director for Program Integration statistics show that the most frequent issues addressed by the board are, in order of frequency, (1) acquisition strategy, (2) affordability, (3) cost growth and control, (4) test results, (5) military requirements, (6) joint service disputes, (7) threshold breach, (8) inefficient production rate, and (9) alternative program tradeoffs. Other OSD officials also told us that senior DOD management tends to focus on issues with which they are familiar and comfortable, and embedded computer resource issues do not fit into this category. For example, an official in the Office of the Deputy Director for Defense Research and Engineering (Research and Advanced Technology) stated that lack of knowledge about computer technology has led oversight officials to not question embedded computer resources when reviewing weapons systems. Embedded computer resource issues may be addressed and resolved at committee or working group levels, and thus never require the board’s attention. However, statistics are not maintained on the types of issues addressed at these two levels. Available records documenting committee and working group deliberations focus on unresolved issues upon which the next level should focus. Although our review of these records for three of the nine weapons systems we surveyed showed little evidence of embedded computer resources being addressed, issues concerning these systems’ computers still could have been addressed and resolved. Officials in weapons system program offices and in OSD oversight offices Page 22 GAO/IMTJXG90-34 DOD Embedded Computer Technology Appendix II DOD Embedded Computer Resources and OSD Oversight of Them that we interviewed stated that embedded computer resource issues are not typically raised at the committee and working group levels. Accord- ing to these officials, OSD takes a holistic approach to overseeing weap- ons systems. That is, OSD is concerned about the weapons system as a whole and whether the total system is meeting overall reliability, availa- bility, and maintainability requirements. If the integrated weapons sys- tem is meeting requirements, they said, there is no reason to focus on a specific subsystem or its embedded computer resources. Case Studies Illustrating To illustrate how OSD’S process for overseeing embedded computer resources has worked, we selected three weapons systems for case stud- the Oversight Process ies. The three systems are the C-17A, Trident II (D-5) missile, and For- ward-Area Air Defense System Line-of-Sight Forward Heavy Armored Track Vehicle. Our findings are summarized below. C-l 7A Airlifter OSD oversight of the C-17A occurs through the Defense Acquisition Board, the Conventional Systems Committee, and the C-17A working group. However, the system’s embedded computer resources do not appear to be an area of focus. According to the working group chairman, the group generally meets quarterly to receive briefings from the Air Force, and the committee meets once a year. The working group chair- man told us that since the C-17A is a Defense Enterprise Program,17 OSD’S role is to “not get too involved.” The chairman also told us that the oversight OSD does perform is not technically oriented, focusing more on such issues as cost, logistics, and functionality. Further, the chairman said that OSD seldom identifies a problem. According to the chairman, the working group designated two software development areas-the mission computer and the electronic flight con- trol system, which are critical to the performance of the C-17A’s mis- sion-as being potentially difficult, and the working group has received briefings from the Air Force on software development schedules. The results of recent briefings on these areas were forwarded in a program status report to the chairman of the Conventional Systems Committee. The committee in turn notified the Defense Acquisition Board prior to the last board review meeting that considerable schedule risk remained in the mission computer software. According to the working group chairman, although software development schedule concerns were raised in this instance, computer resources issues are not addressed at 17A congressional initiative to streamline the acquisition process. (See 10 USC. 2436.) Page 23 GAO/lMTEG90-34 DOD Embedded Computer Technology Appendix II DOD Embedded Computer Resources and OSD Oversight of Them every working group meeting. The chairman also stated that weapons system computer resources is a growing concern that perhaps a separate oversight group should address. C-17A program officials stated that OSD’S oversight should focus on overall compliance with policy guidance. They stated that OSD cannot get involved deeply enough in overseeing embedded computer resources because it has too few people and too little time. As a result, they con- cluded that OSD should limit its oversight to determining whether devel- opment and acquisition of the weapons system as a whole should proceed. Trident II (D-5) Missile OSD oversight of the Trident II occurs through the Defense Acquisition Board, the Strategic Systems Committee, and the Trident II working group. However, the missile’s embedded computer resources do not appear to be an area of focus. According to the Trident program mana- ger, the missile’s embedded computer resources have not been discussed at any board meetings. Additionally, an official in the Office of the Dep- uty Director for Defense Research and Engineering (Strategic and Thea- ter Nuclear Forces) responsible for the day-to-day oversight of missile programs told us that the working group has met once in the last 18 months, and this meeting focused on missile test failures, which are not considered computer resource-related. However, another official in this office stated that the office has overseen the development of the pro- gram continually, and has briefed board officials on the program prior to milestone reviews. We reviewed the briefing provided for the last milestone review and found no mention of embedded computer resources. According to the official, OSD oversight is, by necessity, “man- agement by exception,” and the Trident II’s embedded computer resources have not been a problem and thus have not been specifically addressed. According to program officials, the only attention OSD has paid to the missile’s embedded computer resources occurred when the Office of the Deputy Director, Defense Research and Engineering (Test and Evalua- tion) was briefed on the Trident II’s software development plan. The officials stated that OSD was interested in whether the program office had a coherent plan to address testing and quality assurance and whether resources were adequate. No documentation of this meeting was available, and no OSD direction resulted from it. Page 24 GAO/lMTEG90-34 DOD Embedded Computer Technology Appendix II DOD Embedded Computer Resources and OSD Oversight of Them Forward-Area Air Defense ~SD oversight of the Forward-Area Air Defense System Line-of-Sight System Line-of-Sight Forward Heavy armored tracked vehicle occurs through the Defense Acquisition Board and the Conventional Systems Committee. However, Forward - a--w-eHeavy Armored the system’s embedded computer resources have not been an area of Track Vehicle focus. We reviewed documentation for the last briefing to the committee as well as correspondence from the committee to the board and found no mention of embedded computer resource issues. According to an official in the Office of the Deputy Director for Defense Research and Engineer- ing (Tactical Warfare Programs), even though documentation does not show that such issues were addressed, they could have been raised at any point during the briefing. According to program officials, OSD’S limited involvement in the program stems from the fact that the system is a non-development item, devel- oped with private funds for another purpose and later sold to the Army. These officials stated that OSD’S Office of the Director for Operational Test and Evaluation has reviewed the program’s Test andEvaluation Master Plan, and has described the annex for software test and evalua- tion as the best one the office has seen. Page 26 GAO/IMTEG9O34 DOD Embedded Computer Technology - I ,. L Appendix III Major Contributors to This Report James R. Watts, Associate Director Information John B. Stephenson, Assistant Director Management and Randolph C. Hite, Evaluator-in-Charge Cheryl A. Dottermusch, Computer Scientist Technology Division, Washington, D.C. (510451) Page 26 GAO/IMTECM DOD Embedded Compubr Tedmology
DOD Embedded Computers: Better Focus on This Technology Could Benefit Billion Dollar Weapons Programs
Published by the Government Accountability Office on 1990-04-19.
Below is a raw (and likely hideous) rendition of the original report. (PDF)