United States General Accounting Office GAO Report to the Director, Army Logistics Systems Support Center September 1997 DEFENSE COMPUTERS LSSC Needs to Confront Significant Year 2000 Issues GAO/AIMD-97-149 United States GAO General Accounting Office Washington, D.C. 20548 Accounting and Information Management Division B-277732 September 26, 1997 Mr. Michael A. Whitelaw Director, Logistics Systems Support Center 1222 Spruce Street St. Louis, Missouri 63103 Dear Mr. Whitelaw: On August 22, 1997, we briefed you on the results of our review to date of the Logistics Systems Support Center’s (LSSC) program for solving the Commodity Command Standard System (CCSS) Year 2000 computer system problem.1 The problem results from the inability of computer programs at the year 2000 to interpret the correct century from a recorded or calculated date having only two digits to indicate the year. Unless corrected, this problem could cause the CCSS system to malfunction or produce incorrect information when the year 2000 is encountered during automated data processing. Since CCSS supports the Army’s wholesale logistics supply management business area, which procures supplies and equipment totaling over $23 billion annually for Army forces around the world, LSSC’s inability to ensure that CCSS is Year 2000 compliant could result in a loss of operational support that would be widespread, costly, and potentially debilitating to important Army and other DOD agency missions. Our briefing was based on work we performed as part of our review of the Department of Defense’s (DOD) Year 2000 computer systems effort for the Chairman, Senate Governmental Affairs Committee; the Chairman and Ranking Minority Member of the Subcommittee on Government Management, Information and Technology, House Committee on Government Reform and Oversight; and the Honorable Thomas M. Davis, III, House of Representatives. During our review, we concentrated on determining (1) the status of LSSC’s efforts to correct CCSS Year 2000 system problems and (2) the appropriateness of LSSC’s strategy and actions for ensuring that CCSS Year 2000 issues will be successfully addressed. This letter summarizes the concerns we raised and provides recommendations for addressing these issues. 1 Although CCSS is considered to be one system, LSSC reports that it actually consists of 561 subsystems and supporting applications which contain about 10.2 million lines of code—about 5.5 million of which are being reported by LSSC as potentially requiring renovation. Page 1 GAO/AIMD-97-149 Army LSSC Year 2000 B-277732 The Year 2000 problem is one of the most comprehensive and complex Results in Brief information management projects ever faced by LSSC. If not successfully completed, the procurement of weapon systems and their spare parts, accounting for the sales of Army equipment and services to allies, and the financial management of $9 billion of inventory could be disrupted. As a result, it could be extremely difficult to efficiently and effectively equip and sustain the Army’s forces around the world. To its credit, LSSC has completed several actions to address the CCSS Year 2000 problem. For example, LSSC has inventoried software lines of code, established a Year 2000 task force, and acquired automated software assessment tools. In addition, a Year 2000 project manager and management staff have been designated, a project charter and schedule were developed, and supplementary contractor support was acquired to assist with assessment tasks. Further, regularly scheduled quarterly meetings are held by Army Materiel Command (AMC) headquarters to report LSSC Year 2000 status. These steps are compatible with DOD’s suggested approach and consistent with those found in our five-phased approach for planning, managing, and evaluating Year 2000 projects. Although LSSC commenced its Year 2000 project over a year ago, there are several critical issues facing LSSC that, if not completely addressed, may result in the failure of CCSS to successfully operate at the year 2000. Specifically, LSSC has yet to completely address (1) competing workload priorities and staffing issues, (2) the appropriate mix and scheduling of needed testing data and expertise as well as the development of test plans, (3) the scope and substance of written interface agreements with system interface partners to ensure that CCSS subsystems will be capable of exchanging data at the year 2000, and (4) contingency plan development to help assure that Army’s missions will be accomplished if CCSS is not fully available to users by the year 2000. LSSC’s risk of failure is increased because the agency has not attained the level of software development and maintenance maturity that would provide the foundation needed for successful management of large-scale projects such as the Year 2000 initiative. Because CCSS is used to support military readiness, these critical elements must be resolved and aggressively pursued to enable LSSC to successfully achieve a Year 2000 compliant environment prior to the year 2000. Page 2 GAO/AIMD-97-149 Army LSSC Year 2000 B-277732 In conducting our review, we compared LSSC’s progress in planning and Scope and managing its Year 2000 project to our Year 2000 Assessment Guide.2 We Methodology also reviewed DOD’s Year 2000 Management Plan,3 Department of the Army and Army Materiel Command (AMC) Year 2000 guidance,4 and private industry Year 2000 guidance. We focused our review on Year 2000 work performed by (1) LSSC—the designer, developer, and maintainer of CCSS, and (2) AMC—the Army major command responsible for promulgating Year 2000 policy and guidance and providing assistance to its major subordinate commands and central design activities. To determine the status of LSSC’s Year 2000 project and the appropriateness of its strategy and actions for ensuring successful completion, we interviewed LSSC’s Year 2000 Project Manager, Project Officer, and Focal Point who are responsible for project management, direction, and reporting. We also interviewed the AMC Year 2000 project team and the AMC Year 2000 Logistics Systems Chair. We obtained and analyzed Year 2000 guidance as well as documentation from the CCSS Configuration Control Board, AMC quarterly progress reviews, and AMC Year 2000 Logistics Task Force meetings to determine Year 2000 plans, strategy, and status for each of the five phases. We obtained and discussed software change schedules, workload statistics, and testing procedures and testing resource information with LSSC’s Quality Assurance Division Chief. We discussed software change procedures with the Technical Data Systems Division Chief and impact and workload issues with the Asset Management Systems Chief. To compare LSSC’s workload with its available staff resources, we obtained and discussed staffing information with the agency’s Resources Management Director, Budget and Manpower Division Chief, and the Business Information Systems Director. To determine cost estimates for the project, we interviewed the Year 2000 Project Manager and obtained and analyzed LSSC documents pertaining to cost. We also discussed LSSC’s software maturity capability and efforts to improve its maturity level with the Year 2000 Project Manager and the computer specialist tasked with improving LSSC’s software maturity capability. We conducted our work primarily at the Logistics Systems Support Center in St. Louis, Missouri, and at the U.S. Army Materiel Command in Alexandria, Virginia. Our audit work was performed from December 1996 2 Year 2000 Computing Crisis: An Assessment Guide (Exposure Draft) (GAO/AIMD-10.1.14, February 1997). 3 Department of Defense Year 2000 Management Plan (Version 1.0, April 1997). 4 Army Project Change of Century Action Plan (Revision I, October 1996) and the U.S. Army Materiel Command Year 2000 Action Plan (May 1997). Page 3 GAO/AIMD-97-149 Army LSSC Year 2000 B-277732 through August 1997 in accordance with generally accepted government auditing standards. The Department of Defense provided written comments on a draft of this report. These comments are discussed in the “Agency Comments and Our Evaluation” section and are reprinted in appendix I. LSSC is one of several central design activities (CDAs) for the Army Materiel Background Command (AMC).5 LSSC’s major responsibility is to design, develop, deploy, integrate, and maintain the Commodity Command Standard System (CCSS), a standard automated wholesale logistics system supporting AMC and other Army and DOD organizations. CCSS performs stock control, supply management, cataloging, provisioning, procurement, maintenance, security assistance, and financial management over an inventory of supply items for these organizations. It is the business automation core of AMC’s commodity commands and is linked to other Army logistics systems, such as the Continuing Balance System-Expanded (CBSX).6 CCSS’ financial module also provides general accounting, inventory accounting, billing support, and general ledger and financial reports for both reimbursable and non-reimbursable issues. As one of the world’s largest integrated business systems, CCSS comprises 561 separate subsystems that contain 10.2 million lines of program code in about 5000 programs. These subsystems work collectively to process an annual procurement budget for supplies and equipment of over $23 billion. The Year 2000 problem is rooted in the way dates are recorded and computed in automated information systems. For the past several decades, systems have typically used two digits to represent the year, such as “97” representing 1997, in order to conserve electronic data storage and reduce operating costs. However, with this two-digit format, the year 2000 is indistinguishable from 1900, as is 2001 indistinguishable from 1901. As a result of this ambiguity, system or application programs that use dates to perform calculations, comparisons, or sorting may generate incorrect results when working with years after 1999. 5 AMC is headquartered in Alexandria, Virginia, and is responsible for developing, buying, and maintaining equipment and supplies for U.S. soldiers and allies worldwide. 6 The Continuing Balance System-Expanded (CBSX) is the Army’s central logistics system for reporting the types, quantities, and locations of equipment, generally major end items such as helicopters and tanks. Page 4 GAO/AIMD-97-149 Army LSSC Year 2000 B-277732 LSSC staff recognized the significance of this anomaly in 1991, and at an estimated cost of $5.9 million, recommended making all CCSS date fields Year 2000 compliant in accordance with Army Regulation 25-9.7 The regulation specified that date fields were to be 8 positions in length (YYYYMMDD) including two century positions to be populated with normal values of “19” or “20.” LSSC has requested funding for the recommended changes every year since 1991; however, funding was denied because CCSS was a legacy system designated for replacement by other systems under the DOD Corporate Information Management (CIM) initiative. By 1994, however, it became apparent that emergency system changes would be necessary to allow certain CCSS subsystems to continue forecasting requirements beyond 1999. LSSC reports that, since 1994, it has renovated at least 3.8 million lines of code to accommodate the year 2000. However, initial funding for completing the CCSS Year 2000 effort, now estimated at over $12 million, was not approved until January 1997. The impact of the year 2000 on CCSS is substantial since CCSS is heavily date dependent. Date fields are used in nearly all CCSS subsystems, files, databases, and data used for status accounting, computations, forecasting, financial accounting, and requisition processing. Consequently, faulty turn of the century date processing would significantly impair the Army’s ability to order, manage, sell, and account for commodities such as ammunition, communications, and electronics. In turn, through its other logistics systems connections, it could also impair the Army’s ability to track and manage major end items such as aircraft, missiles, and tanks, as well as the many thousands of repair parts that support them. Because CCSS is the Army’s wholesale logistics system, a loss of CCSS operational support to AMC and other DOD agencies poses a serious threat to overall mission capability. For example, if dates are not processed accurately in CCSS applications that support inventory management and requisition processing, items ordered on or after January 1, 2000, could be identified as 99-year old excess inventory and become candidates for disposal. The cost of such faulty date processing would be great considering the (1) cost of the inventory item, (2) administrative costs involved in requisitioning, shipping, handling, and accounting for the item in the various financial, inventory, and transportation subsystems, and (3) costs associated with designating the item as excess inventory for disposal and the subsequent physical disposal of the item. Such an occurrence could severely impair overall military readiness since the necessary items would not be available for the soldier in the field. More 7 Army Data Management and Standards Program, AR 25-9, effective October 24, 1989. Page 5 GAO/AIMD-97-149 Army LSSC Year 2000 B-277732 importantly, soldiers and military civilians may not be able to properly maintain or replace weapon systems components, which could result in injury or death. Also, military equipment maintenance and overhaul facilities could be temporarily closed for lack of spare parts. In February 1997, we published the Year 2000 Computing Crisis: An Assessment Guide, which addresses common issues affecting most federal agencies and presents a structured approach and checklist to aid in the planning, managing, and implementing of Year 2000 projects. The guide describes five phases —supported by program and project management activities —with each phase representing a major Year 2000 program activity or segment. The guidance draws heavily on the work of the Best Practices Subcommittee of the Interagency Year 2000 Committee, and incorporates guidance and practices identified by leading organizations in the information technology industry. The five phases are consistent with those prescribed by DOD in its Year 2000 Management Plan. The phases and a description of each phase follows: • Awareness—Define the Year 2000 problem and gain executive-level support and sponsorship. Establish a Year 2000 program team and develop an overall strategy. Ensure that everyone in the organization is fully aware of the issue. • Assessment—Assess the Year 2000 impact on the enterprise. Identify core business areas and processes, inventory and analyze systems supporting the core business areas, and rank their conversion or replacement. Develop contingency plans to handle data exchange issues, lack of data, and bad data. Identify and secure the necessary resources. • Renovation—Convert, replace, or eliminate selected platforms, applications, databases, and utilities. Modify interfaces. • Validation—Test, verify, and validate converted or replaced platforms, applications, databases, and utilities. Test the performance, functionality, and integration of converted or replaced platforms, applications, databases, utilities, and interfaces in an operational environment. • Implementation—Implement converted or replaced platforms, applications, databases, utilities, and interfaces. Implement data exchange contingency plans, if necessary. In addition to following the five phases described above, the Year 2000 program should also be planned and managed as a single large information system development effort. Agencies should promulgate and enforce good management practices on the program and project levels. Page 6 GAO/AIMD-97-149 Army LSSC Year 2000 B-277732 LSSC is the Army component responsible for applying the Year 2000 Current Status of five-phased resolution process to CCSS. As such, in July 1996, LSSC initiated LSSC’s Year 2000 a project to address CCSS Year 2000 processing issues. As of July 1997, LSSC Project has completed a number of activities associated with the awareness and assessment phases of the process, including identifying its inventory, establishing a Year 2000 project team, and assessing the date impact on CCSS’ 10.2 million lines of code. LSSC has identified that as much as 54 percent or 5.5 million lines of code may be impacted by the year 2000 due to the fact that entire applications may need to be corrected to accommodate the date change. LSSC officials stated that they still need to determine how specific code will be changed in affected applications. LSSC also reported that an additional 3.8 million lines of code have already been renovated but still need to undergo integrated and regression testing. LSSC plans to implement the Year 2000-compliant CCSS by November 1998 at a cost of over $12 million. Prior to receiving funding in January 1997, the Year 2000 project remained in the awareness phase. During the awareness phase, LSSC completed tasks such as assembling technical and functional representatives into a Year 2000 task force, evaluating automated software assessment tools, and identifying the number of software lines of code. Once the project was officially funded and entered the assessment phase, LSSC officials appointed the project manager and management staff. Also, the Year 2000 project team prepared a project charter and schedule, secured contractor support to assist with assessment tasks, and began to determine the date impact on CCSS program code. As project activity proceeded, project staff routinely reported Year 2000 progress to the AMC Deputy Commanding General, AMC Year 2000 Logistics Task Force, Communications-Electronics Command (CECOM) Year 2000 Project Office, and the CCSS Configuration Control Board. To support project management, LSSC’s Year 2000 project manager drafted a plan which initially did not conform to DOD’s recommended Year 2000 five-phased approach, although the plan did identify some tasks typically associated with Year 2000 projects. For example, the plan included such tasks as beginning risk assessment and contingency plan development, providing assessment tool training, conducting an inventory of CCSS applications, and obtaining contractor support for date impact assessment. As a result of our concerns that the plan did not clearly specify or identify key Year 2000 phases and associated tasks, LSSC’s Year 2000 project manager later revised the plan in an attempt to better identify phases and tasks in accordance with DOD’s five-phased approach. Page 7 GAO/AIMD-97-149 Army LSSC Year 2000 B-277732 In addition to assessing the lines of code for CCSS, LSSC reported that it had cataloged the applications, modules, functional areas served, and languages used. LSSC had also determined that all the source code for CCSS was available and matched production code. In addition, LSSC had acquired automated assessment tools to help identify affected and obsolete code and trained LSSC staff to use these tools for the assessment, renovation, and validation phases. Since the exchange of data with other systems through external interfaces creates the potential to introduce or propagate errors from one system to another, LSSC identified 57 other systems which interface with CCSS and is in the process of confirming data exchange requirements with the external system owners. LSSC also developed a standard memorandum of agreement (MOA) to document coordination of data exchange requirements.8 Since CCSS and its interfacing partners plan to use procedural code9 and sliding window techniques10 to correct the Year 2000 problem, any date formats exchanged would be properly converted through internal program coding changes rather than changes to date formats. As part of its assessment of the level of date impact on CCSS, LSSC assessed the risk of not preparing CCSS for the year 2000. LSSC reported that CCSS, as a whole, is not Year 2000 compliant and that a catastrophic failure of the Army wholesale logistics mission would occur if CCSS is not made compliant. LSSC further reported that no known commercial or government replacements exist for CCSS functionality and that renovation of existing CCSS code was essential to mitigate the risk of failure. In May 1997, LSSC was still addressing the assessment phase activities of identifying a renovation strategy and developing a validation strategy and schedule for testing. According to DOD’s Year 2000 Management Plan and 8 LSSC’s approach to data exchange and its proposal to the external system owners consists of keeping date formats in the current two-digit year configuration in accordance with existing military data formatting standards. CCSS and nearly all DOD logistics systems conform with military data formatting standards which require year dates to be input, stored, and transmitted in a two-digit configuration. While use of the four-digit year field is now the preferred standard for system interfaces, DOD recognizes that a two-digit year field is an acceptable configuration when accompanied by signed MOAs documenting the data exchange requirements of the interfacing partners. 9 Procedural code is code which derives the correct century based on the two-digit year (e.g., any year smaller than year 50 is a 2000 date, and any year 50 or larger is a 1900 date). 10 Sliding windows are similar to procedural code in that they derive the correct century based on the two-digit year, but the numeric constant used to determine the century changes each year. Using the procedural code example above, in the current year, 50 or larger would be a 1900 date, while next year, 51 or larger would be a 1900 date. Page 8 GAO/AIMD-97-149 Army LSSC Year 2000 B-277732 AMC’s Year 2000 Action Plan, the validation strategy should identify the general time frames for the validation of all information resources and include consideration of hardware concerns such as availability of processing cycles and storage as well as human resource issues. In addition, efforts were ongoing to contract with a vendor to perform automated code correction on some CCSS subsystems. To its credit, LSSC recognized the problems inherent in the century date LSSC Must Address change and began seeking funding to address Year 2000 issues years ago. Key Year 2000 Issues However, although some progress has been made, several key project to Increase Chances management actions associated with the assessment phase have not been completed. As a result, LSSC is not presently well-positioned to move of Success forward to the more difficult phases of renovation, validation, and implementation in the Year 2000 process—phases that industry experts estimate could consume as much as three-fourths of Year 2000 project time and resources. LSSC still needs to take a number of actions to increase its chances of success, including (1) managing competing workload priorities, (2) planning for testing, (3) clarifying and coordinating written systems interface agreements, and (4) developing a contingency plan. To increase its chances of successfully managing its Year 2000 program, LSSC will also need to institutionalize a repeatable software change process that can be used from project to project. If these areas are not addressed soon, LSSC could find itself limited in its ability to meet the turn of century date. Given the prominence of date processing in CCSS and its central mission of sustaining the soldier in the field, LSSC cannot afford to delay any longer, and needs to demonstrate that it will perform, all the key actions associated with sound Year 2000 planning and management. LSSC Should Initiate In 1991, the Software Engineering Institute (SEI)11 introduced the Actions to Improve Its Capability Maturity Model (CMM) to assist organizations in assessing the Software Capability maturity level of their software development and maintenance processes. In general, software process maturity serves as an indicator of the likely Maturity range of cost, schedule, and quality of results that can be expected to be achieved by projects within a software organization. Our Year 2000 Assessment Guide points out that few activities within federal agencies operate above CMM level 1, and as a result, organizations lack the basic policies, tools, and practices necessary to successfully manage large-scale efforts. A CMM level 1 is the lowest level and is characterized by a software 11 The Software Engineering Institute (SEI) is a federally funded research and development center sponsored by DOD and operated by Carnegie Mellon University in Pittsburgh, Pennsylvania. Page 9 GAO/AIMD-97-149 Army LSSC Year 2000 B-277732 process that is ad hoc, and occasionally even chaotic. Few processes are defined and success depends on individual effort. We have recommended that federal agency information technology organizations be at least a CMM level 2 which is characterized by an established software development process discipline that is repeatable from project to project. In 1994, LSSC’s software development process was assessed by a team of LSSC and SEI-licensed contract staff. Using the CMM methodology, the team determined that LSSC should be ranked at a level 1 maturity. The assessment results concluded that LSSC lacked the basic software management practices necessary for repeatable software project success. The team also indicated that level 2 maturity could be attainable with a modest effort. Accordingly, the assessment team made recommendations that, if implemented, could provide the basis for LSSC’s attainment of a level 2 maturity. Based on the team’s assessment, LSSC developed an action plan to address the identified deficiencies. However, according to LSSC officials, the action plan was never implemented due to the reassignment of LSSC assessment staff, agency staff reductions, and lack of funding. After a period of nearly 3 years, LSSC resurrected the CMM assessment project in December 1996 to, once again, review the assessment team’s findings and recommendations and to propose follow-on actions to address the deficiencies. The review concluded that a project management system which would allow LSSC to better plan, estimate, and track software projects on an enterprise-wide basis was essential for LSSC to mature to a CMM level 2. While LSSC has an automated project management system under development, a member of the LSSC review said the system is inadequate because it is unable to track all software projects and may not address all level 2 requirements. This information was presented to LSSC’s Executive Steering Council in March 1997. However, at the time of our review, LSSC had made little progress in correcting the software process deficiencies and was still ranked at CMM level 1. Until LSSC moves on to the next CMM level, its ability to contend with the later stages of the Year 2000 effort will be constrained. Recently, LSSC officials informed us of their intent to obtain a CMM level 2 certification following completion of the Year 2000 project. Actions Needed to Mitigate In addition to lacking a mature software development and maintenance Impact of Competing process, LSSC now has 42 percent fewer staff available to make the needed Workload Priorities renovations to CCSS than it had in fiscal year 1990. Moreover, since fiscal year 1990, LSSC’s workload has increased, showing a notable jump in fiscal Page 10 GAO/AIMD-97-149 Army LSSC Year 2000 B-277732 years 1997 and 1998—the 2 years when the majority of Year 2000 actions need to be performed to enable agencies to have a realistic chance of meeting the turn of century date time frame. At the same time that its staff is decreasing and its workload is increasing, LSSC continues to be tasked with other software projects by the Lead AMC Integration Support Office (LAISO). Despite these indicators of potential problems, LSSC has only recently begun to take the steps necessary to augment its staff with contract support for the renovation phase and has yet to fully resolve staffing issues concerning the development of test plans. In addition, LSSC has not prioritized its software project schedule to provide the structure needed to keep the Year 2000 project on schedule and within cost estimates. Until these issues are addressed, they pose unnecessary risk to the success of LSSC’s Year 2000 project. As of June 1997, LSSC reported that it had devoted 7 of its 315 total staff to the Year 2000 project full-time. While four contract support staff had been retained to train LSSC staff to use the automated software assessment tools and help with impact assessment, these contractor staff have since been released. As of August 1997, no contract staff were on board to augment LSSC staff during the renovation phase, although steps were underway to obtain additional contract support and to obtain an automated code correction solution.12 Also, LSSC reported that staff would be tasked to work exclusively on the Year 2000 renovation phase after completing an ongoing major systems change project related to a Base Realignment and Closure (BRAC) decision.13 LSSC officials stated that as the BRAC-related renovation begins to diminish in September 1997, both LSSC and contract staff would be transferred to the Year 2000 project. While we do not question the appropriateness of performing the BRAC-related work prior to Year 2000 work, we are concerned that LSSC’s Year 2000 project approach does not provide for alternatives should the BRAC target completion schedule slip and the subsequent LSSC staff and contractors not become available. 12 LSSC is planning to utilize the services of a private contractor that provides an automated code conversion service to organizations seeking an automated method for correcting Common Business Oriented Language (COBOL) software code. Under this arrangement, LSSC will provide its software code to the contractor’s off-site facility where Year 2000 windowing logic is inserted using the contractor’s proprietary automated software. The converted code will then be returned to LSSC for compiling and testing. 13 The BRAC decision entails reducing the size of DOD and saving money by closing or consolidating DOD facilities. Since one of AMC’s major subordinate commands (MSC), located in St. Louis, Missouri, was dissolved and its functions moved to other MSCs as a result of the 1995 BRAC decision, it has been necessary for LSSC to develop conversion programs to extract aviation and troop support information and transfer the data to the MSCs now responsible for those functions. Page 11 GAO/AIMD-97-149 Army LSSC Year 2000 B-277732 Further, an examination of CCSS software release schedules since fiscal year 1990 shows that the number of projects has increased as much as five-fold. At the same time as the majority of CCSS Year 2000 actions are to be performed, LSSC’s schedule calls for 10 software change projects to be fielded in fiscal year 1997 and 8 in fiscal year 1998. These projects range in terms of complexity and magnitude from routine systems maintenance, which may require minimal effort, to the Year 2000 and BRAC projects, which call for comprehensive changes in many of CCSS’ subsystems. In past years, LSSC routinely accomplished two to four software change projects a year. This significant increase in workload will undoubtedly impact the CCSS Year 2000 project schedule for several reasons. First, LAISO, the workload manager for CCSS, has not ensured that competing projects do not adversely affect LSSC’s ability to complete the Year 2000 effort. Prioritization of projects could result in the postponement or cancellation of some of the competing projects. Second, LSSC has little historical experience dealing with a workload of this magnitude which is compounded by a workforce that has diminished significantly in recent years. LSSC Should Be Planning Prior to commencing the validation (testing) phase of its Year 2000 effort, for the Validation Phase LSSC needs to fully address two key issues regarding its testing requirements and capabilities. Specifically, LSSC should be planning now to (1) assure that enough staff with the appropriate background and experience are available to develop Year 2000 test data and transactions and to review test results and (2) assess whether enough time has been scheduled to perform Year 2000 testing. Without planning how it will address these issues now, LSSC is increasing the risk that CCSS will not be fully validated in time for the change of century date. According to AMC’s Year 2000 Action Plan, many agencies will need to establish test environments which are specific to future date testing and which have no possibility of corrupting or destroying production data.14 Since the current CCSS test files do not contain the necessary Year 2000 test conditions and data, LSSC will need to establish Year 2000-specific test files to certify that CCSS is Year 2000 compliant. Such test data and transactions are typically designed by functional staff knowledgeable of the CCSS business processes. These staff review the testing results to ensure that Year 2000 software changes have processed data correctly and that other 14 Production data are existing data and transactions processed by or generated from everyday, operational use of a computer system. Preparing and using production data for testing does not require knowledge of the internal logic of the software. As a result, production data may not test all the functions or logic paths desired and could produce inconclusive results. Page 12 GAO/AIMD-97-149 Army LSSC Year 2000 B-277732 data and processes have not been inadvertently changed during testing. According to a LSSC official and LSSC staffing statistics, however, there are fewer functional staff now available to identify the date fields in the test transactions or test data needed to ensure that CCSS business processes are not adversely affected by the Year 2000 software changes. Also, LSSC officials stated that they expect the availability of these staff to continue to decrease over the next few years as staff retire and agency staff reductions continue. Further, LSSC is not allowing enough time for Year 2000 testing. While LSSC officials assert that the complexity and scope of the Year 2000 project is about the same as the BRAC project, LSSC’s June 1997 systems change release schedule calls for far less time to test Year 2000 changes than it does for BRAC changes. For example, BRAC testing began in February 1997 and is scheduled for completion in September 1997. Year 2000 testing is scheduled to begin in September 1998 and end almost 8 weeks later in November 1998. A LSSC official acknowledged that the amount of time scheduled for Year 2000 testing is insufficient, but stated that the schedule will be revised once ongoing negotiations to acquire an automated code correction service are resolved. The official also stated that he fully expects the Year 2000 test schedule to greatly increase beyond the currently scheduled 8 weeks but that the increased test time should be offset by the reduced renovation time expected to be garnered by using the automated code correction service. Although LSSC believes that the automated code correction service should provide increased Year 2000 testing time, it could not provide documented analysis to support this conclusion. While LSSC believes it can increase its testing time without increasing the overall Year 2000 project time, we are not as confident given LSSC’s CMM level 1 ranking. Trying to compensate for unrealistic time schedules by either shortening earlier phases of a software change project or by lengthening overall project time is characteristic of level 1 organizations. Until LSSC realistically assesses its testing requirements, capabilities, and time schedules, effective Year 2000 project management will become increasingly difficult to achieve, and LSSC will increase the risk that it may be unable to meet the demand imposed by Year 2000 testing. Page 13 GAO/AIMD-97-149 Army LSSC Year 2000 B-277732 Interface Agreements Lack CCSS’ ability to successfully operate at the year 2000 hinges on the proper Basic Information for and timely exchange of data with other systems, both within the Army and Effective Management and with external Defense components. It is critically important during the Year 2000 effort that agencies ensure that interfacing systems have the Implementation ability to exchange data throughout the transition period and protect against the potential for introducing and propagating errors from one organization to another and ensure that interfacing systems have the ability to exchange data through the transition period. This potential problem may be mitigated through formal agreements between interface partners that describe the method of interface and assign responsibility for accommodating the exchange of data. Both the DOD Year 2000 Management Plan and AMC Year 2000 Action Plan place responsibility on component heads or their designated Year 2000 contact points to document and obtain system interface agreements in the form of memorandums of agreement (MOA) or the equivalent. Further, to help assure that interfaces continue to properly exchange data after systems are renovated for the year 2000, AMC has issued minimum MOA documentation requirements designed to produce consistency, assign accountability, and recognize a level of detail necessary for effective interface renovation among data exchange partners. While LSSC has developed MOAs to document interface specifics between CCSS and its interfacing systems and is in the process of finalizing those agreements with system owners, nearly all the MOAs lack basic information necessary for effective management and implementation of the interfaces. According to AMC Year 2000 guidance and the accompanying requirements of the standard MOA, the agreements are to specify the (1) points of contact for reporting progress and coordinating schedules and (2) date the agreement becomes effective. To successfully implement interface changes, these agreements should also communicate the type, form, and frequency of transactions exchanged, the windowing technique that is being used at each end of the interface, and the review process for monitoring interface renovation progress and reconciling differences. However, our review disclosed that 39 of 41 MOAs that LSSC had finalized as of July 1997 failed to fully follow AMC’s guidance or include other information necessary to ensure that LSSC can successfully communicate with interface partners. Our Year 2000 Assessment Guide stresses the importance of adequately addressing interface and data exchange issues. Without such information, the MOAs do not serve to communicate and coordinate the actions designed to help assure that Year 2000 changes are made properly and promptly by LSSC and its interfacing partners. Page 14 GAO/AIMD-97-149 Army LSSC Year 2000 B-277732 Timely and complete information on all systems interfaces that may be affected by Year 2000 changes is essential to the success of the LSSC Year 2000 compliance program. The amount of work required to coordinate the data being exchanged between systems must be known as early as possible and documented in written MOAs so that LSSC may complete renovation schedules, allocate resources, plan testing, and schedule implementation. Year 2000 Contingency The year 2000 represents a great potential for operational failure to CCSS Plan Needed for CCSS that could adversely impact core business processes as well as those of entities that depend on the CCSS system for information. To mitigate this risk of failure, our Year 2000 Assessment Guide, DOD’s Year 2000 Management Plan, and the Army’s Project Change of Century Action Plan suggest that agencies perform risk assessments and prepare realistic contingency plans that identify alternatives to ensure the continuity of core business processes in the event of operational failure. These alternatives could include performing automated functions manually or using the processing services of contractors. While LSSC has taken the first steps toward development of a contingency plan by assessing the level of risk to each business area that could be affected by processing errors and by determining how that risk can be mitigated or reduced, at the completion of our review, LSSC had not yet developed a contingency plan. Further, despite explicit guidance from DOD and the Army to develop contingency plans should Year 2000 corrections to CCSS not be completed in time, LSSC officials stated that no contingency plan would be developed for CCSS. They maintained that AMC does not require a contingency plan for CCSS because CCSS is not scheduled for replacement prior to the advent of the year 2000. While AMC’s Year 2000 Action Plan states that contingency plan development is only required for replacement systems and implies that all other systems are exempt, the AMC plan also states that the guidance, policy, and responsibilities identified in the Army’s Project Change of Century Action Plan are mandatory and are the basis of the AMC plan. Nevertheless, despite LSSC’s and AMC’s position that a contingency plan is not needed for CCSS because the system is not being replaced prior to the year 2000, the system still risks unanticipated operational failure. Without a contingency plan that identifies specific actions to be taken if CCSS fails at the year 2000, the procurement of weapon systems and their spare parts, accounting for the sale of Army equipment and services to allies, Page 15 GAO/AIMD-97-149 Army LSSC Year 2000 B-277732 and the financial management of $9 billion of inventory could be disrupted. As a result, the Army could be unable to efficiently and effectively equip and sustain its forces around the world. Given the dangers associated with an operational failure of this magnitude, LSSC needs the protection provided by good contingency planning to ensure that options are available if CCSS is not able to operate at the year 2000. Recently, LSSC officials stated that they have begun preparing an initial contingency plan, which they estimate will be completed by September 30, 1997. If CCSS cannot correctly process dates on and after January 1, 2000, Conclusions military equipment, such as tanks, artillery, aircraft, missiles, munitions, trucks, electronics, and other supporting materials for the soldier, in all likelihood, will not be ordered, stored, transported, issued, paid for, or maintained. Mobilization plans and contingencies would be significantly impaired if materiel is delayed. However, LSSC has yet to resolve several critical problems associated with the assessment phase to ensure that (1) systems are adequately tested, (2) contingency plans are developed, and (3) interface partners are fully aware of LSSC’s Year 2000 plans. Furthermore, during the same time that LSSC is addressing the Year 2000 issue, the agency is also working to implement considerably more software projects than it has in the past. This unprecedented workload is compounded by a reduced staff level and LSSC’s basic lack of a mature software development and maintenance process. Together, these factors raise the risk level of the Year 2000 project beyond what is normally expected of a software modification effort of this magnitude. Until these problems are resolved, LSSC is not well-positioned to move forward into the more time-consuming phases of renovation, validation, and implementation. As a result, we believe LSSC will find it increasingly difficult to prepare CCSS in time for the arrival of the year 2000. We recommend that you: Recommendations • Act to improve LSSC’s software development process that will provide the basis for achieving CMM level 2 maturity. • Immediately assess the impact of competing workload and staffing demands on the CCSS Year 2000 project. Based on this assessment, consider (1) canceling or deferring less critical software projects until after the Year 2000 project is substantially completed and (2) augmenting Page 16 GAO/AIMD-97-149 Army LSSC Year 2000 B-277732 the Year 2000 project with staff having the necessary skills to ensure timely completion of the project. • Ensure that LSSC has the capability to complete the testing of all CCSS subsystems and programs. Specifically, LSSC should (1) determine test requirements, (2) identify the testing staff needed, (3) finalize Year 2000 test plans describing how the testing staff will be acquired and scheduled for developing Year 2000 compliant test scenarios and data, and (4) revise the Year 2000 test schedule to assure that enough time is available to meet Army-mandated deadlines for Year 2000 implementation. • Ensure that written interface agreements describe the method of data exchange between interfacing systems, name the entity responsible for performing the system interface modification, and state the completion date. • Develop a contingency plan that includes specific actions for ensuring that the Army’s logistic functions continue to operate at appropriate levels if all or part of CCSS fails to work at the year 2000. In written comments on a draft of this report, the Office of the Under Agency Comments Secretary of Defense (Acquisition and Technology) concurred with all of and Our Evaluation our recommendations to improve the Army’s LSSC Year 2000 program. Specifically, DOD agreed that a contingency plan would be developed by September 30, 1997, to ensure continuity of operations if all or part of CCSS fails to operate by the year 2000. DOD also outlined a number of actions that have recently been initiated that are aimed at reducing and prioritizing LSSC’s current workload, and increasing staff with the necessary skills to help ensure the timely completion of the Year 2000 project. In addition, DOD pointed to several actions, both taken and planned, to improve its capability to complete Year 2000 testing of CCSS subsystems and programs. While we have not reviewed LSSC’s latest actions, if properly implemented, we believe they could help resolve the workload and testing issues we identified. In concurring with our recommendation regarding the need to initiate actions to improve LSSC’s software development process, DOD recognized the value of achieving a CMM level 2 maturity and agreed that LSSC does not have all configuration management procedures in place to reach CMM level 2 at this time. However, DOD stated that LSSC’s history indicates that it can accomplish large projects successfully and that LSSC will meet the mandated dates for the BRAC and Year 2000 projects without achieving CMM level 2. After completion of these projects, LSSC plans to resume its efforts Page 17 GAO/AIMD-97-149 Army LSSC Year 2000 B-277732 to achieve a CMM level 2 maturity. We believe LSSC’s position comes at some risk. The discipline derived from reaching a CMM level 2 maturity can greatly enhance LSSC’s ability to address the Year 2000 challenge. This higher level of maturity is key to reducing the risk of schedule slippage, cost overruns, and poor software quality. As our report states, we have recommended that information technology organizations be at least a CMM level 2 to successfully manage large-scale projects such as the Year 2000 project. Our Year 2000 Assessment Guide provides interim actions that level 1 organizations can take prior to the year 2000 to minimize the risk of failure, such as training staff on proven industry system development and program management practices and soliciting assistance from organizational entities experienced in performing or managing major software conversions. LSSC could benefit from these interim actions. In concurring with our recommendation on strengthening written interface agreements, DOD stated that LSSC will formalize MOAs between interface partners. It also agreed to include specific detailed information in MOAs, but only when appropriate. As our report stated, we believe that, at a minimum, MOAs should also contain essential information for effective management of system interfaces, such as the type, form, and frequency of transactions exchanged, the windowing technique to be used, and the review process for monitoring interface renovation progress and reconciling differences. This additional information would help to ensure that interface partners are sufficiently prepared to handle unforeseen problems that may occur and to plan for contingencies. The full text of DOD’s comments is provided in appendix I. This report contains recommendations to you. Within 60 days of the date of this letter, we would appreciate receiving a written statement on actions taken to address these recommendations. We appreciate the courtesy and cooperation extended to our audit team by LSSC officials and staff. We are providing copies of this letter to the Chairmen and Ranking Minority Members of the Senate Committee on Governmental Affairs; the Subcommittee on Oversight of Government Management, Restructuring and the District of Columbia, Senate Committee on Governmental Affairs; and the Subcommittee on Government Management, Information and Technology, House Committee on Government Reform and Oversight; the Honorable Thomas M. Davis, III, House of Representatives; the Secretary of Defense; the Deputy Secretary of Defense; the Acting Under Secretary of Defense (Acquisition Page 18 GAO/AIMD-97-149 Army LSSC Year 2000 B-277732 and Technology); the Acting Under Secretary of Defense (Comptroller); the Acting Assistant Secretary of Defense (Command, Control, Communications and Intelligence); the Secretary of the Army; Commanders of the Army Materiel Command and Communications- Electronics Command; the Director of the Office of Management and Budget; and other interested parties. Copies will be made available to others upon request. If you have any questions on matters discussed in this letter, please call me at (202) 512-6240, or John B. Stephenson, Assistant Director, at (202) 512-6225. Major contributors to this report are listed in appendix II. Sincerely yours, Jack L. Brock, Jr. Director, Defense Information and Financial Management Systems Page 19 GAO/AIMD-97-149 Army LSSC Year 2000 Contents Letter 1 Appendix I 22 Comments From the Department of Defense Appendix II 27 Major Contributors to This Report Abbreviations AMC Army Materiel Command BRAC Base Realignment and Closure CBSX Continuing Balance System-Expanded CCSS Commodity Command Standard System CDA central design activities CECOM Communications-Electronics Command CIM Corporate Information Management CMM Capability Maturity Model COBOL Common Business Oriented Language DOD Department of Defense LAISO Lead AMC Integration Support Office LSSC Logistics Systems Support Center MOA memorandum of agreement MSC major subordinate command SEI Software Engineering Institute Page 20 GAO/AIMD-97-149 Army LSSC Year 2000 Page 21 GAO/AIMD-97-149 Army LSSC Year 2000 Appendix I Comments From the Department of Defense Note: GAO comment supplementing those in the report text appears at the end of this appendix. See comment 1. Page 22 GAO/AIMD-97-149 Army LSSC Year 2000 Appendix I Comments From the Department of Defense Page 23 GAO/AIMD-97-149 Army LSSC Year 2000 Appendix I Comments From the Department of Defense Page 24 GAO/AIMD-97-149 Army LSSC Year 2000 Appendix I Comments From the Department of Defense Page 25 GAO/AIMD-97-149 Army LSSC Year 2000 Appendix I Comments From the Department of Defense The following is GAO’s comment on the Department of Defense’s letter dated September 12, 1997. 1. DOD provided a number of clarifications to the report that we have GAO Comment incorporated as appropriate. Page 26 GAO/AIMD-97-149 Army LSSC Year 2000 Appendix II Major Contributors to This Report Ronald B. Bageant, Assistant Director Accounting and Carl M. Urie, Technical Advisor Information Brenda A. James, Senior Information Systems Analyst Management Division, Cristina T. Chaplain, Communications Analyst Washington, D.C. Denice M. Millett, Evaluator-in-Charge Kansas City Field Michael W. Buell, Staff Member Office (511627) Page 27 GAO/AIMD-97-149 Army LSSC Year 2000 Ordering Information The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. VISA and MasterCard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent. Orders by mail: U.S. General Accounting Office P.O. Box 37050 Washington, DC 20013 or visit: Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000 or by using fax number (202) 512-6061, or TDD (202) 512-2537. Each day, GAO issues a list of newly available reports and testimony. To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone. A recorded menu will provide information on how to obtain these lists. For information on how to access GAO reports on the INTERNET, send an e-mail message with "info" in the body to: firstname.lastname@example.org or visit GAO’s World Wide Web Home Page at: http://www.gao.gov PRINTED ON RECYCLED PAPER United States Bulk Rate General Accounting Office Postage & Fees Paid Washington, D.C. 20548-0001 GAO Permit No. G100 Official Business Penalty for Private Use $300 Address Correction Requested
Defense Computers: LSSC Needs to Confront Significant Year 2000 Issues
Published by the Government Accountability Office on 1997-09-26.
Below is a raw (and likely hideous) rendition of the original report. (PDF)