United States General Accounting Office GAO Report to the Chairman, Subcommittee on Oversight and Investigations, Committee on Veterans’ Affairs, House of Representatives September 1997 VETERANS BENEFITS MODERNIZATION VBA Has Begun to Address Software Development Weaknesses But Work Remains GAO/AIMD-97-154 United States GAO General Accounting Office Washington, D.C. 20548 Accounting and Information Management Division B-277370 September 15, 1997 The Honorable Terry Everett Chairman, Subcommittee on Oversight and Investigations Committee on Veterans’ Affairs House of Representatives Dear Mr. Chairman: The Veterans Benefits Administration (VBA), a component of the Department of Veterans Affairs (VA), is in the process of modernizing its information systems to improve its administrative operations and thereby upgrade the quality and timeliness of its services. Software development has been identified by many experts as one of the riskiest and most costly aspects of systems development. In June 1996, we reported and testified that VBA was operating at a level 1 software development capability, defined as ad hoc and chaotic.1 In its response to that report, VBA stated it would take actions to address the weaknesses we identified. At your request, we conducted a follow-up review to determine the actions taken by VBA to address management and technical weaknesses identified in the June 19, 1996, hearing on the agency’s modernization effort. This report, the second in a series on actions taken by VBA to address these weaknesses, summarizes our assessment of the agency’s actions to improve its software development capability.2 VBA has taken action to improve its software development capability. Results in Brief Among other things, it has launched a software process improvement initiative, chartered a software engineering process group, and obtained the services of an experienced contractor to assist in developing and implementing a software process improvement effort. Although it has made progress, VBA has not yet fully addressed needed software development improvements. These include a need for (1) a defined strategy to reach the repeatable level and a baseline to measure improvements, (2) a process improvement training program for its software developers, and (3) a process to ensure that VBA’s software 1 Software Capability Evaluation: VA’s Software Development Process Is Immature (GAO/AIMD-96-90, June 19, 1996) and Veterans Benefits Modernization: Management and Technical Weaknesses Must Be Overcome If Modernization Is to Succeed (GAO/T-AIMD-96-103, June 19, 1996). 2 The first report, Veterans Benefits Computer Systems: Risks of VBA’s Year-2000 Efforts (GAO/AIMD-97-79, May 30, 1997), discussed our assessment of VBA’s Year 2000 efforts. Page 1 GAO/AIMD-97-154 Veterans Benefits Modernization B-277370 development contractors are at the repeatable level. VBA generally agrees that these issues need to be addressed and has efforts underway to do so. Until these deficiencies are sufficiently addressed, VBA’s software development capability remains ad hoc and chaotic, subjecting the agency to continuing risk of cost overruns, poor quality software, and schedule delays in software development. Since 1986, VBA has been trying to modernize its old, inefficient Background information systems. It reportedly spent an estimated $294 million on these activities between October 1, 1986, and February 29, 1996. The modernization program can have a major impact on the efficiency and accuracy with which about $20 billion in benefits and other services is paid annually to our nation’s veterans and their dependents. Software development is a critical component of this modernization effort. Also, a mature software development capability will provide added assurance that software developers will be able to effectively make changes to the software needed to address the Year 2000 computing problem. To evaluate VBA’s software development processes, in 1996, we applied the Software Engineering Institute’s (SEI)3 software capability evaluation methodology to those projects identified by VBA as using the best development processes. This evaluation compares agencies’ and contractors’ software development processes against SEI’s five-level software capability maturity model, with 5 being the highest level of maturity and 1 being the lowest.4 In June 1996, we reported that VBA was operating at a level 1 capability. At this level, VBA cannot reliably develop and maintain high-quality software on any major project within existing cost and schedule constraints, which places VBA software development projects at significant risk. Accordingly, VBA must rely on the various capabilities of individuals rather than on an institutional process that will yield repeatable, or level 2, results. VBA did not satisfy any of the criteria for a repeatable or level 2 capability, the minimum level necessary to significantly improve productivity and return on investment. For example, 3 This is a nationally recognized, federally funded research and development center established at Carnegie Mellon University in Pittsburgh, Pennsylvania, to address software development issues. 4 The following are the five levels of the software capability maturity model: level 5 (optimizing) is continuous process improvement; level 4 (managed) is detailed measure collection and process/product control; level 3 (defined) is software process documentation, standardization, and integration; level 2 (repeatable) is software project cost, schedule and functionality tracking and repeatability of successes; and level 1 (initial) is ad hoc and chaotic software processes. The six key process areas for a repeatable capability are (1) requirements management, (2) project planning, (3) project tracking and oversight, (4) subcontract management, (5) quality assurance, and (6) configuration management. Page 2 GAO/AIMD-97-154 Veterans Benefits Modernization B-277370 VBA is extremely weak in the requirements management, software project planning, and software subcontract management areas, with no identifiable strengths or improvement activities. Because of VBA’s software development weaknesses, we recommended that the Secretary of Veterans Affairs • obtain expert advice to improve VBA’s ability to develop high-quality software; • develop and expeditiously implement an action plan that describes a strategy for reaching the repeatable (level 2) level of process maturity; • ensure that any future contracts for software development require the contractor to have a software development capability of at least level 2; and • delay any major investment in new software development—beyond what is needed to sustain critical day-to-day operations—until the repeatable level of process maturity is attained. In commenting on a draft of the June 1996 report, VBA agreed with three of our recommendations but disagreed with delaying major investments in software development. VBA stated that while it agreed that a repeatable level of process maturity is a goal that must be attained, it disagreed that “all software development beyond that which is day-to-day critical must be curtailed.” VBA stated that the payment system replacement projects, the migration of legacy systems, and other activities to address the change of century must continue. In our response to VBA’s comments, we agreed that the change of century and changes to legislation must be continued, and we characterized these changes as sustaining day-to-day operations. However, for those projects that do not meet this criterion, we continue to believe that VBA should delay software development investments until a maturity of level 2 is reached. To assess actions taken by VBA to improve its software capability, we Scope and reviewed VBA documents, such as its “Software Process Improvement Methodology Initiative Strategic Plan,” dated March 1997; the Best Practices Round Up Method; and the Interagency Agreement with the Air Force, dated September 1996, to obtain expert software process improvement assistance. We also reviewed SEI’s IDEALSM: A User’s Guide for Software Process Improvement, dated February 1996, and technical report on Best Training Practices Within the Software Engineering Industry, dated November 1996. In addition, we reviewed VBA contracts, correspondence Page 3 GAO/AIMD-97-154 Veterans Benefits Modernization B-277370 to contractors, and supporting documents to determine what VBA has done to ensure that VBA’s software development contractors are at the repeatable level. We interviewed VBA officials and contractor personnel involved with the software process improvement effort to determine what actions VBA has taken to improve its software capability. We also interviewed selected VBA project managers involved in new systems development on their knowledge of the software process improvement initiative. We performed our work from October 1996 through August 1997 in accordance with generally accepted government auditing standards. We requested comments on a draft of this report from the Secretary-Designate of Veterans Affairs. The Secretary-Designate provided us with written comments, which are discussed in the “Agency Comments and Our Evaluation” section and reprinted in appendix I. VBA has initiated several actions to improve its software development VBA Has Initiated capability. For example, in response to our recommendation that it obtain Actions to Improve Its expert assistance, VBA has hired a contractor—SEI—to (1) assist it in Software developing an integrated set of software practices that will position VBA for successful, lasting improvements, (2) help formulate a software Development improvement program, and (3) provide expertise in executing software Capability improvement activities. SEI is also expected to provide expertise in strategic and tactical planning, training, policy preparation, and action planning. In addition, in response to our recommendation that it develop and implement an action plan describing a strategy for reaching the repeatable level, VBA launched a software process improvement initiative in June 1996 to lay the foundation and build the context for sustainable, measurable improvements to its software development capability. It has developed a strategic plan that describes the purpose and goals of this improvement initiative. One of the plan’s goals is to establish organization policies and guidelines for the management, planning, and tracking of software projects that will enable VBA to repeat earlier successes on projects with similar applications. VBA has also recently initiated two software process improvement projects. The first project, called “Best Practices Round Up,” will identify software development practices that VBA software development teams are doing Page 4 GAO/AIMD-97-154 Veterans Benefits Modernization B-277370 correctly. VBA believes that there are “pockets of excellence” within its organization not identified in our June 1996 report that it can build upon. The Best Practices Round Up project team started its review in April 1997 and briefed VBA’s chief information officer on its results in August 1997. The second project, called “Standards, Policies, and Procedures,” will assess whether VBA’s software development teams are following current software development policies, procedures, and standards. This project was initiated in June 1997 and is expected to be completed by the end of September 1997. The project team is expected to make recommendations to ensure compliance with standards. Although VBA has launched a software process initiative and an Software Process accompanying strategic plan, VBA has not yet clearly presented how it Improvement intends to move from an ad hoc and chaotic level of software development Initiative Lacks capability to a repeatable level. SEI’s IDEALSM: A User’s Guide for Software Process Improvement requires that a plan be developed that includes a Specificity and a schedule for initial activities, basic resource requirements, and benefits to Baseline the organization. VBA’s current plan contains no milestones—beginning, interim, or completion dates—by which to measure the agency’s progress and to identify problems. The plan also contains no analysis or information on costs, benefits, or risks. VBA officials stated that they recognize that the agency’s strategic plan for software process improvement lacks this specificity. At the conclusion of our review, these officials said that VBA intends to address this area in an upcoming action plan for the software process improvement initiative. VBA has also not yet established a baseline from which to measure its software process improvements. According to the SEI IDEALSM: A User’s Guide for Software Process Improvement, an organization needs to understand its current software process baseline so that it can develop a plan to achieve the business changes necessary to reach its software process improvement goals. At the conclusion of our review, VBA officials told us that they plan to use as their baseline the results of our June 1996 report, along with the results from their Best Practices Round Up and Standards, Policies, and Procedures projects. They stated that the baseline should be established by September 1997. Page 5 GAO/AIMD-97-154 Veterans Benefits Modernization B-277370 Training of key staff is critical to achieving level 2 repeatability. According VBA Software Process to SEI’s technical report entitled, Best Practices Within the Software Improvement Training Engineering Industry, best training practices include defining a process for Plan Not Effectively software engineering education. Implemented Although VBA has provided process improvement training to many of the managers in its software engineering process group and management steering group,5 key software personnel—software developers, project managers, and line managers—have not been trained in the process improvement methodology, the principles behind it, and the key process areas. VBA’s software process improvement project manager explained that these key people had not yet been trained because VBA did not want to train them too long before implementing the process improvement projects. The project manager said that VBA plans to train these staff during fiscal years 1998 and 1999. However, VBA does not have a documented training plan to help ensure that these personnel receive training. Unless these individuals are trained in the process improvement methodology, its principles, and the key process areas, it will be difficult for them to implement the new policies and procedures required to reach the repeatable level. At the conclusion of our review, VBA officials stated that a training plan is now under development and will be made part of the software process improvement initiative action plan. In responding to our recommendation that it ensure that contractors have Process for Ensuring a repeatable software development capability, VBA intends to use a new That Contractors Are provision in future software development contracts. This provision, at the Repeatable however, does not require potential contractors to submit supporting documentation to VBA certifying their level of maturity. Validation of Level Is Not Clearly potential contractors’ software development capability maturity level Established should be a key factor in VBA’s software contracting decisions. The Internal Revenue Service, for example, recently started requiring that all current, in-process, and future contract solicitations for software development services require that contractors submit documentation to verify how their software development practices and processes satisfy the repeatable key process areas specified by SEI’s capability maturity model. The Internal Revenue Service plans to use this information when selecting software development services. Also, the Department of the Air Force’s acquisition 5 The software engineering process group is comprised of 8 people, 4 of whom are involved in systems development and 4 are not; the management steering group is comprised of 14 people, primarily senior managers. Page 6 GAO/AIMD-97-154 Veterans Benefits Modernization B-277370 policy states that software capability evaluations should be used for selecting software contractors. Although VBA has asserted that two of its current contractors are at the repeatable level, VBA could not provide documentation to support this. VBA subsequently requested the documentation from the contractors, but the information the contractors provided did not clearly show that they were at the repeatable level. In one case, the contractor presented information on how it assisted federal agencies in achieving the repeatable and/or higher levels of software development capability but did not provide documentation that the contractor was certified. In the second case, a component of the contractor’s organization asserted that it was at the repeatable level but did not provide documentation supporting this assertion. Recognizing the importance of a mature software development capability, Conclusions VBA has initiated actions to address the weaknesses identified in our June 1996 report. These actions will help it move toward a repeatable software capability maturity level, but additional efforts are needed. Specifically, VBA has not (1) developed a detailed strategy for how VBA plans to achieve a repeatable level of software development capability, (2) established a baseline to measure performance improvements, (3) trained its software development teams in the process improvement methodology, and (4) established a process for ensuring that its software development contractors are at the repeatable level. Recognizing that these deficiencies need to be addressed, VBA has efforts underway to do so. If these deficiencies are not sufficiently addressed, VBA’s software development capability will remain ad hoc and chaotic, subjecting the agency to continuing risk of cost overruns, poor quality software, and schedule delays. We recommend that the Secretary of Veterans Affairs direct the Under Recommendations Secretary for Benefits, in conjunction with VBA’s chief information officer, to • define the milestones, costs, tasks, and risks of the software process improvement initiative in order to provide a clear strategy for how VBA plans to improve its software development capability to a repeatable level; • develop and use a baseline showing VBA’s current software development capability from which to measure VBA’s software improvement effort; Page 7 GAO/AIMD-97-154 Veterans Benefits Modernization B-277370 • ensure that a training plan is developed and implemented that will provide key software development staff training in the software process improvement methodology, its principles, and key process areas; and • establish a source selection process to ensure that VBA’s software development contractors have the mature processes necessary for timely, high-quality software development, including evaluating and validating documentation provided by potential contractors establishing that they are at the repeatable level or higher. In comments on a draft of this report, VA concurred with our Agency Comments recommendations. VA also agreed that a repeatable level of process and Our Evaluation maturity is a goal that VBA must attain and described a number of activities underway to improve its software development capability. For example, VBA has developed a draft action plan to define a strategy to reach the repeatable level and specify the activities/tasks, milestones, costs, and timeliness associated with the process improvement effort. VBA also was reviewing and revising the draft plan to fully address the issues raised in our report. VA added that a significant amount of work still remains before this plan is finalized. We are encouraged by VBA’s response and will continue to monitor the agency’s progress in implementing its software improvement effort. We are sending copies of this report to the Ranking Minority Member of the Subcommittee on Oversight and Investigations and the Chairman and Ranking Minority Member of the Subcommittee on Benefits, House Committee on Veterans’ Affairs. We will also provide copies to the Chairmen and Ranking Minority Members of the House and Senate Committees on Veterans’ Affairs and the House and Senate Committees on Appropriations; the Secretary-Designate of Veterans Affairs; and the Director of the Office of Management and Budget. Copies will also be Page 8 GAO/AIMD-97-154 Veterans Benefits Modernization B-277370 made available to other parties upon request. Please contact me at (202) 512-6253 or by e-mail at email@example.com if you have any questions concerning this report. Major contributors to this report are listed in appendix II. Sincerely yours, Joel C. Willemssen Director, Information Resources Management Page 9 GAO/AIMD-97-154 Veterans Benefits Modernization Appendix I Comments From the Department of Veterans Affairs Note: GAO comments supplementing those in the report text appear at the end of this appendix. Page 10 GAO/AIMD-97-154 Veterans Benefits Modernization Appendix I Comments From the Department of Veterans Affairs See comment 1. Page 11 GAO/AIMD-97-154 Veterans Benefits Modernization Appendix I Comments From the Department of Veterans Affairs See comment 1. Page 12 GAO/AIMD-97-154 Veterans Benefits Modernization Appendix I Comments From the Department of Veterans Affairs Page 13 GAO/AIMD-97-154 Veterans Benefits Modernization Appendix I Comments From the Department of Veterans Affairs The following is GAO’s comment on the Department of Veterans Affairs letter dated September 4, 1997. 1. Enclosure (2) has not been included. GAO’s Comment Page 14 GAO/AIMD-97-154 Veterans Benefits Modernization Appendix II Major Contributors to This Report Helen Lew, Assistant Director Accounting and Leonard J. Latham, Technical Assistant Director Information K. Alan Merrill, Technical Assistant Director Management Division, David Chao, Senior Technical Advisor Tonia L. Johnson, Senior Information Systems Analyst Washington, D.C. (511221) Page 15 GAO/AIMD-97-154 Veterans Benefits Modernization Ordering Information The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. VISA and MasterCard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent. Orders by mail: U.S. General Accounting Office P.O. Box 37050 Washington, DC 20013 or visit: Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000 or by using fax number (202) 512-6061, or TDD (202) 512-2537. Each day, GAO issues a list of newly available reports and testimony. To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone. A recorded menu will provide information on how to obtain these lists. For information on how to access GAO reports on the INTERNET, send an e-mail message with "info" in the body to: firstname.lastname@example.org or visit GAO’s World Wide Web Home Page at: http://www.gao.gov PRINTED ON RECYCLED PAPER United States Bulk Rate General Accounting Office Postage & Fees Paid Washington, D.C. 20548-0001 GAO Permit No. G100 Official Business Penalty for Private Use $300 Address Correction Requested
Veterans Benefits Modernization: VBA Has Begun to Address Software Development Weaknesses But Work Remains
Published by the Government Accountability Office on 1997-09-15.
Below is a raw (and likely hideous) rendition of the original report. (PDF)