oversight

Veterans Benefits Modernization: VBA Has Begun to Address Software Development Weaknesses But Work Remains

Published by the Government Accountability Office on 1997-09-15.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                  United States General Accounting Office

GAO               Report to the Chairman, Subcommittee
                  on Oversight and Investigations,
                  Committee on Veterans’ Affairs, House
                  of Representatives

September 1997
                  VETERANS BENEFITS
                  MODERNIZATION
                  VBA Has Begun to
                  Address Software
                  Development Weaknesses
                  But Work Remains




GAO/AIMD-97-154
                   United States
GAO                General Accounting Office
                   Washington, D.C. 20548

                   Accounting and Information
                   Management Division

                   B-277370

                   September 15, 1997

                   The Honorable Terry Everett
                   Chairman, Subcommittee on Oversight
                     and Investigations
                   Committee on Veterans’ Affairs
                   House of Representatives

                   Dear Mr. Chairman:

                   The Veterans Benefits Administration (VBA), a component of the
                   Department of Veterans Affairs (VA), is in the process of modernizing its
                   information systems to improve its administrative operations and thereby
                   upgrade the quality and timeliness of its services. Software development
                   has been identified by many experts as one of the riskiest and most costly
                   aspects of systems development. In June 1996, we reported and testified
                   that VBA was operating at a level 1 software development capability,
                   defined as ad hoc and chaotic.1 In its response to that report, VBA stated it
                   would take actions to address the weaknesses we identified.

                   At your request, we conducted a follow-up review to determine the actions
                   taken by VBA to address management and technical weaknesses identified
                   in the June 19, 1996, hearing on the agency’s modernization effort. This
                   report, the second in a series on actions taken by VBA to address these
                   weaknesses, summarizes our assessment of the agency’s actions to
                   improve its software development capability.2


                   VBA has taken action to improve its software development capability.
Results in Brief   Among other things, it has launched a software process improvement
                   initiative, chartered a software engineering process group, and obtained
                   the services of an experienced contractor to assist in developing and
                   implementing a software process improvement effort. Although it has
                   made progress, VBA has not yet fully addressed needed software
                   development improvements. These include a need for (1) a defined
                   strategy to reach the repeatable level and a baseline to measure
                   improvements, (2) a process improvement training program for its
                   software developers, and (3) a process to ensure that VBA’s software

                   1
                    Software Capability Evaluation: VA’s Software Development Process Is Immature (GAO/AIMD-96-90,
                   June 19, 1996) and Veterans Benefits Modernization: Management and Technical Weaknesses Must Be
                   Overcome If Modernization Is to Succeed (GAO/T-AIMD-96-103, June 19, 1996).
                   2
                    The first report, Veterans Benefits Computer Systems: Risks of VBA’s Year-2000 Efforts
                   (GAO/AIMD-97-79, May 30, 1997), discussed our assessment of VBA’s Year 2000 efforts.



                   Page 1                                       GAO/AIMD-97-154 Veterans Benefits Modernization
             B-277370




             development contractors are at the repeatable level. VBA generally agrees
             that these issues need to be addressed and has efforts underway to do so.
             Until these deficiencies are sufficiently addressed, VBA’s software
             development capability remains ad hoc and chaotic, subjecting the agency
             to continuing risk of cost overruns, poor quality software, and schedule
             delays in software development.


             Since 1986, VBA has been trying to modernize its old, inefficient
Background   information systems. It reportedly spent an estimated $294 million on
             these activities between October 1, 1986, and February 29, 1996. The
             modernization program can have a major impact on the efficiency and
             accuracy with which about $20 billion in benefits and other services is
             paid annually to our nation’s veterans and their dependents. Software
             development is a critical component of this modernization effort. Also, a
             mature software development capability will provide added assurance that
             software developers will be able to effectively make changes to the
             software needed to address the Year 2000 computing problem.

             To evaluate VBA’s software development processes, in 1996, we applied the
             Software Engineering Institute’s (SEI)3 software capability evaluation
             methodology to those projects identified by VBA as using the best
             development processes. This evaluation compares agencies’ and
             contractors’ software development processes against SEI’s five-level
             software capability maturity model, with 5 being the highest level of
             maturity and 1 being the lowest.4 In June 1996, we reported that VBA was
             operating at a level 1 capability. At this level, VBA cannot reliably develop
             and maintain high-quality software on any major project within existing
             cost and schedule constraints, which places VBA software development
             projects at significant risk. Accordingly, VBA must rely on the various
             capabilities of individuals rather than on an institutional process that will
             yield repeatable, or level 2, results. VBA did not satisfy any of the criteria
             for a repeatable or level 2 capability, the minimum level necessary to
             significantly improve productivity and return on investment. For example,

             3
             This is a nationally recognized, federally funded research and development center established at
             Carnegie Mellon University in Pittsburgh, Pennsylvania, to address software development issues.
             4
              The following are the five levels of the software capability maturity model: level 5 (optimizing) is
             continuous process improvement; level 4 (managed) is detailed measure collection and
             process/product control; level 3 (defined) is software process documentation, standardization, and
             integration; level 2 (repeatable) is software project cost, schedule and functionality tracking and
             repeatability of successes; and level 1 (initial) is ad hoc and chaotic software processes. The six key
             process areas for a repeatable capability are (1) requirements management, (2) project planning,
             (3) project tracking and oversight, (4) subcontract management, (5) quality assurance, and
             (6) configuration management.



             Page 2                                         GAO/AIMD-97-154 Veterans Benefits Modernization
                  B-277370




                  VBA is extremely weak in the requirements management, software project
                  planning, and software subcontract management areas, with no
                  identifiable strengths or improvement activities.

                  Because of VBA’s software development weaknesses, we recommended
                  that the Secretary of Veterans Affairs

              •   obtain expert advice to improve VBA’s ability to develop high-quality
                  software;
              •   develop and expeditiously implement an action plan that describes a
                  strategy for reaching the repeatable (level 2) level of process maturity;
              •   ensure that any future contracts for software development require the
                  contractor to have a software development capability of at least level 2;
                  and
              •   delay any major investment in new software development—beyond what
                  is needed to sustain critical day-to-day operations—until the repeatable
                  level of process maturity is attained.

                  In commenting on a draft of the June 1996 report, VBA agreed with three of
                  our recommendations but disagreed with delaying major investments in
                  software development. VBA stated that while it agreed that a repeatable
                  level of process maturity is a goal that must be attained, it disagreed that
                  “all software development beyond that which is day-to-day critical must be
                  curtailed.” VBA stated that the payment system replacement projects, the
                  migration of legacy systems, and other activities to address the change of
                  century must continue. In our response to VBA’s comments, we agreed that
                  the change of century and changes to legislation must be continued, and
                  we characterized these changes as sustaining day-to-day operations.
                  However, for those projects that do not meet this criterion, we continue to
                  believe that VBA should delay software development investments until a
                  maturity of level 2 is reached.


                  To assess actions taken by VBA to improve its software capability, we
Scope and         reviewed VBA documents, such as its “Software Process Improvement
Methodology       Initiative Strategic Plan,” dated March 1997; the Best Practices Round Up
                  Method; and the Interagency Agreement with the Air Force, dated
                  September 1996, to obtain expert software process improvement
                  assistance. We also reviewed SEI’s IDEALSM: A User’s Guide for Software
                  Process Improvement, dated February 1996, and technical report on Best
                  Training Practices Within the Software Engineering Industry, dated
                  November 1996. In addition, we reviewed VBA contracts, correspondence



                  Page 3                           GAO/AIMD-97-154 Veterans Benefits Modernization
                         B-277370




                         to contractors, and supporting documents to determine what VBA has done
                         to ensure that VBA’s software development contractors are at the
                         repeatable level.

                         We interviewed VBA officials and contractor personnel involved with the
                         software process improvement effort to determine what actions VBA has
                         taken to improve its software capability. We also interviewed selected VBA
                         project managers involved in new systems development on their
                         knowledge of the software process improvement initiative.

                         We performed our work from October 1996 through August 1997 in
                         accordance with generally accepted government auditing standards. We
                         requested comments on a draft of this report from the Secretary-Designate
                         of Veterans Affairs. The Secretary-Designate provided us with written
                         comments, which are discussed in the “Agency Comments and Our
                         Evaluation” section and reprinted in appendix I.


                         VBA has initiated several actions to improve its software development
VBA Has Initiated        capability. For example, in response to our recommendation that it obtain
Actions to Improve Its   expert assistance, VBA has hired a contractor—SEI—to (1) assist it in
Software                 developing an integrated set of software practices that will position VBA for
                         successful, lasting improvements, (2) help formulate a software
Development              improvement program, and (3) provide expertise in executing software
Capability               improvement activities. SEI is also expected to provide expertise in
                         strategic and tactical planning, training, policy preparation, and action
                         planning.

                         In addition, in response to our recommendation that it develop and
                         implement an action plan describing a strategy for reaching the repeatable
                         level, VBA launched a software process improvement initiative in June 1996
                         to lay the foundation and build the context for sustainable, measurable
                         improvements to its software development capability. It has developed a
                         strategic plan that describes the purpose and goals of this improvement
                         initiative. One of the plan’s goals is to establish organization policies and
                         guidelines for the management, planning, and tracking of software
                         projects that will enable VBA to repeat earlier successes on projects with
                         similar applications.

                         VBA has also recently initiated two software process improvement projects.
                         The first project, called “Best Practices Round Up,” will identify software
                         development practices that VBA software development teams are doing



                         Page 4                            GAO/AIMD-97-154 Veterans Benefits Modernization
                    B-277370




                    correctly. VBA believes that there are “pockets of excellence” within its
                    organization not identified in our June 1996 report that it can build upon.
                    The Best Practices Round Up project team started its review in April 1997
                    and briefed VBA’s chief information officer on its results in August 1997.

                    The second project, called “Standards, Policies, and Procedures,” will
                    assess whether VBA’s software development teams are following current
                    software development policies, procedures, and standards. This project
                    was initiated in June 1997 and is expected to be completed by the end of
                    September 1997. The project team is expected to make recommendations
                    to ensure compliance with standards.


                    Although VBA has launched a software process initiative and an
Software Process    accompanying strategic plan, VBA has not yet clearly presented how it
Improvement         intends to move from an ad hoc and chaotic level of software development
Initiative Lacks    capability to a repeatable level. SEI’s IDEALSM: A User’s Guide for Software
                    Process Improvement requires that a plan be developed that includes a
Specificity and a   schedule for initial activities, basic resource requirements, and benefits to
Baseline            the organization. VBA’s current plan contains no milestones—beginning,
                    interim, or completion dates—by which to measure the agency’s progress
                    and to identify problems. The plan also contains no analysis or
                    information on costs, benefits, or risks.

                    VBA officials stated that they recognize that the agency’s strategic plan for
                    software process improvement lacks this specificity. At the conclusion of
                    our review, these officials said that VBA intends to address this area in an
                    upcoming action plan for the software process improvement initiative.

                    VBA has also not yet established a baseline from which to measure its
                    software process improvements. According to the SEI IDEALSM: A User’s
                    Guide for Software Process Improvement, an organization needs to
                    understand its current software process baseline so that it can develop a
                    plan to achieve the business changes necessary to reach its software
                    process improvement goals. At the conclusion of our review, VBA officials
                    told us that they plan to use as their baseline the results of our June 1996
                    report, along with the results from their Best Practices Round Up and
                    Standards, Policies, and Procedures projects. They stated that the baseline
                    should be established by September 1997.




                    Page 5                            GAO/AIMD-97-154 Veterans Benefits Modernization
                       B-277370




                       Training of key staff is critical to achieving level 2 repeatability. According
VBA Software Process   to SEI’s technical report entitled, Best Practices Within the Software
Improvement Training   Engineering Industry, best training practices include defining a process for
Plan Not Effectively   software engineering education.

Implemented            Although VBA has provided process improvement training to many of the
                       managers in its software engineering process group and management
                       steering group,5 key software personnel—software developers, project
                       managers, and line managers—have not been trained in the process
                       improvement methodology, the principles behind it, and the key process
                       areas. VBA’s software process improvement project manager explained that
                       these key people had not yet been trained because VBA did not want to
                       train them too long before implementing the process improvement
                       projects. The project manager said that VBA plans to train these staff during
                       fiscal years 1998 and 1999.

                       However, VBA does not have a documented training plan to help ensure
                       that these personnel receive training. Unless these individuals are trained
                       in the process improvement methodology, its principles, and the key
                       process areas, it will be difficult for them to implement the new policies
                       and procedures required to reach the repeatable level. At the conclusion of
                       our review, VBA officials stated that a training plan is now under
                       development and will be made part of the software process improvement
                       initiative action plan.


                       In responding to our recommendation that it ensure that contractors have
Process for Ensuring   a repeatable software development capability, VBA intends to use a new
That Contractors Are   provision in future software development contracts. This provision,
at the Repeatable      however, does not require potential contractors to submit supporting
                       documentation to VBA certifying their level of maturity. Validation of
Level Is Not Clearly   potential contractors’ software development capability maturity level
Established            should be a key factor in VBA’s software contracting decisions. The Internal
                       Revenue Service, for example, recently started requiring that all current,
                       in-process, and future contract solicitations for software development
                       services require that contractors submit documentation to verify how their
                       software development practices and processes satisfy the repeatable key
                       process areas specified by SEI’s capability maturity model. The Internal
                       Revenue Service plans to use this information when selecting software
                       development services. Also, the Department of the Air Force’s acquisition

                       5
                        The software engineering process group is comprised of 8 people, 4 of whom are involved in systems
                       development and 4 are not; the management steering group is comprised of 14 people, primarily senior
                       managers.



                       Page 6                                      GAO/AIMD-97-154 Veterans Benefits Modernization
                      B-277370




                      policy states that software capability evaluations should be used for
                      selecting software contractors.

                      Although VBA has asserted that two of its current contractors are at the
                      repeatable level, VBA could not provide documentation to support this. VBA
                      subsequently requested the documentation from the contractors, but the
                      information the contractors provided did not clearly show that they were
                      at the repeatable level. In one case, the contractor presented information
                      on how it assisted federal agencies in achieving the repeatable and/or
                      higher levels of software development capability but did not provide
                      documentation that the contractor was certified. In the second case, a
                      component of the contractor’s organization asserted that it was at the
                      repeatable level but did not provide documentation supporting this
                      assertion.


                      Recognizing the importance of a mature software development capability,
Conclusions           VBA has initiated actions to address the weaknesses identified in our
                      June 1996 report. These actions will help it move toward a repeatable
                      software capability maturity level, but additional efforts are needed.
                      Specifically, VBA has not (1) developed a detailed strategy for how VBA
                      plans to achieve a repeatable level of software development capability,
                      (2) established a baseline to measure performance improvements,
                      (3) trained its software development teams in the process improvement
                      methodology, and (4) established a process for ensuring that its software
                      development contractors are at the repeatable level. Recognizing that
                      these deficiencies need to be addressed, VBA has efforts underway to do
                      so. If these deficiencies are not sufficiently addressed, VBA’s software
                      development capability will remain ad hoc and chaotic, subjecting the
                      agency to continuing risk of cost overruns, poor quality software, and
                      schedule delays.


                      We recommend that the Secretary of Veterans Affairs direct the Under
Recommendations       Secretary for Benefits, in conjunction with VBA’s chief information officer,
                      to

                  •   define the milestones, costs, tasks, and risks of the software process
                      improvement initiative in order to provide a clear strategy for how VBA
                      plans to improve its software development capability to a repeatable level;
                  •   develop and use a baseline showing VBA’s current software development
                      capability from which to measure VBA’s software improvement effort;



                      Page 7                            GAO/AIMD-97-154 Veterans Benefits Modernization
                         B-277370




                     •   ensure that a training plan is developed and implemented that will provide
                         key software development staff training in the software process
                         improvement methodology, its principles, and key process areas; and
                     •   establish a source selection process to ensure that VBA’s software
                         development contractors have the mature processes necessary for timely,
                         high-quality software development, including evaluating and validating
                         documentation provided by potential contractors establishing that they
                         are at the repeatable level or higher.


                         In comments on a draft of this report, VA concurred with our
Agency Comments          recommendations. VA also agreed that a repeatable level of process
and Our Evaluation       maturity is a goal that VBA must attain and described a number of activities
                         underway to improve its software development capability. For example,
                         VBA has developed a draft action plan to define a strategy to reach the
                         repeatable level and specify the activities/tasks, milestones, costs, and
                         timeliness associated with the process improvement effort. VBA also was
                         reviewing and revising the draft plan to fully address the issues raised in
                         our report. VA added that a significant amount of work still remains before
                         this plan is finalized.

                         We are encouraged by VBA’s response and will continue to monitor the
                         agency’s progress in implementing its software improvement effort.


                         We are sending copies of this report to the Ranking Minority Member of
                         the Subcommittee on Oversight and Investigations and the Chairman and
                         Ranking Minority Member of the Subcommittee on Benefits, House
                         Committee on Veterans’ Affairs. We will also provide copies to the
                         Chairmen and Ranking Minority Members of the House and Senate
                         Committees on Veterans’ Affairs and the House and Senate Committees on
                         Appropriations; the Secretary-Designate of Veterans Affairs; and the
                         Director of the Office of Management and Budget. Copies will also be




                         Page 8                           GAO/AIMD-97-154 Veterans Benefits Modernization
B-277370




made available to other parties upon request. Please contact me at
(202) 512-6253 or by e-mail at willemssenj.aimd@gao.gov if you have any
questions concerning this report. Major contributors to this report are
listed in appendix II.

Sincerely yours,




Joel C. Willemssen
Director, Information Resources Management




Page 9                          GAO/AIMD-97-154 Veterans Benefits Modernization
Appendix I

Comments From the Department of
Veterans Affairs

Note: GAO comments
supplementing those in the
report text appear at the
end of this appendix.




                             Page 10   GAO/AIMD-97-154 Veterans Benefits Modernization
                 Appendix I
                 Comments From the Department of
                 Veterans Affairs




See comment 1.




                 Page 11                           GAO/AIMD-97-154 Veterans Benefits Modernization
                 Appendix I
                 Comments From the Department of
                 Veterans Affairs




See comment 1.




                 Page 12                           GAO/AIMD-97-154 Veterans Benefits Modernization
Appendix I
Comments From the Department of
Veterans Affairs




Page 13                           GAO/AIMD-97-154 Veterans Benefits Modernization
                Appendix I
                Comments From the Department of
                Veterans Affairs




                The following is GAO’s comment on the Department of Veterans Affairs
                letter dated September 4, 1997.


                1. Enclosure (2) has not been included.
GAO’s Comment




                Page 14                           GAO/AIMD-97-154 Veterans Benefits Modernization
Appendix II

Major Contributors to This Report


                       Helen Lew, Assistant Director
Accounting and         Leonard J. Latham, Technical Assistant Director
Information            K. Alan Merrill, Technical Assistant Director
Management Division,   David Chao, Senior Technical Advisor
                       Tonia L. Johnson, Senior Information Systems Analyst
Washington, D.C.




(511221)               Page 15                        GAO/AIMD-97-154 Veterans Benefits Modernization
Ordering Information

The first copy of each GAO report and testimony is free.
Additional copies are $2 each. Orders should be sent to the
following address, accompanied by a check or money order
made out to the Superintendent of Documents, when
necessary. VISA and MasterCard credit cards are accepted, also.
Orders for 100 or more copies to be mailed to a single address
are discounted 25 percent.

Orders by mail:

U.S. General Accounting Office
P.O. Box 37050
Washington, DC 20013

or visit:

Room 1100
700 4th St. NW (corner of 4th and G Sts. NW)
U.S. General Accounting Office
Washington, DC

Orders may also be placed by calling (202) 512-6000
or by using fax number (202) 512-6061, or TDD (202) 512-2537.

Each day, GAO issues a list of newly available reports and
testimony. To receive facsimile copies of the daily list or any
list from the past 30 days, please call (202) 512-6000 using a
touchtone phone. A recorded menu will provide information on
how to obtain these lists.

For information on how to access GAO reports on the INTERNET,
send an e-mail message with "info" in the body to:

info@www.gao.gov

or visit GAO’s World Wide Web Home Page at:

http://www.gao.gov




PRINTED ON    RECYCLED PAPER
United States                       Bulk Rate
General Accounting Office      Postage & Fees Paid
Washington, D.C. 20548-0001           GAO
                                 Permit No. G100
Official Business
Penalty for Private Use $300

Address Correction Requested