oversight

Medicare Transaction System: Success Depends Upon Correcting Critical Managerial and Technical Weaknesses

Published by the Government Accountability Office on 1997-05-16.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                 United States General Accounting Office

GAO              Report to Congressional Requesters




May 1997
                 MEDICARE
                 TRANSACTION SYSTEM
                 Success Depends Upon
                 Correcting Critical
                 Managerial and Technical
                 Weaknesses




GAO/AIMD-97-78
      United States
GAO   General Accounting Office
      Washington, D.C. 20548

      Accounting and Information
      Management Division

      B-275281

      May 16, 1997

      The Honorable Christopher Shays
      Chairman
      The Honorable Edolphus Towns
      Ranking Minority Member
      Subcommittee on Human Resources
      Committee on Government Reform and Oversight
      House of Representatives

      The Honorable Stephen Horn
      Chairman
      The Honorable Carolyn Maloney
      Ranking Minority Member
      Subcommittee on Government Management,
        Information and Technology
      Committee on Government Reform and Oversight
      House of Representatives

      This report responds to your request that GAO evaluate the Health Care Financing
      Administration’s (HCFA) acquisition of its Medicare Transaction System (MTS). Specifically, we
      addressed the extent to which HCFA is (1) effectively managing its interim Medicare processing
      environment, (2) using required practices to manage MTS as an investment, and (3) applying
      sound system development processes to reduce risk. The report makes several
      recommendations that could help HCFA effectively meet its Medicare information technology
      needs and decrease the risk of costly technological decisions and wasted federal expenditures
      that are often associated with large federal information technology projects.

      We are sending copies of this report to the Secretary of Health and Human Services,
      Administrator of the Health Care Financing Administration, Director of the Office of
      Management and Budget, and appropriate congressional committees. We will also make copies
      available to others upon request.

      Please call me at (202) 512-6253 if you or your staff have any questions concerning this report.
      You can also reach me by e-mail at willemssenj.aimd@gao.gov. Major contributors to this
      report are listed in appendix III.




      Joel C. Willemssen
      Director, Information Resources Management
Executive Summary


             Medicare, the nation’s largest health insurer, expects to process over
Purpose      1 billion claims and pay $288 billion in benefits per year by 2000. The
             Health Care Financing Administration (HCFA) is responsible for
             administering this program under the Department of Health and Human
             Services (HHS). Nine separate automated information systems have been
             used to process Medicare claims. HCFA plans to spend about $1 billion to
             replace these systems with a single, unified system—the Medicare
             Transaction System (MTS). HCFA estimates that MTS will begin replacing the
             existing systems in 1998, providing improved service, reduced operating
             expenses, better contractor oversight, and more protection of program
             funds from waste, fraud, and abuse, while also accommodating managed
             care and alternative payment methodologies.

             At the request of the Subcommittees on Human Resources, and
             Government Management, Information, and Technology of the House
             Committee on Government Reform and Oversight, GAO reviewed the
             extent to which HCFA is (1) effectively managing its interim
             claims-processing, including planning for and correcting year 2000-related
             computer problems, (2) using required practices to manage MTS as an
             investment, and (3) applying sound systems-development practices to
             reduce risk.


             HCFA manages the Medicare program through about 70 contractors who
Background   process claims and pay benefits at about 45 sites nationwide. It considers
             MTS an important part of its plans to improve the Medicare claims process.
             By replacing Medicare’s multiple, contractor-operated claims-processing
             systems with a single system, HCFA believes it will obtain significant
             benefits. For example, HCFA explained that when changes in legislative or
             administrative policies require changes in Medicare payments or coverage,
             each of the existing claims-processing systems must be individually
             modified—an expensive, time-consuming process. Under MTS, only one
             system would need to be modified.

             HCFA has made major changes to the development and implementation
             plans for MTS since it was begun. In previous reviews, GAO identified the
             need to reduce MTS acquisition risks through an incremental deployment
             approach.1 HCFA now plans such an approach. In an effort to improve its
             MTS initiative, reduce risk, and achieve some savings before MTS is



             1
              Medicare: New Claims Processing System Benefits and Acquisition Risks (GAO/HEHS/AIMD-94-79,
             Jan. 25, 1994) and Medicare Transaction System: Strengthened Management and Sound Development
             Approach Critical to Success (GAO/T-AIMD-96-12, Nov. 16, 1995).



             Page 2                                        GAO/AIMD-97-78 Medicare Transaction System
                   Executive Summary




                   implemented, HCFA is simultaneously undertaking several interim actions
                   while continuing its development of MTS.

                   HCFA’sinterim efforts consist of selecting a single part A and single part B
                   system from the nine existing systems and consolidating the data
                   processing workload, thereby reducing the number of processing sites
                   from 45 to about 20. (Medicare part A covers institutional care, while part
                   B covers physician, supplier, and other outpatient services.) HCFA plans to
                   move the data processing workload from the interim consolidated
                   processing sites to two planned MTS claims-processing sites by mid-1998.
                   By then, HCFA also plans to have its contractors revise their systems to
                   accommodate year-2000 processing.2

                   On April 4, 1997, HCFA announced that, as a result of a recent management
                   review, it was redirecting its MTS contractor to focus solely on the
                   managed care module—the first of six planned software releases—while it
                   examines alternative ways to achieve its MTS goals. HCFA concluded that its
                   vision of MTS as the best information technology to take Medicare into the
                   21st century had not changed.


                   HCFA  recognizes that the multiple Medicare claims-processing systems are
Results in Brief   difficult to administer, and is looking to MTS to achieve substantial
                   administrative savings, increase claims control, and improve customer
                   service. To its credit, HCFA is using a phased development approach to
                   reduce risks and is beginning to apply investment practices in its
                   management of the MTS project. However, the benefits of modernizing
                   Medicare claims processing at an estimated cost of $1 billion will not be
                   realized unless HCFA overcomes serious management and technical
                   weaknesses in three major areas that place the modernization at great risk.

                   First, HCFA needs to greatly improve its management of the essential
                   interim Medicare processing environment and the changes necessary for
                   operating beyond 2000. To successfully process the claims workload,
                   consolidate existing processing sites, address year 2000-related systems
                   problems, and convert from nine systems to two, careful planning is
                   necessary. However, such planning has not yet been completed. For
                   example, although HCFA has already begun to convert its existing systems

                   2
                    Due to the use of the two-digit format for dates in many computer systems, the year 2000 (represented
                   “00”) will be indistinguishable from the year 1900 (also represented “00”). As a result, unless
                   modifications are made, system or application programs that use dates to perform calculations,
                   comparisons, or sorting may generate incorrect results when working with dates after 1999 as a result
                   of reading “00” as “1900.”



                   Page 3                                            GAO/AIMD-97-78 Medicare Transaction System
Executive Summary




and consolidate its sites, it has not developed plans to guide these
activities. Such plans should include a schedule and estimate of resources
required for the transition, details defining systems-contractor
responsibilities, and an approach for addressing potential year-2000
problems.

The risks associated with concurrently converting major systems while at
the same time managing ongoing development of MTS is magnified by the
fact that changing from the existing claims processing environment to MTS
is a larger, more complex systems-conversion challenge than anything
HCFA has previously faced. Further, HCFA is relying on its Medicare systems
contractors to assess, plan, and implement essential changes for the
year-2000 issue, but is not closely monitoring these critical activities or
receiving certifications or assurances from contractors that the problems
will be corrected.

A schedule and estimate of needed resources for each major stage of the
transition to the interim processing environment would ensure the
availability of needed resources and help HCFA better manage and
coordinate transition activities. HCFA agreed that a transition schedule and
resource estimates would be helpful; a contractor is to assist HCFA in
preparing and integrating them into the overall MTS development schedule,
to be completed by late this spring.

Second, MTS is not being adequately managed as an investment. HCFA has
not followed practices that are essential if management is to make
informed technology investment decisions, including preparing a valid
cost-benefit analysis, considering viable alternatives, and fully evaluating
how the proposed technology benefits will contribute to improvements in
mission performance. HCFA estimates that many programmatic savings will
result from automated edits that identify abusive billing practices and deny
related claims. However, HCFA stated that because the exact nature of MTS’
edits had not been identified, the resulting savings could be significantly
different from the estimated savings. Also, since 1992 when the first
analysis was completed, the total cost of this project has increased from
$151 million to about $1 billion.3 The $1 billion includes estimated costs to
transition to the MTS environment and acquire operating sites.

Finally, HCFA has not applied all the sound systems-development practices
necessary to reduce risk and assist management in controlling
development of system requirements and software. Along with not

3
 All dollar amounts presented in this report are expressed as undiscounted current dollars.



Page 4                                             GAO/AIMD-97-78 Medicare Transaction System
                            Executive Summary




                            developing plans critical to the project’s success, the agency has not
                            adequately overseen its contractor’s software development strategy,
                            adequately managed the project’s schedule, or implemented a program to
                            effectively address risk. More specifically, deficiencies in several critical
                            systems-development processes provide early warning of weaknesses in
                            the management capability of HCFA itself and of its contractors. Plans are
                            inadequate or completed too late, the schedule is incomplete and contains
                            overlap in development phases, HCFA’s risk-management process is
                            inadequate, and its oversight has not prevented an unsound
                            software-development strategy. These factors all increase the risk that MTS
                            cannot be developed into the management information tool that HCFA
                            needs.



Principal Findings

Ineffective Management of   Generally accepted program management practices call for preparing
Interim Processing          detailed plans for system transitions and modifications, which for HCFA
Environment                 should include (1) schedule and resource estimates, (2) defined
                            responsibilities of the selected Medicare part A and part B systems
                            contractors, (3) test plans, and (4) performance measures. These plans are
                            particularly important for HCFA because the interim transition is a larger
                            systems-conversion effort than any previously undertaken by HCFA and is
                            being performed concurrently with HCFA’s management of MTS
                            development.

                            HCFA  has recently taken some actions and plans others. It hired a
                            consultant to help it prepare a schedule and resource estimates for the
                            transition, and awarded a contract defining responsibilities for the
                            selected part B system on April 8, 1997. The part A systems contractor’s
                            statement of responsibilities is to be completed by the end of May of this
                            year.

                            According to HCFA officials, their contractors routinely test and implement
                            systems changes, and they plan to rely on their contractors to successfully
                            test and implement the changes occurring during the transition. However,
                            the transition differs from routine changes. The contractors may not have
                            a particularly high incentive to properly make these conversions, since
                            HCFA plans to eliminate these contractors when MTS is fully implemented.
                            Also, these conversions involve significantly more data and will require




                            Page 5                                GAO/AIMD-97-78 Medicare Transaction System
                           Executive Summary




                           more system capacity than routine modifications. Further, HCFA officials
                           said they do not believe it would be cost-beneficial to use the agency’s
                           limited resources to develop performance measures for interim systems
                           that will be replaced by MTS. GAO believes these measures are needed to
                           ensure that this complex and important interim phase is properly
                           implemented and delivers the benefits expected.

                           HCFA  has also not taken enough initial actions to ensure that it can avoid
                           the systems-related service disruptions that may occur as the year 2000
                           approaches. For example, it has not developed an assessment of the
                           potential severity of the impact of the century change or completed a plan
                           for addressing it. Further, HCFA has not required systems contractors to
                           submit year-2000 plans for approval. It lacks any specific legal agreements
                           with its contractors addressing year-2000 problems, including how, when,
                           or even if such problems will be corrected. The potential risks associated
                           with not being ready for 2000 are serious, since virtually all Medicare
                           transactions depend, to some degree, on dates to determine benefits
                           eligibility—dates of birth, medical procedure, other insurance coverage,
                           and so forth. Unless corrected, if a computer system were to read “00” as
                           1900 instead of 2000, someone born in 1925 would be seen as negative 25
                           years old—not even born yet—rather than the actual age of 75.

                           HCFA  is now surveying its contractors about this issue. The agency has also
                           asked the contractors to provide estimates showing when their systems
                           will be year-2000 compliant. However, HCFA has no plans to independently
                           validate the contractors’ strategies and test plans. HCFA likewise has no
                           plans to approve contractors’ approaches for addressing interface and
                           data exchange issues.


MTS Not Being Adequately   Federal legislation and Office of Management and Budget (OMB) directives
Managed as an Investment   require agencies to manage major information technology acquisitions as
                           investments. Critical elements of technology investment decision-making
                           are processes and data that ensure that (1) the right project proposals are
                           funded on the basis of management evaluations of costs, risks, and
                           expected benefits to mission performance and (2) once funded, projects
                           are controlled by examining costs, the development schedule, and actual
                           versus expected results.

                           These goals are accomplished through preparing valid cost-benefit
                           analyses, considering viable alternatives, having senior management
                           consistently make key decisions on major projects, and ensuring that the



                           Page 6                               GAO/AIMD-97-78 Medicare Transaction System
                      Executive Summary




                      projects support the agency’s mission and goals. Because HCFA has not
                      implemented these critical elements, it has no assurance that its MTS
                      development will reduce risks to the greatest extent possible, and achieve
                      a maximum return on its MTS investment.

                      Since 1992 HCFA has prepared numerous documents showing MTS’
                      estimated costs and benefits. However, all of these documents are flawed
                      in that not all costs were identified, projected savings were overstated or
                      not adequately supported, and alternative solutions were not considered.
                      As a result, reliable cost or benefit estimates for MTS are not yet available.
                      In such an information vacuum, it is impossible to manage a complex
                      technology project such as MTS as an investment because no basis exists
                      on which to predict likely short- or long-term results, and compare them
                      against resources spent. Furthermore, recent data show that an early key
                      HCFA assumption was invalid. HCFA’s 1993 analysis assumed that, without
                      MTS, costs per claim for part A and part B would continually increase from
                      1993 through 2002. However, actual contractor cost reports show that
                      costs per claim for part A and part B decreased for fiscal years 1994
                      through 1996, from $1.41 to $1.27, and from $0.89 to $0.88, respectively.

                      Beyond monetary uncertainties, decision-making without alternatives
                      analysis adds risk. The decision to consolidate a daily claims-processing
                      workload of about 2.6 million claims at two new sites yet to be acquired
                      was made without considering other alternatives, such as using existing
                      processing centers.

                      Related to a sound cost-benefit analysis is the level of management
                      oversight. While HCFA is making positive changes, such as designating a
                      chief information officer and establishing an investment review board,
                      consistent senior-level involvement and investment-based decision-making
                      are still lacking. HCFA’s executive decision-making group—the MTS
                      management board—has not made many of the critical MTS investment
                      decisions, and HCFA has not adequately linked MTS to its agency goals or
                      mission; likewise, it has not established performance measures to evaluate
                      how well MTS supports the goals or programs of the agency.


Lack of Sound         Sound systems-development practices are not being followed for MTS. HCFA
Systems-Development   lacks critical project development plans, including plans for requirements
Practices             management, software development, and systems integration. Without
                      these plans, it is difficult for HCFA to appropriately manage and monitor the
                      MTS contract and apply sound systems-development practices. HCFA’s




                      Page 7                                GAO/AIMD-97-78 Medicare Transaction System
                      Executive Summary




                      contractor oversight likewise departs from critical software development
                      best practices, including an assessment of the software capability level of
                      the MTS development contractor, use of specific software measures to
                      assess the quality of software development, and use of sound
                      software-estimation assumptions to make reasonable project cost and
                      time estimates. Not embracing such practices threatens MTS’ quality,
                      timeliness, and cost.

                      For example, HCFA’s lack of a requirements-management plan contributed
                      to several redirections that resulted in schedule delays. The approach
                      toward defining requirements has been changed, not all requirements are
                      yet defined, many that have been defined have not been formally agreed
                      to, and their volatility can affect cost and development. For example,
                      during a recent 5-month period, the requirements for one software release
                      dropped from 1,639 to 1,499, while the requirements for another release
                      increased from 631 to 868.

                      In addition, the MTS development schedule contains serious flaws.
                      Resource needs have not been included, a critical path (i.e., the sequence
                      of dependent tasks that, if delayed, will delay the entire project) has not
                      been identified, and project development phases—designed to be
                      sequential—are often concurrent. As a result, HCFA cannot rely on the MTS
                      program schedule for management decision-making.

                      Finally, weaknesses in HCFA’s MTS risk management process have not been
                      addressed. HCFA has no quantitative analysis of the cost, schedule, or
                      systems performance impact of identified risks to MTS development;
                      overall responsibility for MTS risk management has not been assigned. At
                      the same time, several long-standing critical risks remain, and serve as a
                      warning that effective risk management practices have not been
                      institutionalized or uniformly practiced.


                      In order to help HCFA effectively manage its interim Medicare processing
Recommendations       environment, GAO recommends that the Secretary of Health and Human
                      Services direct that the Administrator of the Health Care Financing
                      Administration take the following steps:

                  •   Prepare a plan that provides details of the transition to the single Medicare
                      part A and part B systems, and defines how systems will be converted to
                      address potential year-2000 problems. (See chapter 2.)




                      Page 8                                GAO/AIMD-97-78 Medicare Transaction System
    Executive Summary




•   Prepare plans for conducting thorough testing before converting part A
    and part B systems. (See chapter 2.)
•   Establish a means of assessing performance in the crucial early stages of
    the transition, and apply any lessons learned to planning for MTS. (See
    chapter 2.)
•   Help ensure the reliable operation of its systems through the year 2000 by
    identifying responsibilities for managing and monitoring year-2000 actions,
    preparing an assessment of the severity and timing of potential year-2000
    impact, developing contingency plans for critical systems in the event of
    failure, and regularly reporting to HHS on its progress. (See chapter 2.)

    In addition, in order to help HCFA develop and implement MTS, and
    minimize unnecessary spending in the process, GAO recommends the
    Secretary of Health and Human Services withhold funding for the MTS
    operating site contracts until HCFA cost-justifies them. (See chapter 3.) GAO
    also recommends that the Secretary of Health and Human Services direct
    that the Administrator of the Health Care Financing Administration do the
    following:

•   Justify the continuation of MTS by producing a valid cost-benefit and
    alternatives analysis that includes goals for reaching programmatic savings
    and links estimated savings to specific improvements in Medicare claims
    processing, and take appropriate action based on the results of the
    analysis. (See chapter 3.)
•   Establish an investment management approach for MTS by explicitly
    linking the roles and responsibilities of the CIO and the Investment Review
    Board to relevant legislative mandates and requirements, which include
    (1) designing and implementing a process for maximizing the value and
    assessing and managing the risks of information technology acquisitions,
    and integrating that process with the budget, financial, and program
    management decisions of the agency, (2) providing the means for senior
    management to obtain timely information regarding the progress of an
    investment, including a system of milestones for measuring progress, in
    terms of cost, capability of the system to meet specified requirements,
    timeliness, and quality, and (3) ensuring that performance measures are
    applied to measure how well information technology projects support the
    goals and missions of the agency. (See chapter 3.)
•   Complete and implement those plans that are critical to effective systems
    development, including a requirements management plan, software
    development plan, configuration management plan, and systems
    integration plan. (See chapter 4.)




    Page 9                                GAO/AIMD-97-78 Medicare Transaction System
                         Executive Summary




                     •   Require an independent evaluation of the MTS contractor’s software
                         development capability prior to beginning the software development
                         phase. To ensure that the contractor’s MTS development team has the
                         capability required for reasonable assurance of success, it should achieve
                         a rating of at least level 2. (See chapter 4.)
                     •   Complete a new, integrated MTS program schedule that includes a critical
                         path for the entire initiative, including the interim, and resources and costs
                         for each task; it should also minimize overlap in the phases of the
                         systems-development process. (See chapter 4.)
                     •   Mitigate critical risks by designating an accountable official for risk
                         management and ensuring that this individual implements a process that
                         will (1) identify and quantify all significant risks, (2) establish time frames
                         for assessing risk status and specifying target dates for risk mitigation,
                         (3) develop metrics that will compare progress in assessing the
                         effectiveness of risk mitigation efforts, (4) provide a mechanism for
                         alerting management early of risks that are becoming imminent,
                         (5) provide resource estimates of staff, schedule needs, and funding to
                         address identified risks, (6) ensure that the MTS risk management database
                         incorporates all identified risks, and (7) document interdependencies
                         among risks. (See chapter 4.)

                         Additional GAO recommendations to improve the transition, management,
                         development, and implementation of MTS are in chapters 2, 3, and 4.


                         We requested written comments from the Secretary of the Department of
Agency Comments          Health and Human Services and the Director of the Office of Management
and Our Evaluation       and Budget. HHS’ Assistant Secretary for Management and Budget agreed
                         with GAO’s recommendations for effectively managing Medicare’s interim
                         claims-processing environment, using required practices to manage MTS as
                         an investment, and applying sound systems-development practices. HHS
                         said that both the Department and HCFA are committed to complying with
                         these recommendations. Further, HHS said that steps have already been
                         initiated to implement several of GAO’s recommendations, including
                         (1) preparing detailed plans for the transition to the single Medicare part A
                         and part B systems, (2) requesting implementation plans from its
                         contractors on their progress in making their systems
                         millennium-compliant, and (3) reassessing the cost, benefits, and
                         alternative development strategies to MTS. HCFA concluded that GAO has
                         been of significant assistance in offering suggestions for improved
                         management.




                         Page 10                               GAO/AIMD-97-78 Medicare Transaction System
Executive Summary




In comments on a draft of this report, OMB’s Deputy Director for
Management agreed with GAO’s recommendations for improved
management of MTS and concurred that HCFA must take steps to more
adequately plan for the consolidation to the standardized part A and part B
system. It also concurred that HCFA needs to better manage MTS as an
investment, and apply sound systems-development practices to reduce
risk and assist management in controlling the development of systems
requirements and software.

GAO  believes that by effectively implementing its recommendations HCFA
will improve the management of its modernization effort and increase its
assurance that the approach taken will be cost-effective, risk-averse, and
support the agency’s mission and goals.

These comments are discussed in chapters 2, 3, and 4 and are reprinted in
appendixes I and II.




Page 11                              GAO/AIMD-97-78 Medicare Transaction System
Contents



Executive Summary                                                                                   2


Chapter 1                                                                                          16
                       The Medicare Transaction System                                             16
Introduction           Objectives, Scope, and Methodology                                          19

Chapter 2                                                                                          22
                       Schedule and Resource Estimates Are Important                               22
Interim Medicare       HCFA Plans To Define and Control Part A System                              22
Processing               Responsibilities
                       Test Strategy Lacking                                                       23
Environment Needs      HCFA Doubts Benefits of Performance Measures; None                          25
To Be More               Scheduled To Be Used
Effectively Managed    Management Oversight Not Established To Address Potential                   25
                         Year-2000 Problems
                       Conclusions                                                                 27
                       Recommendations                                                             28
                       Agency Comments and Our Evaluation                                          29

Chapter 3                                                                                          31
                       HCFA Has Not Performed an Investment Analysis for Current                   31
MTS Is Not Being         MTS Plan
Managed as an          Operating Site Decision Made Without Analyzing Alternatives or              35
                         Risks
Investment             HCFA Responding to Legislative Requirements but Still Lacks                 36
                         Required MTS Investment Strategy
                       Conclusions                                                                 39
                       Recommendations                                                             39
                       Agency Comments and Our Evaluation                                          41

Chapter 4                                                                                          43
                       Requirements Management Plan Completed Too Late in Systems                  43
Significant              Development Process
Systems-Development    Software Development Plan Is Inadequate                                     46
                       Configuration Management Plan Not Yet Implemented                           47
Practices Not Being    Systems Integration Plan Not Yet Developed                                  48
Followed, Increasing   HCFA Oversight of MTS Contractor’s System Development Is                    49
MTS Risk                 Risky
                       MTS Schedule Incomplete, Dangerously Compressed                             53




                       Page 12                              GAO/AIMD-97-78 Medicare Transaction System
             Contents




             Weaknesses in MTS Risk Management Require Attention Before                  57
               Proceeding With Development
             Conclusions                                                                 59
             Recommendations                                                             60
             Agency Comments and Our Evaluation                                          61

Appendixes   Appendix I: Comments From the Department of Health and                      64
               Human Services
             Appendix II: Comments From the Office of Management and                     70
               Budget
             Appendix III: Key Provisions of Laws, Regulations, and Best                 72
               Practices Relating to Information Technology Investments
             Appendix IV: Major Contributors to This Report                              83

Tables       Table 3.1: Part A Costs Per Claim for Fiscal Years 1994-1996                34
             Table 3.2: Part B Costs Per Claim for Fiscal Years 1994-1996                34
             Table 4.1: Critical Unmitigated MTS Risks                                   58

Figures      Figure 1.1: Strategy For Transition to the Medicare Transaction             18
               System
             Figure 4.1: Volatility of MTS Requirements                                  45
             Figure 4.2: Software Cost Estimating Process                                52
             Figure 4.3: Medicare Transaction System Program Schedule                    55
               Revisions
             Figure 4.4: Medicare Transaction System Program Schedule for                56
               Managed Care—Release 1




             Page 13                              GAO/AIMD-97-78 Medicare Transaction System
Contents




Abbreviations

AIMD       Accounting and Information Management Division
CCA        Clinger-Cohen Act
CIO        chief information officer
CMM        capability maturity model
FASA       Federal Acquisition and Streamlining Act
GAO        General Accounting Office
HCFA       Health Care Financing Administration
HHS        Department of Health and Human Services
IRM        information resources management
IV&V       independent verification and validation
MTS        Medicare Transaction System
OMB        Office of Management and Budget
PRA        Paperwork Reduction Act of 1995
SA-CMM     Software Acquisition Capability Maturity Model
SAF        systems assessment framework
SEI        Software Engineering Institute
SLIM       software life-cycle management


Page 14                           GAO/AIMD-97-78 Medicare Transaction System
Page 15   GAO/AIMD-97-78 Medicare Transaction System
Chapter 1

Introduction


                     Medicare is the nation’s largest health insurer, serving about 38 million
                     Americans by providing federal health insurance to individuals 65 or older
                     and to many of the nation’s disabled. It now provides over $200 billion in
                     health care benefits annually. Medicare’s day-to-day operations are run by
                     the Health Care Financing Administration (HCFA) under the Department of
                     Health and Human Services (HHS). HCFA uses about 70 intermediary and
                     carrier claims processing contractors to administer the Medicare program.
                     Intermediaries are the contractors that handle part A claims submitted by
                     hospitals, skilled nursing facilities, hospices, home health agencies, and
                     rehabilitation agencies. Carrier contractors handle part B claims submitted
                     by physicians, laboratories, equipment suppliers, outpatient providers, and
                     other practitioners. In December 1996, contractors were using three
                     different systems to process part A claims and six different systems to
                     process part B claims.

                     Medicare is expected to process over 1 billion claims and pay $288 billion
                     in benefits per year by 2000. With more claims creating a rapidly
                     expanding workload, HCFA is planning to replace its current
                     claims-processing systems in order to be able to handle the expected
                     increase in numbers of claims and provide better service to its customers.


                     In January 1994, as part of its plans to improve the efficiency and
The Medicare         effectiveness of Medicare program operations, HCFA awarded a contract to
Transaction System   a software developer to design, develop, and implement a new
                     government-owned, automated claims-processing information system, the
                     Medicare Transaction System (MTS).1 HCFA intends to replace the claims
                     processing functions being performed by the nine different systems with a
                     single, unified MTS having improved capabilities to help achieve significant
                     advances in Medicare management and operations.

                     The specific goals of MTS are to improve service to beneficiaries and
                     providers; reduce administrative expenses; allow better oversight of
                     Medicare contractors’ operations; better protect program funds from
                     waste, fraud, and abuse; and accommodate managed care and alternative
                     payment methodologies.




                     1
                      MTS is used throughout the report to refer to the software development project as well as other
                     initiatives, such as interim-phase activities, MTS claims processing sites, and telecommunications.



                     Page 16                                            GAO/AIMD-97-78 Medicare Transaction System
                       Chapter 1
                       Introduction




HCFA Revised Its MTS   HCFA  initially expected contractors to begin processing claims using MTS in
Transition and         late 1996 and to implement MTS at all Medicare contractor locations by
Implementation Plans   December 1998. However, because of difficulty in defining system
                       requirements and a revised system development approach to minimize
                       risk, HCFA now expects to have its first MTS software module—managed
                       care—implemented by June 1998, and complete implementing the
                       remaining modules in 2000 and beyond. It also plans to award contracts
                       for an MTS data operations and analysis center and two MTS claims
                       processing sites in late 1997, and move the claims processing workload to
                       these two processing centers from the existing claims processing locations
                       as the MTS modules become available.

                       The original MTS project schedule was developed on the basis of a grand
                       design approach, in which the complete system would be implemented at
                       one time. HCFA changed its implementation plan to a phased approach after
                       our January 1994 report, in which we discussed the reduced financial,
                       schedule, and technical risks associated with phased implementation
                       strategies.2 HCFA’s new approach includes deploying MTS in increments and
                       making necessary changes to its existing systems to allow claims
                       processing past 2000.

                       Specifically, HCFA plans to move from its current claims processing
                       environment to a fully implemented MTS in two phases—first, to an interim
                       Medicare processing environment and then to a full MTS environment. For
                       the interim phase, HCFA (1) will require its contractors to convert three
                       part A and six part B systems to a single part A and single part B system,3
                       (2) is transferring claims processing from about 45 contractor sites to
                       about 20 sites nationwide, and (3) has funded its contractors to modify any
                       existing software that will still be in use to ensure reliable operations
                       through the change of century. (See figure 1.1.) For the final phase, HCFA
                       will transfer the existing part A and part B systems from the few remaining
                       processing sites to the two planned MTS processing centers. Then,
                       applicable software in these systems will be replaced by MTS modules as
                       they become available, until these systems are completely replaced by MTS.




                       2
                        Medicare: New Claims Processing System Benefits and Acquisition Risks (GAO/HEHS/AIMD-94-79,
                       Jan. 25, 1994) and Medicare Transaction System: Strengthened Management and Sound Development
                       Approach Critical to Success (GAO/T-AIMD-96-12, Nov. 16, 1995).
                       3
                        One each of the existing part A and part B systems will become the single systems used to process all
                       part A and part B claims.



                       Page 17                                           GAO/AIMD-97-78 Medicare Transaction System
                                                                Chapter 1
                                                                Introduction




Figure 1.1: Strategy for Transition to the Medicare Transaction System

       From:                                                                          To:
       9 separate claims-processing                                                   A single part A and a single part B
       systems at about 45 sites                                                      claims-processing system at about 20 sites


                 A1                B1                                                          A
                                             B1                                                                  B
        B4             A2                                                      B1                      A
                                   B5                A3                                                    B
                                                               A3                                                           A
             A2               A2                                          B6               A                                         B
                                             A3                                                                     A
                      B2                                                                           B
                                   B4                               B6                                     B                    B
                                                          B3                                                            B
                                             B5
                                                                     B3                                         B               B
                                        B5
                                                     A3




       From:                                                                          To:
       A single part A and a single part B                                            The Medicare Transaction
       claims-processing system at about 20 sites                                     System at 2 sites


                  A
                                             B
                          A
                                    B
                                                               A
                                                                                               MTS
             A                                                            B

                      B
                                                 A
                                                                                                                        MTS
                                   B                                B
                                                          B

                                             B                       B




                                                                HCFA  has begun its interim-phase activities. It selected the Florida Shared
                                                                System as the single part A system, converted one of the two remaining
                                                                part A systems to the Florida system, and has initiated action to have the
                                                                other system’s contractors begin converting to the Florida system. Second,
                                                                it has asked system user groups to present proposals for consolidating the
                                                                part A processing sites. HCFA awarded a contract on April 8, 1997, for the



                                                                Page 18                                    GAO/AIMD-97-78 Medicare Transaction System
                     Chapter 1
                     Introduction




                     single part B system. Also, it has provided limited funding for several
                     system contractors to begin making millennium-related software changes.

                     HCFA’s final-phase activities include, in addition to the 1994 MTS
                     development contract, issuing a request for proposals for two MTS claims
                     processing sites and one data operations and analysis center. The contract
                     for these sites, originally scheduled to be awarded in March 1997, is now
                     planned for late 1997. The first MTS software module is scheduled to be
                     installed at those sites in May 1998.

                     On April 4, 1997, HCFA announced that, as a result of a recent management
                     review, it was redirecting its MTS contractor to focus solely on the
                     managed care module while it examines alternative ways to achieve its MTS
                     goals.4 HCFA concluded that its vision of MTS as the best information
                     technology to take Medicare into the 21st century had not changed.


                     Our objectives were to determine the extent to which HCFA is
Objectives, Scope,   (1) effectively managing its interim Medicare processing environment,
and Methodology      including planning for and correcting year 2000-related computer
                     problems, (2) using required practices to manage MTS as an investment,
                     and (3) applying sound systems development processes to reduce risks.

                     To review HCFA’s management of its interim environment, we analyzed
                     documents supporting decisions to select part A and part B systems and to
                     consolidate existing processing sites. We also interviewed HCFA officials
                     responsible for ensuring that claims are properly processed during this
                     period. Further, we discussed HCFA’s interim transition decisions with
                     applicable claims system contractors.

                     We assessed HCFA’s activities to address the millennium change by
                     (1) assessing millennium-related project plans and directives,
                     (2) reviewing related budget and funding documents, and (3) discussing
                     these activities with officials responsible for HCFA’s millennium conversion
                     and with HCFA’s system contractors. In addressing HCFA’s efforts in
                     preparing for the millennium, we drew heavily on government and
                     private-industry testimony and guidance.

                     To assess HCFA’s management of MTS as an investment we applied the
                     following criteria:

                     4
                      Managed Care is the first of six planned releases. The remaining five releases, in order of planned
                     implementation are the common working file and the beneficiary insurance file, consolidated financial
                     server, carrier claims processing, intermediary claims processing, and encounter data processing.



                     Page 19                                           GAO/AIMD-97-78 Medicare Transaction System
    Chapter 1
    Introduction




•   The Clinger-Cohen Act of 1996 (P.L. 104-106, Division E; Feb. 10,
    1996) (Effective Aug. 8, 1996).
•   The Paperwork Reduction Act of 1995, as amended (P.L. 104-13; May 22,
    1995).
•   The Federal Property and Administrative Services Act of 1949, § 313(b), as
    amended by the Federal Acquisition And Streamlining Act of 1994 (FASA),
    P.L. 103-355; October 13, 1994.
•   The Government Performance and Results Act of 1993 (P.L. 103-62; Aug. 3,
    1993).
•   Office of Management and Budget Circular A-11, Part 3, “Planning,
    Budgeting, and Acquisition of Fixed Assets,” July 1996; Circular A-130
    Revised, “Management of Federal Information Resources” (Feb. 8, 1996);
    Bulletin 95-03, “Planning and Budgeting for the Acquisition of Fixed
    Assets” (June 27, 1995) (now superseded by OMB Circular A-11); Evaluating
    Information Technology Investments, A Practical Guide (Version 1.0,
    November 1995); and executive memorandum M-97-02, Funding
    Information Systems Investments (Oct. 25, 1996).

    (See appendix II for key segments of these investment management laws,
    regulations, and guidance.)

    In assessing HCFA’s management of MTS as an investment, we analyzed HCFA
    documentation related to planning and managing information technology
    and interviewed members of HCFA’s MTS management committees. We also
    obtained and analyzed HCFA’s MTS cost models and discussed them with
    HCFA officials in assessing how well HCFA had identified and justified
    alternatives.

    To assess HCFA’s use of effective systems development processes, we
    compared them with HCFA’s systems development life cycle procedures,
    our Systems Assessment Framework (SAF), criteria contained in
    Carnegie-Mellon University’s Software Engineering Institute’s (SEI)
    Software Capability Maturity Model and Software Acquisition Capability
    Maturity Model, and other generally accepted systems development
    practices.5 We also analyzed HCFA’s software and systems development
    management documents including HCFA’s draft MTS Requirements
    Management Plan, risk management reports, systems development
    methodology, and MTS Change Management Manual.




    5
     Systems Assessment Framework: A Guide for Reviewing Information Management and Technology
    Issues in the Federal Government, version 1.0, (GAO, August 1996).



    Page 20                                      GAO/AIMD-97-78 Medicare Transaction System
Chapter 1
Introduction




To determine how HCFA is overseeing MTS development, we reviewed
applicable technical plans and MTS software cost and schedule estimation
model results, and compared them to generally accepted practices. To
assess whether HCFA is managing the MTS schedule, we obtained and
analyzed HCFA’s monthly MTS program schedules to determine whether
they were realistic and contained all necessary schedule elements. To
determine whether HCFA is appropriately identifying and managing risks
associated with the MTS development, we reviewed HCFA and MTS
contractors’ risk abatement activities reported in the risk management
reports. We also obtained copies of monthly risk reports developed by
HCFA and MTS contractors and assessed how these risks were being
mitigated. Finally, we interviewed HCFA officials responsible for MTS
development, as well as HCFA’s contractors for system development and
for independent verification and validation. Further, we analyzed a
commissioned study on program controls, interviewed the contractor that
performed the study, and obtained HCFA’s response to the
recommendations.6

We performed our work at HCFA headquarters in Baltimore, Maryland, and
its Kansas City, Missouri, regional office; the MTS software development
contractor’s offices in Tampa, Florida, and Baltimore, Maryland; the
Independent Verification and Validation (IV&V) contractor’s office in
Baltimore Maryland; and several part A and part B system contractors’
offices.

We performed our work from October 1996 through April 1997, in
accordance with generally accepted government auditing standards. HHS
and OMB provided written comments on a draft of this report. Their
comments are presented and evaluated in chapters 2, 3, and 4, and are
included in appendixes I and II.




6
 HCFA MTSI Program Management Final Assessment Report, (Robbins-Gioia, July 11, 1996).



Page 21                                        GAO/AIMD-97-78 Medicare Transaction System
Chapter 2

Interim Medicare Processing Environment
Needs To Be More Effectively Managed

                       HCFA   is not effectively managing its interim Medicare claims processing
                       environment, increasing the risks of transition delays, excessive costs, and
                       not achieving the goals of the transition. The transition from the current
                       claims processing environment to MTS represents a major challenge in that
                       it is a larger systems-conversion effort than any previously undertaken by
                       HCFA, and is being performed concurrently with HCFA’s management of MTS
                       development. Although such a major undertaking requires careful planning
                       and management, HCFA has not followed generally accepted program
                       management practices, which call for detailed plans for systems’
                       transitions and modifications. Also, it has not ensured that it will avoid the
                       potential systems-related problems that accompany the year-2000 change.


                       A schedule and estimate of needed resources for each major stage of the
Schedule and           transition to the MTS processing environment would help HCFA manage and
Resource Estimates     coordinate its transition activities and ensure that required resources are
Are Important          available at each stage. Specifically, this would include overseeing
                       planning, testing, and implementing the conversion of the nine part A and
                       part B systems to two, managing the site preparation and migration from
                       45 processing centers to about 20, shifting the workloads of local
                       contractors who decide not to renew their contracts for processing
                       Medicare claims, and converting systems to address potential year-2000
                       problems. The need for such a transition plan is highlighted in our
                       August 1996 guide for reviewing information management and technology
                       issues.1

                       HCFA agreed that a transition schedule and resource estimates would be
                       useful. As part of a consulting contract to help HCFA develop a complete
                       MTS initiatives program schedule, the consultant is to assist HCFA in
                       preparing a schedule and resource estimates for HCFA’s transition and then
                       integrating them into the overall MTS program development schedule work.
                       These are due to be completed by late spring 1997.


                       HCFA selected its single part A system contractor on May 23, 1996, and as of
HCFA Plans To Define   March 17, 1997, had paid the contractor about $870,000 to convert one of
and Control Part A     the two remaining part A systems to the selected system. However, HCFA
System                 has no legally binding document to define the responsibilities of the
                       selected part A systems contractor for the conversion. Instead, the work is
Responsibilities       being performed under a 1991 agreement that the contractor will maintain
                       the part A system. This agreement does not define the contractor’s

                       1
                        Systems Assessment Framework (GAO, August 1996).



                       Page 22                                      GAO/AIMD-97-78 Medicare Transaction System
                            Chapter 2
                            Interim Medicare Processing Environment
                            Needs To Be More Effectively Managed




                            responsibilities for consolidating and maintaining a single part A system,
                            supporting the movement of claims processing from current sites to MTS
                            operating sites, and developing agreements with users which outlines
                            customer expectations warranties and guarantees.

                            Generally accepted practices for managing and overseeing such work
                            include (1) preparing a statement to document the requirements of
                            relevant systems contractors and (2) establishing a change control
                            mechanism to manage and control changes to these requirements. HCFA
                            awarded a contract defining responsibilities for the selected part B system
                            on April 8, 1997. According to HCFA officials, they plan to (1) prepare a
                            statement of responsibilities for the part A system work by the end of May
                            1997, and (2) use this statement as a basis for a legally binding agreement
                            with the part A system contractor. According to the MTS project manager, a
                            board to manage changes to part A began work on April 3, 1997. HCFA also
                            intends to establish a similar control board for the part B system.


                            As addressed in our guide for reviewing information management and
Test Strategy Lacking       technology issues, a sound test strategy should include testing, such as the
                            selected part A and part B systems, to ensure that they meet certain
                            specified requirements.2 To date, HCFA’s involvement in the ongoing part A
                            conversion has been limited primarily to informal discussions with
                            contractors and users.

                            The testing should measure whether the systems (1) perform required
                            claims processing and other functions, (2) have the capacity for
                            processing the total part A and part B workload, (3) can process claims
                            using data converted from the old systems, and (4) will operate in the year
                            2000 and beyond. A test strategy would ensure that sufficient testing is
                            conducted and the results evaluated so that converted systems will be able
                            to reliably process the increased workload; this is vital to ensuring that
                            Medicare claims will be processed in an uninterrupted fashion.

                            To date HCFA has not taken any of the following steps, each necessary for a
                            sound testing approach:

                        •   defining its role in planning or overseeing the testing,
                        •   assigning responsibility for overseeing and approving the part A and part B
                            conversion or approving the contractors’ acceptance testing and results,


                            2
                             Systems Assessment Framework (GAO, August 1996).



                            Page 23                                      GAO/AIMD-97-78 Medicare Transaction System
    Chapter 2
    Interim Medicare Processing Environment
    Needs To Be More Effectively Managed




•   developing criteria for evaluating the contractors’ test plans to ensure that
    the systems can adequately handle the combined increased workload and
    that the systems will operate properly in the year 2000 and beyond,
•   identifying how it will provide resources to manage the testing,
•   providing for an independent validation and verification of whether test
    results meet requirements, and
•   determining how it will ensure that problems uncovered in testing are
    corrected promptly.

    According to HCFA officials, their contractors routinely test and implement
    systems changes in response to legislative mandates, without a HCFA test
    plan or strategy. Consequently, they said, they plan to rely largely on their
    contractors to successfully implement transition-related changes. They
    further stated that they expect system users (local Medicare contractors)
    to ensure that the testing is adequate.

    The transition to the MTS system differs in several key ways, however, from
    routine changes in response to legislation. First, the selected part A and
    part B contractors may not have a particularly high incentive to properly
    make these conversions, since HCFA plans to eliminate these contractors
    when MTS is fully implemented. Second, converting a system owned by
    someone else may be more difficult than making changes to an existing
    system, and the selected contractors have no choice over which systems
    will be converted to their system—all part A systems will be converted to
    the selected part A system, and all part B systems will be converted to the
    selected part B system. Third, these conversions involve significantly more
    data and will require more system capacity than routine modifications.
    Finally, as we reported in 1992 and 1994, two of HCFA’s previous system
    conversions did encounter problems.3 Specifically, when HCFA shifted an
    outgoing contractor’s claims processing workload to another contractor’s
    system, serious disruptions in getting claims processed and payments
    made to physicians ensued, as did increases in erroneous payments and
    decreases in payment safeguards, possibly resulting in overpayments.

    At the conclusion of our review, the MTS project manager told us that HCFA
    is exploring the feasibility of procuring an IV&V contractor for the
    transitions to the single part A and part B systems.




    3
     Medicare: Shared Systems Policy Inadequately Planned and Implemented (GAO/IMTEC-92-41, Mar. 18,
    1992) and Medicare: Shared System Conversion Led to Disruptions in Processing Maryland Claims
    (GAO/HEHS-94-66, May 23, 1994).



    Page 24                                        GAO/AIMD-97-78 Medicare Transaction System
                       Chapter 2
                       Interim Medicare Processing Environment
                       Needs To Be More Effectively Managed




                       Measuring the results of the implementation of the interim Medicare
HCFA Doubts            systems is required by legislation and can be useful to HCFA in
Benefits of            understanding and tracking how these systems have altered Medicare
Performance            processing. The Clinger-Cohen Act requires agency heads to implement a
                       process for managing the risks of information technology acquisitions,
Measures; None         which includes a method of identifying quantifiable measurements for
Scheduled To Be Used   determining the net benefits and risks of the investment. Further the
                       agency head is to ensure that performance measurements are prescribed
                       for the information technology used by the agency and that they measure
                       how well the information technology supports the agency’s programs. The
                       process is to provide the means for senior management to obtain timely
                       information regarding the progress of the investment. The Paperwork
                       Reduction Act also requires agencies to use effective methods for
                       measuring the progress of technology in meeting their goals, and OMB
                       guidance emphasizes the need for such performance measures.

                       However, HCFA’s plan for evaluating the performance of its transition
                       systems are inadequate. Its transition plan does not contain elements that
                       would allow the agency to determine (1) whether Medicare systems will
                       continue to provide reliable processing and adequate service throughout
                       the transition period, (2) whether expected administrative savings are
                       being achieved, and (3) how MTS plans might be refined on the basis of
                       results of the transition systems, such as determining the design and
                       configuration of MTS. HCFA officials said they do not believe it would be
                       cost-beneficial to use the agency’s limited resources to develop
                       performance measures for interim systems that will be replaced by MTS.
                       We believe performance measures are needed to ensure that this complex
                       and important interim phase is properly implemented. Appendix II cites
                       legislation that mandates the implementation and use of such performance
                       measures.


                       As we approach the year 2000, information systems worldwide could
Management             malfunction or produce incorrect information simply because they have
Oversight Not          not been designed to handle dates beyond 1999; Medicare claims
Established To         processing systems are no different. Failure to adjust the systems for the
                       year 2000 could result in payment delays and in losses due to bypassed
Address Potential      automated controls that flag claims that should be paid by the
Year-2000 Problems     beneficiaries’ other insurers. The danger is that, if not corrected, systems
                       could well read the computer-coded “00” as 1900, not 2000. All
                       date-dependent calculations would therefore be affected, having an
                       obvious impact on age and beneficiary status.



                       Page 25                                   GAO/AIMD-97-78 Medicare Transaction System
Chapter 2
Interim Medicare Processing Environment
Needs To Be More Effectively Managed




The timing of HCFA’s transition strategy will make addressing the year-2000
issue even more of a major challenge. For example, the single part B
system will face converting five other systems to the selected system,
while concurrently modifying the selected system to make it year-2000
compliant. Because HCFA now estimates it will not complete the transition
to the single part B system until shortly before 2000, it has provided initial
funding to make four of the six part B systems year-2000 compliant—the
selected single part B system and three of the remaining five systems. The
Medicare part A systems contractor has started to modify its software to
make it year-2000 compliant, and estimates that it will complete testing
this software and be ready to implement it by July 1997.

Because HCFA’s Medicare contractors routinely make system modifications
in response to legislation, HCFA is relying on them to make year-2000
changes. However, the scope of the work required for contractors to make
year-2000 changes is significantly broader than other systems changes
contractors have had to make in the past. Specifically, it requires review of
all software programs and systems interfaces, and all systems components
that can be affected by date problems; this includes hardware, operating
systems, communications applications, and databases. Yet, again, HCFA is
not adequately overseeing this process and further is not requiring
contractors to certify that they will correct the year-2000 problem.

Adequately addressing the potential year-2000 systems problems for the
Medicare program requires management attention and a wide range of
managerial activities. As detailed in our February 1997 year-2000
assessment guide,4 among the most important of these activities are
(1) developing an overall year-2000 plan, (2) identifying responsibilities for
managing and monitoring year-2000 actions, (3) preparing an assessment
of the severity and timing of potential year-2000 impact, (4) conducting an
inventory of the systems on which Medicare claims processing depends,
(5) prioritizing and scheduling work to convert, replace, or eliminate these
systems, (6) developing validation strategies and test plans for systems,
(7) addressing interface and data exchange issues, and (8) developing
contingency plans for critical systems in the event of failure.

According to HCFA officials, the agency has prepared an overall year-2000
plan for its internal systems, intends to include Medicare claims
processing systems in this plan at a future date, and is collecting
information from systems contractors on both their progress and their
planned year-2000 activities. To date, however, HCFA has not required

4
 Year 2000 Computing Crisis: An Assessment Guide (GAO, February 1997).



Page 26                                        GAO/AIMD-97-78 Medicare Transaction System
              Chapter 2
              Interim Medicare Processing Environment
              Needs To Be More Effectively Managed




              systems contractors to submit year-2000 plans for approval. Further, it
              does not have contracts or other specific legal agreements with any
              contractors, other than the selected contractor for the single part B
              system, which state how or when the year-2000 problem will be corrected
              or whether contractors will certify that they will correct the problem.

              HCFA  has also not identified critical areas of responsibility for year-2000
              activities. Although HCFA’s regional offices have a role in overseeing
              contractor efforts, their specific year-2000 responsibilities have not been
              defined, nor has guidance been prepared on how to monitor or evaluate
              contractor performance. While HCFA has been assessing the impact of the
              year-2000 on its internal systems, it has not completed a similar review of
              Medicare contractors’ claims processing systems. Further, HCFA has not
              required its contractors to prepare an assessment of the severity of
              potential year-2000 problems.

              On March 26, 1997, HCFA asked its Medicare contractors to provide an
              inventory of the Medicare applications affected by the year-2000 change
              and their schedules for converting, replacing, or eliminating these systems.
              However, HCFA had no plans to independently validate the contractors’
              strategies and test plans. In addition, while HCFA has asked the contractors
              to identify their system interfaces, it had no plans for approving the
              contractors’ approaches for addressing interface and data exchange
              issues.

              HCFA had also not developed contingency plans in the event that year-2000
              systems fail. HCFA officials are again relying on the contractors to identify
              and complete the necessary work in time to avoid problems. Yet, the part
              A and part B contractors not only have not developed contingency plans,
              they said they do not intend to do so because they believe this is HCFA’s
              responsibility.

              On April 22, 1997, at the conclusion of our review, HCFA provided us with
              information regarding a technical workgroup, which is to identify and
              resolve any year-2000 technical issues. However, this workgroup, which
              was established on January 10, 1997, had not yet discussed or resolved any
              technical issues.


              HCFAfaces a challenging array of tasks as it operates in an interim
Conclusions   Medicare claims processing environment over the next few years. It
              expects this interim phase to better ensure a successful transition to MTS



              Page 27                                   GAO/AIMD-97-78 Medicare Transaction System
                      Chapter 2
                      Interim Medicare Processing Environment
                      Needs To Be More Effectively Managed




                      and ensure reliable claims processing during this period. However, HCFA
                      has not prepared the necessary plans to help it manage this interim period,
                      help ensure that these goals will be met, or evaluate the performance of its
                      interim claims processing environment. Further, unless timely, effective
                      systems changes are implemented as the year-2000 approaches, HCFA may
                      be unable to process claims accurately and within required time frames.


                      To better ensure the success of claims processing during the interim
Recommendations       before MTS implementation, we recommend that the Secretary of Health
                      and Human Services direct that the Administrator, Health Care Financing
                      Administration, manage and be accountable for the following actions.

                  •   Preparing a detailed transition plan, which includes sections that
                      (1) provide a schedule and estimate of resources needed for each major
                      stage of the transition to the interim processing environment (2) define
                      how software changes to the part A and part B systems will be controlled
                      and managed, (3) identify how HCFA will ensure reliable processing while
                      reducing the number of processing centers and shifting the workloads of
                      local Medicare contractors who decide not to renew their Medicare claims
                      processing contracts, and (4) define how systems will be converted to
                      address potential year-2000 problems.
                  •   Obtaining a legally binding agreement with the part A contractor, which
                      identifies all responsibilities for conversion and maintenance of the part A
                      system, before providing any additional funds for this effort.
                  •   Preparing plans for conducting thorough testing before converting part A
                      and part B systems. These plans should, at a minimum, (1) define HCFA’s
                      role in planning or overseeing the testing, (2) assign responsibility for
                      overseeing and approving the part A and part B conversion or approving
                      the contractors’ acceptance testing and results, (3) develop criteria for
                      evaluating the contractors’ test plans to ensure that the systems can
                      adequately handle the combined increased workload and that the systems
                      will operate properly in the year 2000 and beyond, (4) identify how it will
                      provide resources to manage the testing, (5) provide for an independent
                      validation and verification of whether test results meet requirements, and
                      (6) determine how it will ensure that problems uncovered in testing are
                      corrected promptly.
                  •   Establishing a means of assessing performance in the crucial early stages
                      of the transition, and applying any lessons learned to planning for MTS. The
                      performance measures should include elements that allow HCFA to
                      determine (1) whether Medicare systems will continue to provide reliable
                      processing and adequate service throughout the transition period,



                      Page 28                                   GAO/AIMD-97-78 Medicare Transaction System
                         Chapter 2
                         Interim Medicare Processing Environment
                         Needs To Be More Effectively Managed




                         (2) whether expected administrative savings are being achieved, and
                         (3) how the design and configuration of MTS might be refined on the basis
                         of results of the interim systems performance.
                     •   Helping ensure the reliable operation of its systems through the year-2000
                         by identifying responsibilities for managing and monitoring year-2000
                         actions, preparing an assessment of the severity and timing of potential
                         year-2000 impact, and developing contingency plans for critical systems in
                         the event of failure. Further, HCFA should require its contractors to submit
                         for review and approval (1) plans for identifying and correcting potential
                         problems, including a certification that their changes will correct the
                         problem, (2) validation strategies and test plans for systems, and (3) plans
                         for addressing interface and data exchange issues. Finally, HCFA should
                         regularly report to HHS on its progress in addressing the year 2000 issue,
                         including the amount of funds spent on this effort.


                         HHS agreed with our recommendations to more effectively manage the
Agency Comments          interim Medicare processing environment. HHS stated that it
and Our Evaluation
                     •   has prepared a part A transition plan and plans to complete the part B
                         transition plan in early summer 1997;
                     •   agrees with our recommendation to conduct thorough testing before
                         converting the part A and part B systems;
                     •   has established performance metrics to monitor the software development
                         contract and, as a result, was able to initiate corrective action when the
                         work was not progressing satisfactorily; and
                     •   has requested implementation plans from its contractors on their progress
                         in bringing their systems into millennium compliance, and is in the process
                         of analyzing these plans.

                         HHS said that both the Department and HCFA are committed to
                         implementing our recommendations. It also commented that many of our
                         recommendations will assist them in better managing the transformation
                         from antiquated, redundant information and processing systems to the
                         planned modernized system.

                         We believe that the actions HHS has outlined to manage the interim
                         Medicare processing environment are positive and expect that, if
                         effectively implemented, they will help the Department and HCFA achieve a
                         successful transition to MTS. In addition, just as software development
                         performance measures are critical to the software development process,
                         performance measures are also critical to the success of the transition.



                         Page 29                                   GAO/AIMD-97-78 Medicare Transaction System
Chapter 2
Interim Medicare Processing Environment
Needs To Be More Effectively Managed




Further, while requesting year-2000 implementation plans from HCFA
contractors can also help by improving the overall understanding of the
status of the year-2000 effort, to ensure success, these contractors must
have their responsibilities specifically defined and their actions reviewed
and approved by HCFA.

OMB  concurred with our recommendations encouraging HCFA to take steps
to more adequately plan for the transition to standardized part A and part
B systems, and said it is continuing to work with HCFA on this transition.
OMB also said it will look into our recommendations concerning the
year-2000 contractor systems’ conversion issue, and take appropriate
action.




Page 30                                   GAO/AIMD-97-78 Medicare Transaction System
Chapter 3

MTS Is Not Being Managed as an Investment


                       HCFA has begun to respond to legislative and OMB requirements for
                       managing information technology projects as investments; however, it has
                       not applied effective investment practices in managing MTS. Such practices
                       include: preparing valid cost-benefit analyses, considering viable
                       alternatives and assessing risks, and having senior management involved
                       in the critical decision-making process. Without these, HCFA has no
                       assurance that its planned system will be cost-effective, risk-averse, and
                       support the agency’s mission and goals.


                       As early as 1992, when HCFA began planning for MTS, OMB Circular A-11
HCFA Has Not           required that planned information technology acquisitions be based on a
Performed an           cost-benefit analysis. Similarly, since 1992, OMB Circular A-94 has required
Investment Analysis    that decisions to initiate government projects be based on an analysis of
                       expected life-cycle costs and benefits (justified on economic grounds
for Current MTS Plan   using net present value calculations), and that alternative means of
                       achieving program objectives be considered.1

                       Over the years, agencies have experienced numerous failures in acquiring
                       information technology. Billions of dollars have been wasted on systems
                       that did not work as planned or cost significantly more than expected.
                       Both the Congress and OMB have recognized this problem and have
                       recently established requirements that more specifically require agencies
                       to manage their major acquisitions using sound information technology
                       investment practices. For example, the Clinger-Cohen Act requires
                       agencies to focus more on the results achieved through their investments
                       and to use more rigor and structure in their processes for selecting and
                       managing their information technology projects. The act also provides OMB
                       with additional authority to ensure that the act’s provisions are carried
                       out. In an October 1996 policy memorandum, OMB specified that major
                       investments in information technology should demonstrate a projected
                       return on the investment that is “clearly equal to or better than alternative
                       uses of available public resources.” (See appendix II for details.)

                       Since HCFA began its MTS project in 1992, the estimated cost to develop MTS
                       software and move to the new system has increased from about
                       $151 million to about $1 billion.2 The $1 billion includes estimated costs to


                       1
                        Present value dollars represent the current worth of an amount or series of amounts payable or
                       receivable (cost-benefits) in the future, determined by discounting the future amount or amounts at a
                       predetermined rate of interest.
                       2
                        All dollar amounts presented in this report are expressed as undiscounted current dollars unless
                       otherwise noted. They are not adjusted for inflation and are not present values.



                       Page 31                                           GAO/AIMD-97-78 Medicare Transaction System
                             Chapter 3
                             MTS Is Not Being Managed as an Investment




                             transition to the MTS environment and acquire operating sites. In spite of
                             these significant estimated project cost increases, as discussed below,
                             HCFA’s cost analyses have not (1) identified the specific MTS applications
                             that justify its estimates of $2.1 billion in reduced program costs over 10
                             years (stated as a present value), (2) adequately documented the
                             assumptions used to estimate total administrative savings of almost
                             $1.5 billion, or (3) addressed available alternative solutions to MTS, which
                             could provide much of the estimated program and administrative savings
                             within a shorter time and at substantially lower costs. When asked about
                             the lack of a cost-benefit analysis, HCFA responded that the benefits of MTS
                             were obvious.


Reduced Program Costs        HCFA’s initial MTS cost-benefit analysis, developed by an outside contractor
Not Linked to Specific MTS   in April 1992, and updated in December 1993, compared the MTS alternative
Processing Improvements      with the status quo claims processing environment at that time of 80
                             contractors, 62 processing sites, and 22 claims processing systems.3 Since
                             that time, from February 1995 through November 1996, HCFA has
                             developed, in-house, a series of cost models, which update the cost and
                             savings estimates of the previous contractor-developed analyses. In
                             September 1996, HCFA’s Office of the Actuary provided a report of its
                             estimate of program savings from MTS, and these estimates were
                             incorporated into HCFA’s November 1996 cost model.

                             HCFA’s Office of the Actuary estimated that MTS would provide annual
                             programmatic savings of $570 million by fiscal year 2005, resulting in
                             10-year life cycle (fiscal years 1997 through 2006) programmatic savings of
                             about $2.1 billion (stated as a present value). It estimated that program
                             cost savings would result from (1) increasing the use of automated edits to
                             identify abusive billing practices and deny related claims, (2) improving
                             and standardizing “medical necessity” review edits, which would result in
                             an increase in the number of inappropriate claims identified and denied,
                             and (3) developing a centralized beneficiary insurance file, which would
                             increase the amount of savings under the Medicare Secondary Payment
                             program.4 However, the Office of the Actuary qualified its savings
                             estimate, stating that too many details of MTS implementation were not

                             3
                              HCFA’s 1993 alternative analysis only compares MTS versus continuing its current operations. It did
                             not assess viable alternative solutions to MTS.
                             4
                              Medical necessity review edits involve identifying claims for inappropriate or excessive medical
                             services. Abusive billing involves such practices as billing for the same procedure twice although only
                             provided once or billing for an assistant surgeon when one was not warranted. Medicare secondary
                             payment savings result from identifying instances in which beneficiaries have other health insurance
                             that should be the primary payer, rather than Medicare.



                             Page 32                                            GAO/AIMD-97-78 Medicare Transaction System
                           Chapter 3
                           MTS Is Not Being Managed as an Investment




                           available, especially the exact nature of the edits that will be built into the
                           software and any requirement for Medicare contractors to implement
                           those edits. It concluded that because of this lack of information about the
                           MTS development, the actual savings associated with MTS could prove to be
                           significantly different from its estimate.

                           As of April 14, 1997, HCFA had not identified the exact nature of the edits
                           that will be built into MTS. Further, HCFA’s MTS development strategy is not
                           based on maximizing potential savings early. The MTS releases that include
                           new, but as yet undefined edit routines are not planned for
                           implementation until 1999. Finally, for several years both we and the HHS
                           Office of the Inspector General have reported that HCFA could save
                           hundreds of millions of dollars annually in program costs by immediately
                           implementing commercially available automated edits to detect and
                           prevent fraud, waste, and abuse.5 In May 1995 we reported that HCFA could
                           save billions of dollars by using commercially available software
                           containing edits to detect and correct billing abuses. HCFA is currently
                           evaluating this type of software.

                           We also reported in January 1996 that some contractors were using several
                           different medical review edits which, if implemented nationally, could
                           save up to $150 million annually by denying claims that were not medically
                           necessary.6 These edits can be implemented on existing systems without
                           spending hundreds of millions of dollars to develop MTS and install two MTS
                           claims-processing sites. HCFA has not developed a strategy or schedule to
                           implement these edits and is continuing to allow local contractors to
                           decide whether to use them.


Estimated Administrative   HCFA’s November 1996 MTS cost analysis estimates that the system will
Savings Not Adequately     provide net administrative savings of $697 million. This $697 million
Supported                  estimate is based on two key assumptions. First, during MTS’ 10-year
                           estimated life, expenditures for software improvements will be
                           substantially less than would have been required for the existing part A
                           and part B systems. HCFA projected that $788 million of its total $1.5 billion
                           administrative savings estimate would result from these reduced
                           expenditures. However, according to actual and estimated system
                           improvements trend data presented in HCFA’s budgets for fiscal years 1995

                           5
                            Commercial Technology Could Save Billions Lost to Billing Abuse (GAO/AIMD-95-135, May 5,
                           1995) and Antifraud Technology Offers Significant Opportunity to Reduce Health Care Fraud
                           (GAO/AIMD-95-77, Aug. 11, 1995).
                           6
                            Millions Can Be Saved by Screening Claims for Overused Services (GAO/HEHS-96-49, Jan. 30, 1996).



                           Page 33                                          GAO/AIMD-97-78 Medicare Transaction System
                                        Chapter 3
                                        MTS Is Not Being Managed as an Investment




                                        through 1998, if MTS were not implemented, HCFA would need only
                                        $83.6 million to implement improvements to existing systems during these
                                        10 years. In addition, HCFA did not include in-house costs to develop MTS in
                                        its administrative savings analysis, which it estimates will be about
                                        $49 million during fiscal years 1997 through 2000.

                                        Second, HCFA’s current administrative savings estimate assumes that total
                                        costs to process Medicare claims will continue to increase unless MTS is
                                        developed. Further, HCFA’s 1993 analysis, which was the last analysis to
                                        show how much it cost to process individual claims, assumed that,
                                        without MTS, costs per claim for part A and part B would continually
                                        increase from 1993 through 2002. However, actual contractor cost reports
                                        show that costs per claim for part A and part B actually decreased for
                                        fiscal years 1994 through 1996, from $1.41 to $1.27 and from $0.89 to $0.88
                                        respectively. (See tables 3.1 and 3.2.)

Table 3.1: Part A Costs Per Claim for
Fiscal Years 1994-1996                                                         Part A costs per claim
                                        Fiscal years                      HCFA estimatea              Actual costsb       Difference
                                        1994                                           $1.62                    $1.41          $0.21
                                        1995                                            1.66                     1.33           0.33
                                        1996                                            1.70                     1.27           0.43
                                        a
                                        HCFA’s 1993 MTS Alternative Analysis.
                                        b
                                            Medicare contractors’ fiscal years 1994-1996 expenditure reports.



Table 3.2: Part B Costs Per Claim for
Fiscal Years 1994-1996                                                         Part B costs per claim
                                        Fiscal years                      HCFA estimatea              Actual costsb       Difference
                                        1994                                           $1.06                    $0.89          $0.17
                                        1995                                            1.06                     0.94           0.12
                                        1996                                            1.07                     0.88           0.19
                                        a
                                        HCFA’s 1993 MTS Alternative Analysis.
                                        b
                                            Medicare contractors’ fiscal years 1994-1996 expenditure reports.



                                        In addition, HCFA’s 1993 savings estimates were based on an existing claims
                                        processing environment that included 62 separate contractor processing
                                        sites using 22 different software systems. HCFA assumed that it would
                                        replace the 22 different systems with MTS and make substantial reductions
                                        in the number of processing sites by developing MTS-based sites. However,




                                        Page 34                                            GAO/AIMD-97-78 Medicare Transaction System
                         Chapter 3
                         MTS Is Not Being Managed as an Investment




                         some of these savings may have already occurred without MTS as part of
                         HCFA’s interim consolidation effort. As we mentioned earlier, HCFA has
                         reduced the number of existing systems from 22 to 9, and is further
                         reducing them to only a standard part A and part B system prior to
                         implementing MTS. HCFA has also consolidated its 62 processing sites to
                         about 45 and plans further consolidations to about 20 sites.


HCFA Has Not Evaluated   OMB requires agencies to prepare cost-benefit analyses that consider
Alternative Solutions    alternative means of achieving program objectives. For example, when
                         evaluating a decision to acquire a capital asset, OMB requires that the
                         analysis should consider alternatives such as upgrading, renovating,
                         sharing or converting existing government property, leasing, or
                         contracting for services. Although HCFA has prepared several estimates of
                         MTS costs and benefits, none has included assessments of alternative
                         solutions to MTS. For example, HCFA has not assessed and compared the
                         costs and benefits of implementing readily available, commercial medical
                         review edits and abusive billing software routines to its plans for similar
                         MTS features. Likewise, HCFA has not assessed whether the actual and
                         planned reductions in existing systems and claims processing sites have
                         achieved or will substantially achieve most of the projected administrative
                         savings estimated for MTS.


                         As part of MTS initiatives, HCFA decided to consolidate its daily
Operating Site           claims-processing workload of about 2.6 million claims at two claims
Decision Made            processing sites using MTS software. It also plans to acquire a data
Without Analyzing        operations and analysis center. However, these decisions were made with
                         inadequate decision criteria or analysis for comparing alternatives.
Alternatives or Risks    Further, technical risk analyses to support the planned facilities are
                         incomplete.

                         In April 1996 HCFA issued a request for proposals for the three MTS
                         operating sites. The criteria for determining the number of processing sites
                         were limited to data storage and disaster recovery requirement
                         considerations. More explicit criteria were not used to evaluate and
                         prioritize alternatives such as HCFA’s interim-phase Medicare processing
                         sites or other processing centers such as the Department of Veterans’
                         Affairs data center in Austin, Texas.

                         Thorough risk assessments have not been conducted to ensure that the
                         planned MTS processing sites will perform as required and meet system



                         Page 35                                     GAO/AIMD-97-78 Medicare Transaction System
                         Chapter 3
                         MTS Is Not Being Managed as an Investment




                         goals. Three major steps commonly used in such assessments have not
                         been completed. First, a realistic workload analysis, using high volumes of
                         data as input and output to realistically simulate the Medicare system, has
                         not been conducted. Second, a formal capacity analysis has not been
                         conducted to determine if the commercial middleware7 already selected
                         on the basis of a market survey can handle the high frequency
                         transmission of input and output data required by MTS. Finally, a security
                         risk analysis is essential. In the MTS case, it is being prepared out of
                         sequence, after the security engineering analysis, and is not planned for
                         completion until the summer of 1997. The risk in this approach is that
                         when the security risk analysis is complete, any major security-related
                         risks identified during this analysis will need to be addressed in the
                         security engineering analysis. Thus, the security engineering analysis may
                         have to be modified at added cost to correspond to the security risk
                         analysis.


                         In response to explicit Clinger-Cohen Act requirements and OMB guidance,
HCFA Responding to       HCFA has begun action to follow practices essential for making informed
Legislative              information technology investment decisions. However, it does not yet
Requirements but Still   have consistent senior management involvement or investment-based
                         decision-making on MTS issues, and has not explicitly demonstrated how
Lacks Required MTS       MTS will help the agency meet its mission, goals, and objectives.
Investment Strategy
HCFA Taking Action but   The Clinger-Cohen Act requires agencies to designate a chief information
MTS Management           officer (CIO) to develop, maintain, and facilitate the implementation of a
Decisions Not Yet        sound and integrated information technology architecture.8 Further, this
                         act, and the Federal Acquisition Streamlining Act, require agencies to
Investment-Based         provide a means for senior management to obtain timely information
                         regarding the progress of an investment in an information system. In
                         addition, OMB guidance on investment management practices encourages
                         senior management to (1) be involved in making decisions in a disciplined
                         and structured management forum, (2) monitor the progress of ongoing
                         information technology projects against projected cost, schedule,
                         performance, and benefits, (3) document all management decisions and

                         7
                          The term middleware is used to describe software that resides between an application program and
                         an operating system and network, enabling diverse systems to communicate in a common client-server
                         environment.
                         8
                          The Paperwork Reduction Act, as amended by the Clinger-Cohen Act, specifically requires agencies,
                         such as HHS, to designate a CIO. Further, the conferees of this legislation also anticipated that major
                         subcomponents or bureaus, such as HCFA, would also appoint CIOs responsible for ensuring that the
                         management and acquisition of information technology is implemented consistent with the law.



                         Page 36                                            GAO/AIMD-97-78 Medicare Transaction System
Chapter 3
MTS Is Not Being Managed as an Investment




supporting data to avoid replication of effort in analysis, and (4) evaluate
how common problems and their solutions apply to other information
technology projects.9

In response to the Clinger-Cohen Act, HHS designated a CIO. The CIO told us
that HHS intends to be more involved in overseeing HCFA’s management of
this project than in it has been in prior years.

Similarly, HCFA has begun to respond by appointing a CIO and establishing
an investment review board. As envisioned by HCFA, the board will provide
an integrated process for strategic information technology planning,
budget development, and performance-based management and evaluation
of major information technology/system investments. HCFA also has
recently announced an agencywide reorganization that recognizes the
significant role that the CIO will have in agency management and planning.
The reorganization is planned for completion by July 1, 1997.

Although HCFA is progressing in its overall investment management
approach, it still lacks an MTS investment management strategy. For
example, HCFA established its most senior MTS decision-making body, its
MTS Initiatives Management Board, on December 14, 1994. The Board
comprises several senior program and information managers, and reports
directly to the HCFA Administrator. It is responsible for reviewing the
progress of MTS and keeping decisions on schedule, as well as reviewing
MTS planning to ensure that all future needs are met and its strategic vision
maintained. Members told us that the Board (1) acts as the front line MTS
decisionmaker, (2) provides leadership for and facilitates MTS project
decisions, and (3) provides overall strategies and planning for MTS. The
Board meetings have recently been expanded to include joint sessions
with a core group of technical advisers. The decision-making and project
management responsibilities that existed with the original Board remain
unchanged with the combined group.

The Board has not, however, been systematically involved with MTS. For
example, between 1995 when it was established and January 1997, the
Board was responsible for only one of 34 major MTS project decisions, that
being the decision to locate one print-mail facility at each MTS claims
processing site. Other critical decisions, such as the selection of the single
part A claims processing system or the implementation of a new MTS



9
  Evaluating Information Technology Investments: A Practical Guide, Office of Management and
Budget, November 1995.



Page 37                                         GAO/AIMD-97-78 Medicare Transaction System
                        Chapter 3
                        MTS Is Not Being Managed as an Investment




                        transition strategy, were made by individual managers or executives, or
                        lower-level MTS groups.


MTS Strategic Plan      When organizations manage information systems projects as investments,
Developed, but Not      they view projects as efforts to improve mission performance, not simply
Adequately Linked to    as actions to implement information technology. Since its passage in 1980,
                        the Paperwork Reduction Act has required agencies to ensure that
Agency Mission, Goals   information technology is acquired and used in a manner that improves
                        service delivery and program management, and increases productivity. In
                        its 1995 revision, the act explicitly required agencies to (1) ensure that
                        information resources management operations and decisions are
                        integrated with organizational planning, budget, financial management,
                        human resources management, and program decisions, and (2) establish
                        goals for improving information resources management’s contribution to
                        program productivity, efficiency, and effectiveness, and methods for
                        measuring progress toward achieving those goals. More recent legislation
                        supports this requirement and mandates that performance measures be
                        established to gauge how well information technology supports agency
                        programs.

                        In response to these requirements, in February 1994, HCFA developed an
                        agency strategic plan, which includes its overall agency mission, goals to
                        support that mission, and objectives to achieve those goals. Annually, the
                        agency also develops 5-year Information Resources Management (IRM)
                        plans that document its IRM goals, accomplishments, and major
                        information technology initiatives. HCFA’s September 1996 IRM plan
                        (1) provides a mechanism for incorporating its IRM planning process with
                        its budget process and its strategic plan, (2) graphically links the agency’s
                        information technology initiatives and each of the strategic goals, and
                        (3) lists each of its strategic goals, along with accomplishments that HCFA
                        believes have moved it along toward achieving each of those goals.

                        Although HCFA’s IRM plan provides a mechanism for integrating its IRM
                        planning process with its agencywide strategic planning, as mandated by
                        the Paperwork Reduction Act of 1995, it does not adequately apply this
                        integrated approach in its information technology planning. MTS and other
                        information technology projects are still ranked on the basis of budget
                        priorities determined by a HCFA budget review group rather than on how
                        well the systems will help fulfill HCFA’s mission and meet its program
                        needs. Using this budget approach, without cost-benefit analyses and
                        ranked key investment technology projects as required by legislation and



                        Page 38                                     GAO/AIMD-97-78 Medicare Transaction System
                  Chapter 3
                  MTS Is Not Being Managed as an Investment




                  OMB, HCFA has no assurance that it is funding the most important
                  information technology projects that are necessary to meet its mission and
                  goals.

                  Further, while MTS is HCFA’s major information technology project, it is not
                  highlighted in HCFA’s IRM plan. This plan includes a series of graphics
                  linking numerous information technology initiatives, including MTS, with
                  the seven strategic goals that they support. The plan also lists each of the
                  agency’s strategic goals, along with accomplishments that HCFA believes
                  has moved it closer to achieving those goals. However, MTS has been linked
                  to only one of the seven strategic goals, which is to “create excellence in
                  the design and administration of HCFA’s programs.” According to the plan,
                  other important goals, such as “be a leader in health care information
                  resources management” or “provide leadership in the continuing evolution
                  of the health care system” are not being addressed by MTS. Further, the
                  plan shows that MTS supports only 1 of the 30 strategies that HCFA has
                  developed to achieve its goal of creating excellence in HCFA’s programs.
                  According to HCFA officials, they are revising their strategic plan and are
                  arraying their projects in support of all strategic goals rather than tying
                  them to individual goals.


                  HCFA’s MTS  initiative has been under development for over 3 years,
Conclusions       however, it still has not been adequately cost-justified, as required by
                  legislation and OMB directives. Consequently, by moving forward with the
                  MTS development contract and planning to award contracts for MTS
                  operating sites without preparing required analyses, HCFA continues to put
                  at risk the opportunity for the most cost-effective and beneficial Medicare
                  claims processing environment possible. Further, HCFA is limiting its MTS
                  investment management approach with its lack of consistent oversight
                  from its senior decision-making body, as required by statutes and OMB
                  directives. Until HCFA officials implement an investment management
                  approach and produce adequate analyses to justify the cost of MTS and its
                  related initiatives, HCFA has no assurance that the project will be
                  cost-effective, delivered within estimated time frames, or result in a more
                  efficient or effective Medicare claims processing environment.


                  We recommend that the Secretary of Health and Human Services better
Recommendations   ensure the success of MTS by




                  Page 39                                     GAO/AIMD-97-78 Medicare Transaction System
    Chapter 3
    MTS Is Not Being Managed as an Investment




•   withholding funding for the MTS operating site contracts until an approach
    has been selected on the basis of an alternatives analysis; alternatives are
    ranked on the basis of cost, benefit, performance, risk, and technical
    factors, and are justified with valid cost-benefit analyses; and supported
    with a thorough risk assessment, which includes a realistic workload
    analysis, formal capacity analysis, and a sound security risk analysis;
•   requiring the Administrator of the Health Care Financing Administration to
    justify the continuation of MTS by producing a valid cost-benefit and
    alternatives analysis that includes goals for reaching programmatic savings
    and links estimated savings to specific improvements in Medicare claims
    processing, and take appropriate action based on the results of the
    analysis; and
•   requiring the Administrator of the Health Care Financing Administration to
    establish an investment management approach for MTS by explicitly linking
    the roles and responsibilities of the CIO and the Investment Review Board
    to relevant legislative mandates and requirements, which include
    (1) designing and implementing a process for maximizing the value and
    assessing and managing the risks of information technology acquisitions,
    and integrating that process with the budget, financial and program
    management decisions of the agency, (2) utilizing specific quantitative and
    qualitative criteria for comparing and prioritizing alternative information
    systems investment projects, (3) providing the means for senior
    management to obtain timely information regarding the progress of an
    investment, including a system of milestones for measuring progress, in
    terms of cost, capability of the system to meet specified requirements,
    timeliness, and quality, and (4) ensuring that performance measures are
    applied to measure how well the information technology supports the
    goals and missions of the agency.

    Completing these actions by the end of 1997 is essential so that the
    Congress, in monitoring HCFA’s progress, has assurance that HCFA is
    pursuing a course that will efficiently fulfill its goals and missions.

    We recommend that the Secretary of Health and Human Services assist
    HCFA in its modernization effort by providing oversight in accordance with
    provisions in the Clinger-Cohen, Paperwork Reduction, and Federal
    Acquisition and Streamlining Acts. This should include requiring the
    Department’s Chief Information Officer to (1) review the MTS project at
    predetermined project milestones to measure its progress in terms of cost,
    capability of meeting specified requirements, timeliness and quality, and
    (2) identify suitable actions to be taken, including termination, if the




    Page 40                                     GAO/AIMD-97-78 Medicare Transaction System
                     Chapter 3
                     MTS Is Not Being Managed as an Investment




                     project falls significantly behind schedule, over budget, or is not in
                     compliance with performance or capability requirements.

                     We also recommend that the Director of the Office of Management and
                     Budget utilize the enforcement authority provided by section 5113
                     (b)(5) of the Clinger-Cohen Act to ensure that the Health Care Finance
                     Administration complies with the act’s provisions, including the
                     requirement to justify major information technology projects such as MTS
                     with sound cost-benefit and alternatives analyses.


                     HHS stated that improvement is needed in its MTS investment management
Agency Comments      activities, and essentially concurred with our recommendations. HHS also
and Our Evaluation   stated that both the Department and HCFA agree that no funds will be
                     obligated for the MTS operating site procurements until a reassessment of
                     the project has been completed and a revised development strategy is
                     established. Further, HCFA stated that (1) it will continue to analyze the
                     cost and savings of MTS development strategies and evaluate alternatives
                     and (2) it concurs with our recommendation to link the roles and
                     responsibilities of the CIO and Investment Review Board. However, HCFA
                     commented that the following clarifications of our analyses were
                     warranted. First, it said that we failed to recognize the positive efforts it
                     has made in managing MTS, such as preparing multiple investment models,
                     hiring consultants, and broadening the agency’s MTS management team.
                     Second, HCFA said that because the scope of the costs included in its 1992
                     estimate differ from those included in the 1996 estimate, it is misleading to
                     state that MTS costs have increased from $151 million to about $1 billion.
                     Moreover, HCFA noted that, while the amount of the estimated MTS
                     investment has increased, administrative savings have been significant in
                     every analysis performed. Third, HCFA stated that while available
                     commercial software has value, it does not provide the sophistication
                     necessary to detect and prevent a significant amount of abusive billing.
                     HCFA also said that it uses several types of commercial software to detect
                     inappropriate billing; and is using the Los Alamos National Laboratory to
                     identify prepayment techniques for detecting inappropriate billing.

                     Throughout the report, we have recognized the positive efforts that HCFA
                     has made in managing MTS. For example, we noted that HCFA had prepared
                     a series of cost models, hired consultants such as its IV&V contractor, and
                     responded to legislation and OMB regulations. Yet, although HCFA has
                     prepared a series of cost models, none of these models included all costs
                     or evaluations of available alternatives. Thus, a complete cost-benefit



                     Page 41                                     GAO/AIMD-97-78 Medicare Transaction System
Chapter 3
MTS Is Not Being Managed as an Investment




analysis that includes an assessment of viable alternatives has not been
performed.

Further, our statement that estimated MTS costs have increased from
$151 million to about $1 billion is accurate. Whether the scope of the
project has changed is immaterial to the fact this project has incurred a
substantial cost increase, which we described to emphasize that HHS and
HCFA need to manage this project as an investment. One concern we have
always had is that HCFA has not finalized the requirements for MTS. Until
these requirements are finalized, no reliable cost estimate can be prepared.
Further, although all of HCFA’s MTS cost analyses included estimates of
substantial administrative savings, these estimates were based on an
invalid assumption—that Medicare claims processing costs would
continue to increase anyway. Actual cost data have shown this to be
incorrect. In addition, the cost-benefit analyses do not consider other
alternatives for administrative savings, such as those offered by
consolidating existing processing sites. Accordingly, we believe that HCFA
needs to reassess the estimated administrative savings MTS will provide.

Finally, although commercially available software may not provide the
sophistication HCFA desires for MTS, we believe it has potential for HCFA’s
claims-processing systems. HCFA does not expect MTS to be fully
implemented for years. This commercial software for detecting fraud and
abusive billing, along with the consolidation of the existing claims
processing software and claims processing sites, offers opportunities for
substantial program and administrative cost savings in the interim.

OMB agreed with our recommendations for HCFA to manage MTS as an
investment, and for OMB to ensure that HCFA justifies its major information
technology projects such as MTS with sound cost-benefit and alternatives
analyses. OMB said it will request that HCFA develop benefit, risk, return on
investment, and alternatives design analyses that correspond to the
changes in the MTS software design and that will address methodological
concerns raised in our report. Such analyses should greatly assist
management in making sound investment management decisions.




Page 42                                     GAO/AIMD-97-78 Medicare Transaction System
Chapter 4

Significant Systems-Development Practices
Not Being Followed, Increasing MTS Risk

                      HCFA  has never before managed a systems development project the size
                      and complexity of MTS. Recognizing its lack of systems development and
                      risk management experience, HCFA has relied on contractors to (1) provide
                      the hardware, telecommunications, and software required to process
                      Medicare claims, (2) review and make recommendations on the efficiency
                      and effectiveness of the MTS program, including risk management,
                      (3) assist in developing an integrated program schedule and identifying the
                      related critical path and risks associated with it, and (4) assist in
                      identifying the scope and components of MTS integration, and the risks
                      associated with this integration.

                      HCFA’s ability to adequately monitor and oversee the work of its
                      contractors, however, remains a question, given its own inexperience.
                      Deficiencies in several critical systems-development practices provide
                      early signs of weakness in HCFA’s system acquisition management
                      capability and its contractors’ software development practices. Plans
                      essential to MTS’ success are either inadequate, incomplete, or are being
                      completed too late in the development cycle; the project schedule is
                      incomplete and contains risky overlap in development phases; HCFA’s
                      risk-management process is inadequate; and its oversight has not
                      prevented a risky software-development strategy.


                      A requirements management plan describes the process that will be used
Requirements          to define, validate, rank, and control systems requirements.1 This plan is
Management Plan       critical because requirements are the key element of any system design.
Completed Too Late    Because requirements provide the foundation for designing, developing,
                      testing, and implementing the system software, it is essential that they be
in Systems            defined and implemented early in the systems development life cycle. In
Development Process   addition, requirements must be clearly defined to avoid ambiguity,
                      overlap, and duplication, and should completely and logically describe all
                      features and functions needed in the planned system. Using an appropriate
                      requirements management plan to define and manage requirements
                      significantly reduces the risk that requirements defects will cause
                      technical problems such as unacceptable system response times and
                      inadequate software interfaces. It also reduces the likelihood of needing to



                      1
                       Requirements management plans are addressed in the Capability Maturity Model (CMM) Software
                      Version 1.1 Software Engineering Institute (Pittsburgh, Pennsylvania: February 1993); Software
                      Acquisition Capability Maturity Model (SA-CMM) Version 1.01, Software Engineering Institute
                      (Pittsburgh, Pennsylvania: December 1996); and Guidelines for Successful Acquisition and
                      Management of Software-Intensive Systems (Air Force Guidelines), Department of the Air Force,
                      Software Technology Support Center, Volume 1, (February 1995).



                      Page 43                                         GAO/AIMD-97-78 Medicare Transaction System
Chapter 4
Significant Systems-Development Practices
Not Being Followed, Increasing MTS Risk




make time-consuming requirements changes later in the development
when they are more costly and risky to implement.

Although HCFA officials stated that they are following an interim
requirements management process, they have not yet made final or
implemented an official requirements management plan. Instead, they
have been relying on a draft MTS Business Requirements Writer’s Guide for
developing and managing MTS business requirements. The requirements
management process discussed in the guide does not describe how
requirements are to be assessed to determine their impact on the overall
project.

HCFA’s lack of a requirements management plan contributed to several
redirections that caused schedule delays. First, HCFA has twice redirected
the requirements definition approach and, 3 years into the MTS contract,
requirements have yet to be completely defined or approved. Second,
although requirements for the managed care module—the first MTS
release—were expected to be completed and documented in a systems
requirements document by November 1996, they have not yet been
officially approved by HCFA, nor have they been baselined.2 Despite the
lack of approved requirements, the MTS contractor has progressed into the
development phase for the managed care module. Further, as shown in
figure 4.1, the system requirements have been volatile. During a recent
5-month period, the requirements for one software release dropped from
1,639 to 1,499, while the requirements for another release increased from
631 to 868. Such volatility will affect cost, schedule, defects, and overall
quality of the software.




2
 A baseline is a specification or product that has been formally reviewed and agreed upon, and
thereafter serves as the basis for further development. It should then only be changed through formal
change-control procedures.



Page 44                                           GAO/AIMD-97-78 Medicare Transaction System
                                                                              Chapter 4
                                                                              Significant Systems-Development Practices
                                                                              Not Being Followed, Increasing MTS Risk




Figure 4.1: Volatility of MTS Requirements


 Release 1 (Managed Care) requirements                                                                 Release 2 (Common Working File) Requirements [FEE FOR SERVICE]
 1,660                                                                                                 900

 1,640
                                                                                                       850
 1,620

 1,600                                                                                                 800

 1,580
                                                                                                       750
 1,560

 1,540                                                                                                 700

 1,520
                                                                                                       650
 1,500

 1,480                                                                                                 600
         9/96           10/96         11/96            12/96           1/97    2/97                          9/96         10/96         11/96           12/96         1/97        2/97
                                  Contractor's monthly status report                                                               Contractor's Monthly Status Report




 Release 3 (Financial Server) requirements [FEE FOR SERVICE]                                           Release 4 (Carrier Processing) requirements [FEE FOR SERVICE]
 845                                                                                                   880

 840                                                                                                   860

 835                                                                                                   840

 830                                                                                                   820

 825                                                                                                   800

 820                                                                                                   780

 815                                                                                                   760

 810                                                                                                   740
       9/96           10/96          11/96            12/96            1/97    2/97                          9/96         10/96          11/96          12/96              1/97   2/97
                                 Contractor's monthly status report                                                                 Contractor's monthly status report




 Release 5 (Intermediary Processing) requirements [FEE FOR SERVICE]                                    Release 6 (Encounter Data) requirements [FEE FOR SERVICE]
 860                                                                                                   900

 840
                                                                                                       850
 820
                                                                                                       800
 800

 780
                                                                                                                                  To Be Determined
                                                                                                       750

 760
                                                                                                       700
 740

 720                                                                                                   650

 700                                                                                                   600
       9/96           10/96          11/96              12/96          1/97     2/97
                                  Contractor's monthly status report                                                                  Contractor's monthly status report

                                                                                                       NOTE: Scale above [600-900] for purposes of illustration only; numbers of
                                                                                                             requirements for this release are not yet available.




                                                                              Page 45                                                   GAO/AIMD-97-78 Medicare Transaction System
                      Chapter 4
                      Significant Systems-Development Practices
                      Not Being Followed, Increasing MTS Risk




                      HCFA’s IV&V contractor has recommended that HCFA develop and implement
                      a requirements management plan, and has warned that without such a
                      plan, a high probability exists that a system will be delivered that does not
                      meet customer needs. In early January 1997, HCFA officials said that they
                      expect to complete a requirements management plan by January 31, 1997.
                      As of April 1997, no plan had yet been completed.


                      The software development plan is a key document that reflects the
Software              contractor’s overall approach to developing software and serves as a
Development Plan Is   benchmark for monitoring how well the contractor adheres to approved
Inadequate            procedures and activities.3

                      HCFA does not have an adequate integrated software development plan. It
                      did not require such a software development plan in its January 1994 MTS
                      contract, and did not specifically request one during its contract
                      renegotiation in May 1996. HCFA officials explained that as part of the
                      negotiation process, they provided the MTS contractor with a template
                      containing the essential requirements of a software development plan.
                      They also stated that while the negotiations document met HCFA’s overall
                      requirements, several elements of a software development plan are
                      contained in various contract deliverables. According to HCFA officials, the
                      need for a software development plan for MTS has been fulfilled. The IV&V
                      contractor, who, in May 1995, indicated that the lack of a software
                      development plan was an area of significant risk, stated that the
                      negotiation document received from the MTS development contractor
                      incorporates most of the requirements for such a plan. The IV&V contractor
                      added that the negotiation document, along with other internal MTS
                      contractor practices, are sufficient to satisfy the software development
                      plan requirement.

                      While the MTS negotiation document contains the technical approach for
                      developing MTS, it lacks critical components of a software development
                      plan, such as a description of the software development library standards
                      and metrics. A software development library is critical to the software
                      development effort because it provides two fundamental capabilities:
                      (1) storage of computer software in machine readable form for computer
                      operation and (2) storage of computer software documentation in
                      human-readable form. Software development metrics are measures used
                      to indicate progress or achievement. Without these elements, the orderly

                      3
                      Software development plans are addressed in the (1) Software Engineering Institute’s Capability
                      Maturity Model, Software and its Software Acquisition Capability Maturity Model, and (2) Air Force’s
                      Guidelines for Successful Acquisition and Management of Software-Intensive Systems.



                      Page 46                                           GAO/AIMD-97-78 Medicare Transaction System
                      Chapter 4
                      Significant Systems-Development Practices
                      Not Being Followed, Increasing MTS Risk




                      development and subsequent support of software cannot be supported,
                      and the quality of software cannot be measured respectively.

                      According to HCFA officials, other parts of the MTS software development
                      plan are contained in numerous documents. However, this makes it
                      difficult for effective use by the MTS contractor and for HCFA to oversee and
                      manage the software development process. For example, HCFA provided
                      nine pages of references to various documents that it considered
                      contained components of a software development plan. Such document
                      dispersal precludes effectively managing software development. Sound
                      software development practices require the entire plan to be reviewed and
                      approved by all user groups, including those responsible for
                      documentation, testing, training, installation, and quality assurance.
                      Further, at each phase of review the plan is to be updated. These practices
                      are important to obtaining user agreement on procedures and to holding
                      individuals accountable for the delivered software products. Not having
                      these documents integrated and formally agreed to by all parties
                      responsible for software development increases the risk that they will not
                      be adequately reviewed.


                      A configuration management plan describes how changes to software and
Configuration         the total system including key documents will be managed and controlled
Management Plan Not   throughout the software development process.4 This process facilitates
Yet Implemented       communication among development team members regarding the status of
                      software engineering efforts, and ensures that proposed changes to
                      software or other system components, and related documents, are
                      reviewed by a configuration control board to determine whether the
                      changes should be approved. While system development contractors are
                      responsible for developing configuration management plans, agencies
                      acquiring information systems—such as HCFA—also need such a plan to
                      control all activities associated with the software development initiative.
                      The configuration management plan should be completed and used for
                      system development.

                      HCFA  has not yet implemented a configuration management plan for MTS. It
                      relied on the MTS development contractor’s configuration management
                      plan to manage changes to MTS development, but this plan is limited to
                      software related products. Configuration management for other MTS
                      components, including those that will be part of the planned data

                      4
                      Configuration management plans are addressed in the (1) Software Engineering Institute’s Capability
                      Maturity Model, Software and its Software Acquisition Capability Maturity Model and (2) Air Force’s
                      Guidelines for Successful Acquisition and Management of Software-Intensive Systems.



                      Page 47                                          GAO/AIMD-97-78 Medicare Transaction System
                      Chapter 4
                      Significant Systems-Development Practices
                      Not Being Followed, Increasing MTS Risk




                      operations and analysis center and processing sites, has not yet been
                      addressed. In response to this weakness, HCFA’ s change management
                      development team developed a plan in February 1997 which documented
                      the configuration management process for the entire MTS initiative.
                      According to HCFA, it will use this process to manage and control changes
                      across all MTS products. However, the plan has not yet been implemented.

                      Without such a process, changes to items such as software requirements,
                      and key documents can not be effectively managed or controlled. For
                      example, MTS’ managed care requirements are being defined without a
                      change management process. As a result, HCFA may not know whether all
                      managed care requirements that the contractor is using to develop the first
                      module adequately represent and fulfill Medicare’s functions and MTS’
                      goals. Also, without a configuration management process, HCFA will be
                      unable to effectively communicate hardware changes to the planned
                      operating sites that affect other members of the MTS development
                      community.

                      HCFA’sconfiguration management plan outlines a process for documenting
                      and reporting all requests for changes to MTS configuration items and
                      provides change management support for related MTS documentation. HCFA
                      intends to integrate the plan with the MTS design contractor’s configuration
                      management plan.


                      A systems integration plan is developed and used to ensure that the
Systems Integration   hardware, software, and telecommunications standards are adhered to so
Plan Not Yet          that components of the system will interface seamlessly with each other
Developed             and with users.5 A well-defined systems integration plan identifies related
                      interfaces between hardware and software, and includes procedures for
                      managing and controlling these interfaces. These procedures should
                      include provisions for identifying the functional and physical
                      characteristics between the applicable software or hardware units and
                      ensuring that they are compatible. Control over the interface structure
                      helps management make cost-effective, functional allocations among
                      systems and can provide needed safeguards during systems integration
                      testing.

                      In 1995, the IV&V contractor cited that HCFA lacked a systems integration
                      plan. However, HCFA has not yet developed a systems integration plan to

                      5
                       Systems integration plans are addressed in the Systems Engineering Management Guide, Defense
                      Systems Management College, January 1990.



                      Page 48                                         GAO/AIMD-97-78 Medicare Transaction System
                             Chapter 4
                             Significant Systems-Development Practices
                             Not Being Followed, Increasing MTS Risk




                             help ensure that all required interfaces will be developed and
                             implemented. Without a systems integration plan, HCFA cannot ensure that
                             all of the legacy systems and MTS software and hardware components will
                             interface with each other as they should. In September 1996, HCFA tasked a
                             consultant with (1) defining the scope of the MTS systems integration effort
                             and identifying its key systems integration activities and tasks,
                             (2) identifying who will perform key systems integration activities, and
                             (3) developing an overall plan and schedule for systems integration
                             activities. On April 1, 1997, HCFA received a copy of the consultant’s report.
                             HCFA plans to use this report as a basis for developing a detailed system
                             integration plan for MTS. On April 22, 1997, at the conclusion of our review,
                             HCFA had not established a date for the final plan.



                             Leading software-related organizations enhance their software
HCFA Oversight of            development capabilities by applying modern software development
MTS Contractor’s             standards and assessing their software engineering processes. Many such
System Development           organizations improve their software development programs by using a
                             capability maturity model to assess their capability to produce high-quality
Is Risky                     software.6 These organizations also define and use software measures, or
                             metrics, to assess the quality of software development, and prepare
                             software development cost and schedule estimates, as part of the initial
                             planning process.


Software Capability          The MTS development contractor estimates that about 2 million lines of
Maturity Is Key to Success   software code will need to be designed and developed for MTS. Even
                             though the software was the crucial component of MTS, HCFA did not
                             require prospective contractors to provide the results of independent
                             assessments of their software capability maturity, as recommended by the
                             Software Engineering Institute and other organizations, including the
                             Department of Defense.7 Such assessments help agencies ensure that the
                             selected contractor has the key software development processes in place
                             to reduce risks.



                             6
                              The Software Engineering Institute has developed two models to assist organizations in assessing the
                             maturity of their software development processes. These models are the Capability Maturity Model for
                             Software, and the Software Acquisition Capability Maturity Model. The Institute provides leadership in
                             advancing the state of the practice of software engineering to improve the quality of systems that
                             depend on software.
                             7
                              Guidelines for Successful Acquisition and Management of Software- Intensive Systems: Weapon
                             Systems Command and Control Systems Management Information Systems, Department of the Air
                             Force, February 1995.



                             Page 49                                           GAO/AIMD-97-78 Medicare Transaction System
                           Chapter 4
                           Significant Systems-Development Practices
                           Not Being Followed, Increasing MTS Risk




                           According to the MTS development contractor, although its MTS software
                           development team had not yet been evaluated using the software
                           capability maturity model, its corporate goal is to have the entire
                           corporation attain a maturity level three in 3 years.8 This is but a goal; as
                           yet a plan to achieve it has not been developed. Further, the contractor
                           had not established a software improvement process.

                           In addition, the systems development methodology being used was
                           developed specifically for MTS and, as a result, the contractor’s MTS team
                           has no experience with this methodology. Further, it is still incomplete.
                           For example, a complete systems development methodology consists of a
                           series of steps and tasks that are used by developers to provide a
                           structured approach for systems development from systems planning and
                           design through implementation and support. The MTS contractor’s
                           methodology only addressed the design, development, and testing and
                           validation phases. It does not address other key phases, which consist of
                           the analysis and implementation phases.

                           An inadequate software development methodology greatly increases
                           development risk because this methodology is used to control the key
                           software development phases, such as planning, design, development,
                           testing, and implementation. In addition, the contractor is not addressing
                           all phases in the proper sequence. For example, it has already moved into
                           the development phase for the Managed Care module before HCFA has
                           approved these requirements.


MTS Contractor Not Using   Software development metrics are numerical measures used to predict a
Complete Software          dimension of software quality throughout the project. Early detection and
Development Metrics        avoidance of problems and control of software development projects are
                           possible through the collection, validation, and analysis of metrics. Useful
                           metrics include numbers of defects found at various stages of
                           development, costs to repair defects, and the extent of test coverage.
                           Metrics such as the number and frequency of errors associated with a
                           specific software module are used to analyze software quality. Such
                           analysis can identify questionable or unacceptable situations.

                           Despite the importance of software metrics, the MTS contractor has only
                           included metrics that measure how much of the planned activity has been
                           completed at each measurement point and metrics that help refine future

                           8
                            Level 3 is the “defined” level on a scale ranging from one to five, with five being the highest rating.
                           Level 3 means that the software process for both management and engineering activities is
                           documented, standardized, and integrated into an organization-wide software process.



                           Page 50                                              GAO/AIMD-97-78 Medicare Transaction System
                           Chapter 4
                           Significant Systems-Development Practices
                           Not Being Followed, Increasing MTS Risk




                           cost and schedule estimates. While these are essential for assessing the
                           contractor’s overall performance, they are incomplete because they do not
                           contain critical measures to determine the quality of the software being
                           developed, such as the number of defects identified and corrected per
                           software module.


Questionable Assumptions   Software estimating tools, used along with sound assumptions about a
Used to Develop Software   planned project, help developers make reasonable projections about how
Cost and Schedule          much a planned project will cost and how long it will take to develop. The
                           MTS software developer applied the widely used software life-cycle
Estimates                  management (SLIM) model for the MTS estimate.

                           Since HCFA began MTS in 1994, it has worked with the software developer in
                           making many software development cost and schedule projections. HCFA’s
                           original MTS contract called for the software to be developed by late 1996
                           for about $18 million and was supposed to have been completed by
                           October 1996. Now, following contract renegotiations, the software
                           development will cost over $90 million including award fees, and will not
                           be completely implemented until after 2000. During the renegotiations,
                           HCFA directed the MTS contractor to estimate the cost and schedule for
                           developing the software.

                           The processes that should be used for estimating software cost are
                           illustrated in figure 2. Planning and cost estimating should be one of the
                           first set of activities the project team responsible for producing software
                           performs. Sound planning helps ensure that resources will be used
                           effectively to support corporate strategic objectives. Good software
                           estimating provides the basis for cost trade-offs and resource allocation
                           decisions. Together, planning and software estimates set the stage for
                           project controls and for managing progress.




                           Page 51                                     GAO/AIMD-97-78 Medicare Transaction System
                                       Chapter 4
                                       Significant Systems-Development Practices
                                       Not Being Followed, Increasing MTS Risk




Figure 4.2: Software Cost Estimating
Process                                  Model inputs                                                        Model outputs

                                         Cost factors                                                        Estimated schedule
                                         Requirements, size,
                                         and complexity                                                      Resource loading

                                                                                                             Estimated cost
                                         Constraints                         Software Cost
                                         Schedule, financial                                                 Confidence factors
                                                                            Estimation Model
                                                                                                             Tentative work
                                                                                                             breakdown structure

                                         Other Inputs                                                        Contingency
                                         Risk factors (skills,
                                         development process)                                                Expected defect rates

                                                                       Historical project database
                                                                       containing statistics such as size,
                                                                       complexity, and application domain




                                       The MTS software developer, with direction from HCFA, applied to the SLIM
                                       model a series of assumptions and constraints on such factors as the
                                       desired completion date and staff resources and skills to calculate outputs,
                                       including cost, development schedule, and degree of risk. Based on this
                                       model, the MTS project will require over 1.7 million lines of code.

                                       However, several assumptions used as input for the SLIM model do not
                                       realistically portray the MTS software development environment. For
                                       instance, one assumption used in the model rated the team as “very
                                       experienced” with the software infrastructure. This was based on plans to
                                       use specific, contractor-owned, proprietary software. Since these
                                       estimates were made, however, the software developer has decided to use
                                       a commercial off-the-shelf product. The “very experienced” indicator has
                                       not been adjusted to reflect that the software development team had never
                                       before used this commercial product. In another assumption, the software
                                       developer characterized the volume of data to be processed as “average”
                                       when, in fact, MTS will be required to process enormous amounts of data.
                                       Furthermore, requirements have been described as being “defined.” Yet,
                                       HCFA has not approved the software requirements specifications or
                                       documentation for any of the software releases. In addition, the
                                       specifications and requirements for the infrastructure needed to support
                                       the six releases have likewise not yet been approved.




                                       Page 52                                       GAO/AIMD-97-78 Medicare Transaction System
               Chapter 4
               Significant Systems-Development Practices
               Not Being Followed, Increasing MTS Risk




               These types of assumptions cause the SLIM model to inflate a key
               estimation variable—the productivity index—above the rating it would
               have received if more accurate assumptions had been used. This results in
               unrealistic software cost and schedule estimates. According to the SLIM
               vendor, for every point the productivity index drops or rises, a project will
               take 10 percent more or less time, and cost 30 percent more or less to
               develop than other typical, large-scale development projects.

               According to the software developer, the estimated number of lines of
               code will be reviewed during the design phase on the basis of further
               knowledge about software component size and complexity, better
               estimates of anticipated code reuse, and commercial off-the-shelf software
               use; this in turn will result in cost and schedule revisions.


               Generally accepted systems development practices require project
MTS Schedule   managers to continually monitor a project’s schedule to ensure that its
Incomplete,    activities are completed as planned. In addition, OMB requires that agencies
Dangerously    establish appropriate controls to help ensure that information technology
               acquisitions remain on schedule. Further, to avoid technical problems,
Compressed     federal systems acquisition guidance calls for agencies to minimize
               overlap of systems development phases—analysis, design, software
               development, testing, conversions, and deployment.9 In other words, to
               minimize the risk of major system development rework, all activities that
               need to be completed in one phase should be completed before the next
               system development phase begins.

               HCFA  has integrated its program schedule with that of its development
               contractor, and has added broadly defined transition tasks. However, it
               has not yet identified (1) how the schedule’s tasks interrelate or (2) a
               critical path showing the sequence of tasks that is longer than any other.
               Such critical paths are used to determine the overall project completion
               date. Also, HCFA has not developed resource estimates for each task,
               without which realistic projections about the time to complete each task
               or the entire project cannot be made. Further, detailed plans for several
               major MTS initiatives are still not included in the current program schedule,
               such as the transition, year-2000 effort, and the release of the final
               software module. Finally, key critical tasks are inappropriately scheduled.
               For example, according to the program schedule, the first MTS software



               9
                Guidelines for Successful Acquisition and Management of Software Intensive Systems, Department of
               the Air Force, Software Technology Support Center, February 1995.



               Page 53                                         GAO/AIMD-97-78 Medicare Transaction System
Chapter 4
Significant Systems-Development Practices
Not Being Followed, Increasing MTS Risk




release will be installed at the planned MTS processing centers before the
new processing centers are scheduled for implementation.

HCFA  has hired a contractor to assist in developing these missing program
schedule elements. By mid-summer 1997, HCFA plans to have
(1) project-level resource requirements for each of the approximately 30
projects that constitute the MTS initiative and (2) a program-level critical
path.

Finally, MTS project phases still overlap considerably for both the overall
MTS project as well as the development phases within each MTS module
release. As we reported in 1994, if a contractor advances too far into a
succeeding systems development phase before sufficient progress has
been made in previous phases, the risk of technical problems is
significantly increased.10 HCFA’s current program schedule shows
concurrency in all overall project phases and, between September 1997
and September 1998, HCFA plans to perform all five systems development
phases at one time. (See figure 4.3.) In addition, each one of the five
scheduled MTS software releases contains project phases that overlap.
Figure 4.4 shows the overlapping phases scheduled for the first
release—Managed Care.




10
  GAO/HEHS/AIMD-94-79, Jan. 25, 1994.



Page 54                                     GAO/AIMD-97-78 Medicare Transaction System
                                                          Chapter 4
                                                          Significant Systems-Development Practices
                                                          Not Being Followed, Increasing MTS Risk




Figure 4.3: Medicare Transaction System Program Schedule Revisions

Analysis phase
              3/94    7/94
              3/94                                              9/96
 6/93                                                                                              9/98


Design phase
                      7/94   3/95
                      7/94                                      10/96
 6/93                                                                                              10/98


                                    Development phase
                                    3/95 10/95
                                                         2/96
                                                                        10/96                                                1/00


                                                 Testing and validation phase
                                                 10/95             12/96
                                                         3/96
                                                                                          9/97                                 2/00

                                                                           Implementation phase
                                                                           12/96                          12/98
                                                                                          9/97                        9/99
                                                                        10/96                                                         5/00



1993           1994                   1995                 1996                    1997           1998               1999              2000


                                                          Original MTS Schedule - 4/94

                                                          MTS Program Schedule - 11/95

                                                          MTS Program Schedule - 1/97




                                                          Page 55                                             GAO/AIMD-97-78 Medicare Transaction System
                                               Chapter 4
                                               Significant Systems-Development Practices
                                               Not Being Followed, Increasing MTS Risk




Figure 4.4: Medicare Transaction System Program Schedule for Managed Care—Release 1

                 Analysis phase
                 7/96           11/96

            Design phase
             6/96               1/97


                           Development phase
                         10/96                                    1/98


                                                         Testing and validation phase
                                                         9/97            3/98


                        Implementation phase
                        10/96                                                                          2/99



          1996                             1997                                    1998                           1999




                                               HCFA’s IV&V contractor has also expressed concern about the MTS schedule.
                                               In HCFA’s February 1997 risk tracking report, the contractor stated that the
                                               software development schedule was risky because it “did not allow any
                                               slack time to accommodate slippage or partial performance.” The
                                               contractor explained that the schedule has never contained sufficient time
                                               to completely perform tasks leading to key deliverables such as the
                                               systems requirements document and systems external specifications.11
                                               The IV&V contractor said that because the MTS development contractor was
                                               1 month late in delivering these two products, additional resources would
                                               have to be taken from other tasks to complete this work. The IV&V
                                               contractor concluded that “This, in turn, will have a direct negative impact
                                               on the tasks being performed with reduced resources.”




                                               11
                                                A systems requirements document specifies the requirements that are to be included in a planned
                                               software system. System external specifications describe interfaces to the planned system, and include
                                               descriptions of subsystem and related software programs.



                                               Page 56                                           GAO/AIMD-97-78 Medicare Transaction System
                    Chapter 4
                    Significant Systems-Development Practices
                    Not Being Followed, Increasing MTS Risk




                    Federal statutes and OMB directives require effective risk management as
Weaknesses in MTS   an essential part of successful information technology system
Risk Management     development. OMB Memorandum 97-02 directed agencies to reduce risks in
Require Attention   major information systems investments by establishing clear measures and
                    accountability for project progress. The Paperwork Reduction Act of 1995
Before Proceeding   and the Clinger-Cohen Act of 1996 require agencies to manage risks
With Development    associated with information technology investments. Further, an
                    investment guide signed jointly by us and OMB recommends the use of key
                    investment control techniques, including risk assessments, to expose
                    potential technical and managerial weaknesses that could impair project
                    success.12

                    In addition to these criteria, guidance developed by federal agencies,
                    including the Department of Defense, suggests other key elements that
                    should be part of an effective risk management program. These include
                    quantitatively estimating risk impact, developing success criteria and
                    measures when the risk can be considered mitigated, assigning a full-time
                    risk management officer, and contracting for IV&V services.

                    Measuring HCFA and MTS against these criteria reveals a disappointing
                    picture. First, HCFA’s risk mitigation plans contain no established time
                    frame for assessing risk status and do not specify target dates for risk
                    mitigation. Second, metrics have not been developed to provide HCFA with
                    the means for comparing progress in assessing the effectiveness of risk
                    mitigation efforts. Third, the risk management process has no mechanism
                    for providing management with early warnings of risks becoming
                    imminent. Fourth, resource estimates of staff, schedule needs, and funding
                    to address risks have not been identified. Fifth, the MTS risk management
                    database does not incorporate all identified risks. Finally, documents do
                    not identify interdependencies among risks.

                    Overall responsibility for risk management has not been formally assigned.
                    The chapter on risk management in HCFA’s program management plan
                    assigns broad responsibility for risk management to the MTS Initiatives
                    Management Group, Office Leads, and Project Owners.13 The MTS project
                    manager serves as the de facto risk management official. The recent
                    assignment of a team responsible for risk management oversight under

                    12
                       Evaluating Information Technology Investments, OMB Office of Information and Regulatory Affairs,
                    Information Policy and Technology Branch, November 1995.
                    13
                     The MTS Management Group is a core advisory group that shares definition, development and
                    implementation responsibilities for MTS. The MTS Office Leads is composed of representatives of
                    HCFA bureaus, who meet regularly to discuss MTS integration, coordination, program monitoring, and
                    communications issues. The Project Owners are responsible for discrete MTS project segments.



                    Page 57                                          GAO/AIMD-97-78 Medicare Transaction System
                                      Chapter 4
                                      Significant Systems-Development Practices
                                      Not Being Followed, Increasing MTS Risk




                                      this official provides support for risk management activities; however,
                                      current risk management policies and procedures remain unchanged.

                                      In February 1997, HCFA formed a team responsible for risk oversight and
                                      management; this team reports to the MTS project manager. The team’s
                                      responsibilities include incorporating all identified risks into the MTS risk
                                      database; helping to identify additional risks; updating the risk
                                      management section of the program management plan; and helping to
                                      monitor, and mitigate reported risks to the point where they are removed
                                      from the risk management report.

                                      Long-standing unmitigated risks indicate that effective risk management
                                      practices have not been institutionalized and uniformly applied. Table 4.1
                                      describes critical risks for which the cost, schedule, and performance
                                      impact has not yet been adequately quantified.

Table 4.1: Critical Unmitigated MTS
Risks                                                        Date/by whom
                                      Risk                   identified            Description            Impact
                                      Lack of a software     May 1995 by IV&V      A software             HCFA will be unable
                                      development plana      contractor            development plan       to assess MTS
                                                                                   describing the         software
                                                                                   methodology and        development. It will
                                                                                   approach to            be difficult for HCFA
                                                                                   developing and         to move to a new
                                                                                   testing MTS software   MTS software
                                                                                   has not been           development
                                                                                   created.               contractor effectively
                                                                                                          and cost-efficiently.
                                      Lack of a              November 1994 by      A requirements         Difficulty in tracing
                                      requirements           IV&V contractor       management plan is     requirements to MTS
                                      management plan                              needed to control      or Medicare
                                                                                   new and changing       functions. Managing
                                                                                   requirements.          changing
                                                                                                          requirements may
                                                                                                          result in a system
                                                                                                          that does not meet
                                                                                                          HCFA’s needs.
                                      Lack of a system       June 1995 by IV&V     A well-defined         HCFA cannot ensure
                                      integration plan       contractor            process is needed      that systems are
                                                                                   to describe how the    interfacing
                                                                                   various systems        appropriately and
                                                                                   supporting MTS will    producing the
                                                                                   be integrated.         correct results.
                                                                                                                    (continued)




                                      Page 58                                     GAO/AIMD-97-78 Medicare Transaction System
              Chapter 4
              Significant Systems-Development Practices
              Not Being Followed, Increasing MTS Risk




                                        Date/by whom
              Risk                      identified               Description               Impact
              Lack of a                 May 1995 by IV&V         HCFA has not              HCFA cannot ensure
              configuration             contractor               developed a               that the integrity of
              management plan                                    configuration             MTS products is
                                                                 management plan to        maintained and that
                                                                 manage and control        only approved
                                                                 changes to MTS            changes are being
                                                                 products such as          made to MTS
                                                                 requirements,             throughout the
                                                                 software, and other       system development
                                                                 contract deliverables.    life cycle.
              Compressed MTS            June - December          The MTS software          Any delay in a task
              development               1996 by IV&V and         development               on the critical path
              schedule                  MTS software             schedule does not         will result in a delay
                                        development              provide time for any      in the overall
                                        contractors              delays.                   schedule. Therefore,
                                                                                           the expected MTS
                                                                                           completion date will
                                                                                           not be met, and
                                                                                           additional funds to
                                                                                           complete the project
                                                                                           may be required.
              Lack of Medicare          October 1996 by          The MTS software          Insufficient Medicare
              subject-matter            MTS software             development               expertise can delay
              experts                   development              contractor lacks          defining
                                        contractor               sufficient Medicare       requirements, and
                                                                 business analysts,        result in needing to
                                                                 necessary to              rework requirements.
                                                                 analyze and define
                                                                 MTS requirements
                                                                 for releases 2
                                                                 through 5.
              Incorrect metrics         October 1996 by          Software and              Incorrect metrics
                                        MTS software             financial metrics         may lead to negative
                                        development              either understate or      MTS cost, schedule,
                                        contractor               overstate MTS             and performance
                                                                 software quality and      trends and
                                                                 performance.              unacceptable
                                                                                           software.

              a
               Although this risk item was taken off the risk report by the IV&V contractor in November 1996, we
              believe that it should still be classified as a risk because it has not been mitigated.




              HCFA  has attempted to address systems requirements, schedule, and risk
Conclusions   management issues that we have been reporting since 1994, yet many
              critical problems remain inadequately addressed. Critical management
              plans have not been adequately developed, the contractor is using a risky
              software development strategy, the MTS schedule is incomplete and



              Page 59                                          GAO/AIMD-97-78 Medicare Transaction System
                      Chapter 4
                      Significant Systems-Development Practices
                      Not Being Followed, Increasing MTS Risk




                      dangerously compressed, HCFA is not using sound risk management
                      procedures, and its IV&V contractor is not ensuring that potential critical
                      risks are routinely assessed. Unless HCFA solves these problems before
                      proceeding further with MTS development or implementation, it risks
                      extensive development rework, substantial cost increases, and a system
                      that will not fully meet HCFA’s needs.


                      To better ensure the success of MTS, we recommend that the Secretary of
Recommendations       Health and Human Services require the Administrator, Health Care
                      Financing Administration, to direct and remain accountable for the
                      following actions before proceeding further with MTS development.

                  •   Complete a requirements management plan to assist in identifying,
                      approving, managing, and controlling the requirements development
                      process.
                  •   Support the development of MTS software with an integrated software
                      development plan, which includes a description of such critical elements
                      as software development library standards and metrics. All critical
                      elements should be in a single document to facilitate the review, approval,
                      and use by involved individuals.
                  •   Implement a configuration management process that includes change
                      controls for requirements as well as all other related MTS issues such as
                      hardware changes to the planned operation sites.
                  •   Complete a comprehensive systems integration plan to ensure that all
                      MTS-related interfaces are identified, developed, managed, and controlled.
                  •   Require an independent evaluation of the MTS contractor’s software
                      development capability prior to beginning the software development
                      phase. To ensure that the contractor’s MTS development team has the
                      capability required for reasonable assurance of success, it should achieve
                      a rating of at least level 2.
                  •   Improve software development oversight by requiring the MTS developer to
                      include measures of the quality of software in the software development
                      metrics.
                  •   Direct the MTS developer to rerun the SLIM model using appropriate
                      assumptions and constraints, and use the results in reassessing the cost
                      and time required to develop MTS.
                  •   Complete a new, integrated MTS program schedule that includes a critical
                      path for the entire initiative, including the interim Medicare processing
                      environment, and resources and costs for each task. The schedule should
                      also minimize overlap in the phases of the system development process.




                      Page 60                                     GAO/AIMD-97-78 Medicare Transaction System
                         Chapter 4
                         Significant Systems-Development Practices
                         Not Being Followed, Increasing MTS Risk




                     •   Mitigate critical risks by designating an accountable official for risk
                         management and ensuring that this individual implements a process,
                         which will (1) identify all significant risks, (2) quantify the impact of
                         identified risks, (3) establish time frames for assessing risk status and
                         specifying target dates for risk mitigation, (4) develop metrics that will
                         compare progress in assessing the effectiveness of risk mitigation efforts,
                         (5) provide a mechanism for alerting management early of risks that are
                         becoming imminent, (6) provide resource estimates of staff, schedule
                         needs, and funding to address identified risks, (7) ensure that the MTS risk
                         management database incorporates all identified risks, and (8) document
                         interdependencies among risks. Further, this accountable official should
                         (1) ensure that mitigation plans are developed to address identified risks,
                         (2) hold individuals in authority accountable for prompt completion and
                         implementation of risk mitigation plans, and (3) periodically evaluate the
                         adequacy of HCFA’s progress in mitigating risks and identify new risks.
                     •   Require the IV&V contractor to assist HCFA in mitigating risks by quantifying
                         the impacts of identified risks on program cost and schedule. HCFA should
                         also reflect these in its program status reports.

                         To help HCFA improve its ability to use effective systems development
                         practices and improve its software acquisition capability, we recommend
                         that the Secretary of Health and Human Services direct the Administrator,
                         Health Care Financing Administration, to (1) obtain an independent
                         assessment of its software acquisition capabilities using the Software
                         Engineering Institute’s software acquisition capability maturity model, and
                         implement improvements to correct any identified weaknesses, and
                         (2) report its findings to both HHS and OMB.


                         HHS  agreed with our recommendations in this chapter. It specifically
Agency Comments          concurred with our recommendations regarding the need for plans critical
and Our Evaluation       to effective systems development, a complete and integrated program
                         schedule, and a designated official accountable for risk management.
                         While HHS agreed that the SEI certification of software development
                         contractors at a level 2 has value, and plans to use this rating as one of its
                         selection criteria for future software development contractors, it said that
                         requiring the current software developer to achieve this rating would have
                         little if any value, and that using the level-2 rating as a minimum
                         qualification would limit the range of potential competitors.

                         We believe that such a rating is critical to HCFA’s assurance that its
                         software developers have the capability to successfully complete contract



                         Page 61                                     GAO/AIMD-97-78 Medicare Transaction System
Chapter 4
Significant Systems-Development Practices
Not Being Followed, Increasing MTS Risk




requirements. Further, we believe that limiting the choice of contractors to
those who have achieved a level-2 rating is not only appropriate, but
necessary to the successful development of MTS.

OMB agreed with our recommendations for HCFA to apply sound
systems-development practices to reduce risks and assist management in
controlling MTS. It also commented that it would request an evaluation of
the benefits of performing an independent assessment of HCFA’s software
acquisition capability and, if the benefits are confirmed, would incorporate
the results of this evaluation in its overall discussions with HCFA regarding
the next steps for MTS.

Based on the complexity of this project and the difficulties HCFA has had in
managing it, we are certain that HCFA will benefit from an independent
assessment of its software acquisition capabilities using SEI’s software
acquisition capability maturity model.




Page 62                                     GAO/AIMD-97-78 Medicare Transaction System
Page 63   GAO/AIMD-97-78 Medicare Transaction System
Appendix I

Comments From the Department of Health
and Human Services




             Page 64       GAO/AIMD-97-78 Medicare Transaction System
Appendix I
Comments From the Department of Health
and Human Services




Page 65                                  GAO/AIMD-97-78 Medicare Transaction System
Appendix I
Comments From the Department of Health
and Human Services




Page 66                                  GAO/AIMD-97-78 Medicare Transaction System
Appendix I
Comments From the Department of Health
and Human Services




Page 67                                  GAO/AIMD-97-78 Medicare Transaction System
Appendix I
Comments From the Department of Health
and Human Services




Page 68                                  GAO/AIMD-97-78 Medicare Transaction System
Appendix I
Comments From the Department of Health
and Human Services




Page 69                                  GAO/AIMD-97-78 Medicare Transaction System
Appendix II

Comments From the Office of Management
and Budget




              Page 70      GAO/AIMD-97-78 Medicare Transaction System
Appendix II
Comments From the Office of Management
and Budget




Page 71                                  GAO/AIMD-97-78 Medicare Transaction System
Appendix III

Key Provisions of Laws, Regulations, and
Best Practices Relating to Information
Technology Investments
                         The citations presented in this appendix include key laws, regulations, and
                         guidance pertaining to information technology investment issues. They are
                         organized on the basis of the major sections of this report.



Information
Technology Project
Transitions and
Year 2000

Information Technology   The Clinger-Cohen Act of 1996 (CCA), (formerly named the Information
Transition Environment   Technology Management Reform Act of 1996), P.L. 104-106, Division E;
                         February 10, 1996, effective August 8, 1996, 40 USC 1423(3): Agency heads
                         shall “ensure that performance measurements are prescribed for
                         information technology used by, or to be acquired for the executive
                         agency, and that the performance measurements measure how well the
                         information technology supports programs of the executive agency.”

                         CCA, 40 USC 1425(c)(2): The agency CIO shall “monitor the performance of
                         information technology programs of the agency, evaluate the performance
                         of those programs on the basis of the applicable performance
                         measurements, and advise the head of the agency regarding whether to
                         continue, modify, or terminate a program or project.”

                         The Federal Property and Administrative Services Act of 1949, § 313(b), as
                         added by the Federal Acquisition And Streamlining Act of 1994 (FASA), P.L.
                         103-355; October 13, 1994, 41 USC 263(b): “The head of each executive
                         agency shall approve or define the cost, performance, and schedule goals
                         for major acquisition programs of the agency.”

                         The Paperwork Reduction Act of 1995 (PRA) as amended (P.L. 104-13;
                         May 22, 1995), 44 USC 3506(b)(3)(C): Requires agencies “to develop and
                         maintain an ongoing process . . . to establish goals for improving
                         information resources management’s contribution to program
                         productivity, efficiency, and effectiveness; methods for measuring
                         progress toward those goals, and clear roles and responsibilities for
                         achieving those goals.”




                         Page 72                              GAO/AIMD-97-78 Medicare Transaction System
                         Appendix III
                         Key Provisions of Laws, Regulations, and
                         Best Practices Relating to Information
                         Technology Investments




                         Office of Management and Budget (OMB) Circular No. A-130, “Management
                         of Federal Information Resources,” February 8, 1996, 8b(3)(f): “Agencies
                         shall establish information systems management oversight mechanisms . . .
                         that ensure that major information systems proceed in a timely fashion
                         towards agreed-upon milestones in an information system life cycle, meet
                         user requirements, and deliver intended benefits to the agency and
                         affected publics through coordinated decision making about the
                         information, human, financial, and other supporting resources.”

                         OMB  Memorandum M-97-02, Funding Information Systems Investments,
                         October 25, 1996: Investments in major information systems proposed for
                         funding in the President’s budget should “be implemented in phased,
                         successive chunks as narrow in scope and brief in duration as practicable,
                         each of which solves a specific part of an overall mission problem and
                         delivers a measurable net benefit independent of future chunks.”

                         OMB, Evaluating Information Technology Investments: A Practical Guide,
                         version 1.0, Office of Information and Regulatory Affairs, Information
                         Policy and Technology Branch, November 1995: “Senior managers need to
                         compare the preliminary results being achieved by a project against its
                         projected costs, benefits, and risks, and to identify actual or potential
                         managerial, organizational, or technical problems.”

                         General Accounting Office (GAO), Systems Assessment Framework: A
                         Guide for Reviewing Information Management and Technology Issues in
                         the Federal Government (SAF), version 1.0, August 1996: During the design,
                         development, and deployment of systems, agencies are to prepare
                         products and documents, including (1) a completed work plan with human
                         resources, scheduling, and funding for each step, (2) a transition plan,
                         including conversion from the legacy environment to the replacement
                         system, and (3) performance measures that link to the users’ operations.
                         Further, agency management responsibilities include (1) review and
                         approval of the system test and conversion plans, (2) formal verification
                         and validation of the developed system, (3) review and approval of the
                         transition plan, including procedures for site surveys, conversion
                         preparation, and contingencies, and (4) agreement that acceptance test
                         results meet management’s criteria.


Information Technology   GAO, Year 2000 Computing Crisis: An Assessment Guide, Exposure Draft,
Measures to Address      February, 1997: This guide presents a structured approach and a checklist
Year 2000                to aid federal agencies in planning, managing, and evaluating their year



                         Page 73                                    GAO/AIMD-97-78 Medicare Transaction System
                       Appendix III
                       Key Provisions of Laws, Regulations, and
                       Best Practices Relating to Information
                       Technology Investments




                       2000 programs. The most important year-2000 activities are (1) developing
                       an overall year-2000 plan, (2) identifying responsibilities for managing and
                       monitoring the year-2000 efforts, (3) preparing an assessment of the
                       severity and timing of potential year-2000 impact, (4) conducting an
                       inventory of the systems on which Medicare claims processing depend,
                       (5) prioritizing and scheduling activities to convert, replace, or eliminate
                       these systems, (6) developing validation strategies and test plans for
                       systems, (7) addressing interface and data exchange issues, and
                       (8) developing contingency plans for critical systems in the event of
                       failure.



Managing Information
Technology Projects
as Investments

Required Investment    CCA, 40 USC 1427: The agency head shall identify major information
Analyses               technology acquisition programs that have significantly deviated from the
                       cost, performance, or schedule goals established for the program in the
                       IRM plan required by the PRA.


                       FASA, 41 USC 263(a): “It is the policy of Congress that the head of each
                       executive agency should achieve, on average, 90 percent of the cost and
                       schedule goals established for major and nonmajor acquisition programs
                       of the agency without reducing the performance or capabilities of the
                       items being acquired.”

                       FASA, 41 USC 263(c): The agency head shall “(1) determine whether there is
                       a continuing need for programs that are significantly behind schedule,
                       over budget, or not in compliance with performance or capability
                       requirements; and (2) identify suitable actions to be taken, including
                       termination, with respect to such programs.”

                       OMB Bulletin No. 95-03, Planning and Budgeting for the Acquisition of
                       Fixed Assets, June 27, 1995 (superseded): “The planning for fixed asset
                       acquisitions should be based on a systematic analysis of expected benefits
                       and costs. The fundamental method of formal economic analysis is
                       benefit-cost analysis.”




                       Page 74                                    GAO/AIMD-97-78 Medicare Transaction System
                            Appendix III
                            Key Provisions of Laws, Regulations, and
                            Best Practices Relating to Information
                            Technology Investments




                            OMB Circular No. A-11, “Preparation and Submission of Budget Estimates,”
                            Part 3 Planning, Budgeting and Acquisition of Fixed Assets (July 1996);
                            Appendix 300A(b): “The planning for fixed asset acquisitions should be
                            based on a systematic analysis of expected benefits and costs. The
                            fundamental method of formal economic analysis is benefit-cost analysis.”

                            OMB  Circular No. A-94, “Guidelines and Discount Rates for Benefit-Cost
                            Analysis of Federal Programs,” October 29, 1992; (5)(c)(3): Benefit-cost
                            analyses should “consider alternative means of achieving program
                            objectives by examining different program scales, different methods of
                            provision, and different degrees of government involvement. For example,
                            in evaluating a decision to acquire a capital asset, the analysis should
                            generally consider: (i) doing nothing; (ii) direct purchase; (iii) upgrading,
                            renovating, sharing, or converting existing government property; or
                            (iv) leasing or contracting for services.”

                            OMB Circular No. A-130, 8b(1)(c): Agencies shall “conduct benefit-cost
                            analyses to support ongoing management oversight processes that
                            maximize return on investment and minimize financial and operational
                            risk for investments in major information systems on an agency-wide
                            basis.”

                            OMB  Memorandum M-97-02: “Investments in major information systems
                            proposed for funding in the President’s budget should . . . demonstrate a
                            projected return on the investment that is clearly equal to or better than
                            alternative uses of available public resources.”


Required Alternatives and   CCA, 40 USC 1422: Agency heads are to “design and implement in the
Risk Analyses               executive agency a process for maximizing the value and assessing and
                            managing the risks of the information technology acquisitions of the
                            executive agency.” The process is to (among other things) provide for the
                            selection of information technology investments using minimum criteria
                            on whether to undertake an investment (including quantitatively
                            expressed projected net, risk-adjusted return on investment, and specific
                            quantitative and qualitative criteria for comparing and ranking alternative
                            information systems investment projects) and to provide a means for
                            senior management to obtain timely information regarding progress (at
                            established milestones) in terms of cost, capability of the system to meet
                            specified requirements, timeliness, and quality.




                            Page 75                                    GAO/AIMD-97-78 Medicare Transaction System
                        Appendix III
                        Key Provisions of Laws, Regulations, and
                        Best Practices Relating to Information
                        Technology Investments




                        OMB Bulletin No. 95-03, Attachment A (superseded): “Alternative fixed
                        assets, information system designs, and other solutions to meet a mission
                        need should always be explored. Analyses of fixed asset acquisitions
                        should include a comprehensive set of options. If it is decided to acquire
                        the services of fixed assets, the decision whether to purchase or lease may
                        be analyzed by comparing the present value of expected life-cycle costs
                        over the period during which the services of the asset will be needed.”

                        OMB Circular No. A-94; (5)(b): “A program is cost-effective if, on the basis
                        of life cycle cost analysis of competing alternatives, it is determined to
                        have the lowest costs expressed in present value terms for a given amount
                        of benefits.” (5)(c)(3) “Analyses should also consider alternative means of
                        achieving program objectives by examining different program scales,
                        different methods of provision, and different degrees of Government
                        involvement. For example, in evaluating a decision to acquire a capital
                        asset, the analysis should generally consider: (i) doing nothing; (ii) direct
                        purchasing; (iii) upgrading, renovating, sharing, or converting existing
                        Government property; or (iv) leasing or contracting for services.”


Required Investment     CCA, 40 USC 1422(b)(2): The information technology investment process of
Management Strategies   executive agencies is to “be integrated with the processes for making
                        budget, financial, and program management decisions within the
                        executive agency.”

                        CCA, 40 USC 1423(3): “The head of an executive agency shall . . . ensure
                        that performance measurements are prescribed for information
                        technology used by or to be acquired for the executive agency and that the
                        performance measurements measure how well the information technology
                        supports programs of the executive agency.”

                        PRA,   44 USC 3506(b)(2): Each agency is to develop and maintain a strategic
                        IRM   plan on how IRM activities help accomplish agency missions.

                        PRA, 44 USC 3506(b)(3)(A): Requires agencies to “ensure that information
                        resources management operations and decisions are integrated with
                        organizational planning, budget, financial management, human resources
                        management and program decisions” 44 USC 3506(b)(3)(C), and to
                        “establish goals for improving information resources management’s
                        contribution to program productivity, efficiency, and effectiveness,
                        methods for measuring progress towards those goals, and clear roles and
                        responsibilities for achieving those goals.”



                        Page 76                                    GAO/AIMD-97-78 Medicare Transaction System
                          Appendix III
                          Key Provisions of Laws, Regulations, and
                          Best Practices Relating to Information
                          Technology Investments




                          OMB Circular No. A-130, 8b(2): “Agencies shall establish and maintain
                          strategic information resources management planning processes” that
                          include “[s]trategic IRM planning that addresses how the management of
                          information resources promotes the fulfillment of an agency’s mission.”

                          OMB’s  Practical Guide: Agency processes should include a disciplined and
                          structured management forum for making information technology
                          investment decisions, with the authority to approve, cancel, or delay
                          projects, mitigate risks, and validate expected returns. Also, the agency
                          should establish an executive management team that makes funding
                          decisions based upon comparisons and trade-offs among competing
                          project proposals, especially for those projects expected to have
                          agencywide impact. All management decisions should be documented
                          along with data supporting the required changes. Common problems and
                          their solutions, which are applicable to one information technology
                          project, should be evaluated as to how they apply to other information
                          technology projects under management’s purview.



Generally Accepted
Systems Development
Best Practices

Critical Systems          Department of the Air Force, Guidelines for Successful Acquisition and
Development Plans:        Management of Software-Intensive Systems (Air Force Guidelines)
Requirements Management   Volume 1, Version 1.1, February 1995: Requirements must constantly be
                          managed because they significantly affect total system development costs
                          and schedule. Management of requirements must stress a commitment to
                          an iterative process that utilizes structured requirements methods and
                          appropriate tracking and analysis tools. In addition, traceability from
                          original, identified needs to their derived requirements, designs, and
                          implementation must be assured.

                          Software Engineering Institute, Capability Maturity Model Software
                          (CMM),Version 1.1, February 1993: Requirements management includes
                          (1) managing and documenting the system requirements and their
                          allocation throughout the project’s life, (2) providing adequate resources
                          and funding for managing the allocated requirements, and (3) ensuring
                          that members of the software engineering group and other




                          Page 77                                    GAO/AIMD-97-78 Medicare Transaction System
                       Appendix III
                       Key Provisions of Laws, Regulations, and
                       Best Practices Relating to Information
                       Technology Investments




                       software-related groups are trained to perform their requirements
                       management activities.

                       Software Engineering Institute, Software Acquisition Capability Maturity
                       Model (SA-CMM), Version 1.01, December 1996: Requirements management
                       ensures that software requirements are unambiguous, traceable, testable,
                       documented, and controlled. A plan describing requirements management
                       should be developed prior to contractual actions and should cover:
                       (1) objectives of the project team’s requirements development and
                       management activities, (2) activities to be performed, including
                       requirements identification, (3) identification of the groups, and
                       intergroup coordination, associated with requirements development and
                       management activities, (4) the extent of end-user involvement in the
                       acquisition, (5) procedures for requirements development, including
                       planning, identification, analysis, and verification, (6) procedures for
                       requirements management, including baseline establishment, change
                       control, and status reporting, and (7) procedures for impact analysis of
                       changes to requirements or introduction of new requirements, including
                       performance, cost, and schedule. The plan should also describe a
                       mechanism for tracing requirements during software development to
                       ensure that requirements have been included in the implemented work
                       products and services.


Critical Systems       Air Force Guidelines: The software development plan is the key software
Development Plans:     document reflecting the contractor’s overall software development
Software Development   approach. It includes resources, organization, schedules, risk
                       identification and management, data rights, metrics, quality assurance,
                       control of nondeliverable computer resources and identification of
                       commercial off-the-shelf systems, reuse, and government-furnished
                       software that the contractor intends to use.

                       CMM: The software development plan provides the basis for performing and
                       managing the software project’s activities and addresses the commitments
                       to the software project’s customer according to the resources, constraints,
                       and capabilities of the software project. The software development plan
                       includes (1) the software project’s purpose, scope, goals, and objectives,
                       (2) selection of a software life cycle, (3) identification of the selected
                       procedures, methods, and standards for developing and/or maintaining the
                       software, (4) identification of software work products to be developed,
                       (5) size estimates of the software work products and changes to the
                       software work products, (6) estimates of the software project’s effort and



                       Page 78                                    GAO/AIMD-97-78 Medicare Transaction System
                           Appendix III
                           Key Provisions of Laws, Regulations, and
                           Best Practices Relating to Information
                           Technology Investments




                           costs, (7) estimated use of critical computer resources, (8) the software
                           project’s schedules, including identification of milestones and reviews,
                           (9) identification and assessment of the project’s software risks, and
                           (10) the project’s software engineering facilities and support tools.

                           SA-CMM:The project team should track the contractor’s development of the
                           software engineering environment required to support the software.


Critical Systems           Air Force Guidelines: Requests for proposals should require offerers to
Development Plans:         provide a configuration management plan that addresses change control
Configuration Management   throughout the development process, the offerer’s configuration
                           management organization, the tools to be used, configuration management
                           personnel experience, and a description of the offerer’s configuration
                           management training program. The government should also have a
                           configuration management plan to control changes to functional and
                           performance requirements. This plan is needed when the software and its
                           documentation are released to the government.

                           CMM: The purpose of software configuration management is to establish
                           and maintain the integrity of the products of the software project
                           throughout the project’s software life cycle. This practice includes the
                           development of a configuration management plan in the early stages of
                           overall project planning, which is used as the basis for performing
                           configuration management activities, including establishing a
                           configuration management library system as a repository for the software
                           baselines, and identifying the software products to be placed under
                           configuration management.

                           SA-CMM: The customer’s project team should develop and implement the
                           plans for moving and supporting the acquired software products. One goal
                           for moving software from the contractor to the customer is maintaining
                           configuration management throughout the transition. Software acquisition
                           planning documents such as a configuration management plan should be
                           developed prior to contractual actions.


Critical Systems           Defense Systems Management College: Systems Engineering Management
Development Plans:         Guide, January 1990: A primary role of systems engineering is to ensure
Systems Integration        that the many diverse elements constituting a system are compatible and
                           ready when needed. This avoids the situation in which hardware or
                           software, when integrated into the system, fails to function as intended as



                           Page 79                                    GAO/AIMD-97-78 Medicare Transaction System
                           Appendix III
                           Key Provisions of Laws, Regulations, and
                           Best Practices Relating to Information
                           Technology Investments




                           part of the system. Integration ensures that all pieces of the system will
                           work together to realize system goals.


Contractor’s Software      Air Force Guidelines: Prior to or during source selection, contracting
Development Strategy and   agencies should evaluate offerers’ software development capabilities. This
Capability                 can start with an overall assessment, such as SEI’s software capability
                           evaluation, which focuses on the details of tools, metrics, personnel
                           facilities, and management control. Contracting agencies should require
                           that offerers selected for best and final offers submit to a capability
                           evaluation with the goal of achieving a level-3 rating.

                           Air Force Guidelines: A prudent contractor will implement a measurement
                           process that includes collecting and receiving actual data and analyzing
                           that data. For measurement to be effective, a metrics usage plan should be
                           developed to determine what data should be collected and analyzed.
                           Contractors should be required to submit this plan to the government,
                           since it describes to what extent and at what frequency offerors will
                           provide metrics to the government and how they will be used internally to
                           manage the proposed program.

                           CMM: In order for software development organizations to achieve lasting
                           results from process improvement efforts, it is necessary to design an
                           evolutionary path that increases an organization’s software process
                           maturity in stages. The capability maturity model for software provides
                           software organizations with guidance on how to gain control of their
                           processes for developing and maintaining software and how to evolve
                           toward a culture of software engineering and management excellence. The
                           CMM guide was designed to assist software organizations in selecting
                           process improvement strategies by determining current process maturity
                           and identifying the issues most critical to software quality and process
                           improvement.

                           Measurements are made and used to determine the functionality and
                           quality of software products. Examples of these measurements include the
                           numbers, types, and severity of defects identified in the software products
                           tracked cumulatively and by stage. Other measurements are made and
                           used to determine the status of software product engineering activities
                           such as the cost to implement and test software changes. As part of
                           software project tracking and oversight, the size of the software products
                           is tracked, actual size of code is compared with estimates documented in




                           Page 80                                    GAO/AIMD-97-78 Medicare Transaction System
                          Appendix III
                          Key Provisions of Laws, Regulations, and
                          Best Practices Relating to Information
                          Technology Investments




                          the software development plan, and the overall projected size of the
                          software is refined, monitored, and adjusted regularly.

                          SAF: Agencies should ensure that they design, develop, test and deploy
                          their automated systems in accordance with conventional management
                          and technical practices. This activity includes evaluating the agency’s
                          software engineering processes for critical attributes such as configuration
                          management, software subcontract management, and software quality
                          assurance. Also, agencies are responsible for preparing estimated project
                          cost schedules using commercially available software packages, such as
                          the constructive cost model or the software life cycle management (SLIM)
                          model, or customized in-house techniques.


Project Schedule          CCA,40 USC 1427: The agency head shall identify in the agency’s IRM plan
Development and           (required by PRA) “any major information technology acquisition program,
Management                or phase or increment of such a program, that has significantly deviated
                          from the cost, performance, or schedule goals established for the
                          program.”

                          SA-CMM:  A project management plan is required to manage the critical
                          dependencies and critical paths of the project’s overall software
                          acquisition schedule. The project’s overall acquisition schedule typically
                          specifies that milestones, tasks, commitments, critical dependencies,
                          staffing, costs, and reviews are allocated in the schedule consistent with
                          the project’s defined software acquisition process. In addition, critical
                          dependencies and paths defined and reflected in the schedule should be
                          tracked on a regular basis.


Project Risk Management   CCA,40 USC 1422: Requires executive agency heads to design and
Processes                 implement a process to assess and manage the risks of the information
                          technology acquisitions.

                          PRA,44 USC 3506(h)(5): Requires agencies to assume responsibility for
                          assessing and managing the risks of major information systems initiatives
                          through a process that is (1) integrated with budget, financial, and
                          program management decisions, and (2) used to select, control, and
                          evaluate the results of major information systems initiatives.




                          Page 81                                    GAO/AIMD-97-78 Medicare Transaction System
Appendix III
Key Provisions of Laws, Regulations, and
Best Practices Relating to Information
Technology Investments




OMB Memorandum 97-02: Directs federal agencies to reduce risks
associated with information technology investments “by establishing clear
measures and accountability for project progress.”

Air Force Guidelines: Provides detailed guidance on the development,
components, and operation of an effective risk management process.

SA-CMM: Recommends that organizations identify risks as early as possible,
adjust the acquisition strategy to manage those risks, and develop and
implement a risk management process as an integral part of the
organization’s software acquisition process.

Improving Mission Performance Through Strategic Information
Management and Technology, GAO/AIMD-94-115: Recommends that agencies
include a disciplined risk management process based on explicit criteria,
to assess risk as a component of managing information systems projects
as investments.

OMB’s Practical Guide: Recommends that senior managers compare the
preliminary results of information technology projects against projected
costs, benefits, and risks, and identify actual or potential managerial,
organizational, or technical problems.




Page 82                                    GAO/AIMD-97-78 Medicare Transaction System
Appendix IV

Major Contributors to This Report


                       Mark E. Heatwole, Assistant Director
Accounting and         Leonard J. Latham, Technical Assistant Director
Information            Danny R. Latta, Technical Adviser
Management Division,   Yvette R. Banks, Senior ADP Telecommunications Analyst
                       James Houtz, Senior Information Systems Analyst
Washington, D.C.       Elizabeth A. Roach, Senior Business Process Analyst
                       J. Michael Resser, Business Process Analyst
                       Michael P. Fruitman, Communications Analyst


                       John Mollet, Senior Evaluator
Kansas City Regional
Office




(511212)               Page 83                           GAO/AIMD-97-78 Medicare Transaction System
Ordering Information

The first copy of each GAO report and testimony is free.
Additional copies are $2 each. Orders should be sent to the
following address, accompanied by a check or money order
made out to the Superintendent of Documents, when
necessary. VISA and MasterCard credit cards are accepted, also.
Orders for 100 or more copies to be mailed to a single address
are discounted 25 percent.

Orders by mail:

U.S. General Accounting Office
P.O. Box 6015
Gaithersburg, MD 20884-6015

or visit:

Room 1100
700 4th St. NW (corner of 4th and G Sts. NW)
U.S. General Accounting Office
Washington, DC

Orders may also be placed by calling (202) 512-6000
or by using fax number (301) 258-4066, or TDD (301) 413-0006.

Each day, GAO issues a list of newly available reports and
testimony. To receive facsimile copies of the daily list or any
list from the past 30 days, please call (202) 512-6000 using a
touchtone phone. A recorded menu will provide information on
how to obtain these lists.

For information on how to access GAO reports on the INTERNET,
send an e-mail message with "info" in the body to:

info@www.gao.gov

or visit GAO’s World Wide Web Home Page at:

http://www.gao.gov




PRINTED ON    RECYCLED PAPER
United States                       Bulk Rate
General Accounting Office      Postage & Fees Paid
Washington, D.C. 20548-0001           GAO
                                 Permit No. G100
Official Business
Penalty for Private Use $300

Address Correction Requested