oversight

Audit of the Department's Contract for the COD System.

Published by the Department of Education, Office of Inspector General on 2005-01-28.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                  Audit of the Department’s Contract 

                         for the COD System





                                   FINAL AUDIT REPORT
                                            ED-OIG/A07-E0003
                                              January 2005




Our mission is to promote the efficiency,                      U.S. Department of Education
effectiveness, and integrity of the                               Office of Inspector General
Department’s programs and operations.                                  Kansas City, Missouri
                                        NOTICE 

            Statements that managerial practices need improvements, as well as other 

                         conclusions and recommendations in this report 

          represent the opinions of the Office of Inspector General. Determinations of 

corrective action to be taken will be made by the appropriate Department of Education officials.


         In accordance with Freedom of Information Act (5 U.S. C. § 552) reports 

                  issued by the Office of Inspector General are available to 

   members of the press and general public to the extent information contained therein is not 

                              subject to exemptions in the Act. 

                           UNITED STATES DEPARTMENT OF EDUCATION
                                           OFFICE OF THE INSPECTOR GENERAL




                                                       JAN 2 8 2005
MEMORANDUM


TO:            Theresa S. Shaw
               Chief Operating Officer
               Federal Student Aid

FROM:          Helen Lew           fle~ W
               Assistant Inspector General for Audit

SUBJECT: 	     Final Audit Report - Audit a/the Department's Contract for the COD Syslem
               COllIro/ No. ED-OIG/A07-E0003

Attached is the subject final audit report that covers the results of our review of the Department's
share-in-savings contract for the COD system. An electronic copy has been provided to your
Audit Liaison Officer. We received your comments concurring with three of the finding and the
corresponding recommendations in our draft report. You non-concurred with two of our
findings and agreed with most of the recommendations in these findings. Based on the
information provided in your comments, we eliminated our finding that FSA paid a deliverable­
based payment for a system that did not meet requirements and modified our finding that FSA
overstated savings by not adjusting for all costs incurred.

Corrective actions proposed (resolution phase) and implemented (closure phase) by your office
will be monitored and tracked through the Department's Audit Accountability and Resolution
Tracking System (AARTS). ED policy requires that you develop a final corrective action plan
(CAP) for our review in the automated system within 30 days of the issuance of this report. The
CAP should set forth the specific action items, and targeted completion dates, necessary to
implement final corrective actions on the finding and recommendations contained in this final
audit report.

In accordance with the Inspector General Act of 1978, as amended, the Office of Inspector
General is required to report to Congress twice a year on the audits that remain unresolved after
six months from the date of issuance.

In accordance with the Freedom of Information Act (5 U.S.c. §552), reports issued by the Office
of Inspector General are available to members of the press and general public to the extent
infonnation contained therein is not subject to exemptions in the Act.

We appreciate the cooperation given us during this review. If you have any questions, please
call Richard J. Dowd, Regional Inspector General for Audit, at 312-886-6503.

Artachmem
                                      400 MARYLAND AVE .• S.W.. WASHINGTON. DC 20202 - 1510
                                                                 W\O.'W.ed.gov

             Our m;ss;orl is tD erlsure equal access ID eduCGliorl arid ID promote educational excel1erlce throughDUl the rtation.
     Audit of the Department’s Contract for the COD System 



                               Table of Contents 




Executive Summary …………………………………………………………………..….….1 


Background ………………………………………………………………………….………3 


Audit Results ……………………………………………………………………...…..……..4 


   Finding 1 - FSA Contract Requirements Were Not Tied to COD Performance 

              Goals………………...……………………….….……….…...………….....4 


   Finding 2 – FSA Lacked Adequate Oversight of Contract Deliverables Related 

               to System Development Life-Cycle (SDLC) Management Activities..…...6 


   Finding 3 - FSA Overstated Savings by Not Adjusting for All Costs Incurred.….…..11 


   Finding 4 - FSA Failed to Ensure Continuation of Service and Adequate 

              Audit Access to the System….…………...………...……….……….……13 


Other Matters………………………………………………………………………………..15 


Objectives, Scope, and Methodology……………………………………………………….16 


Statement on Internal Controls……………………………………………………………...17 


Appendix - Auditee Comments……………………………………………………………..18 





                                  ED-OIG/A07-E0003

           Audit of the Department’s Contract for the COD System 



                                   Executive Summary

Our objectives in reviewing the Department of Education’s (the Department) share-in-savings
(SIS) agreement for the Common Origination and Disbursement (COD) system were to 1)
determine whether the COD system’s performance is consistent with the contract terms for
system acceptance; 2) assess the effectiveness of contract oversight by the contracting officer’s
representative and the contracting officer over the contractor’s system development life-cycle
management efforts to ensure that products and services are delivered and contract terms were
met; 3) determine if work related to the SIS may have been done under prior task orders; 4)
evaluate issues related to ownership of the COD system code; and 5) review contractor invoices
for its share of savings.

Although we found no violation of contract requirements regarding system acceptance, we identified
contract weaknesses that may have contributed to the continued operational problems identified by
our systems audit. Specifically, the contract requirements for system acceptance were not tied to the
overall contract performance measures. In addition, we found that the Department’s Office of
Federal Student Aid (FSA)

• 	 Lacked adequate oversight of contract deliverables related to system development life-cycle
    (SDLC) management activities. FSA contracted for the process re-engineering, cost and
    benefit analysis, and user and functional requirements related to its SDLC management of the
    COD system development. However, the contract requirements included in the task orders did
    not specify the level of detail needed to satisfy the requirements for these SDLC management
    activities. Although requirements for the deliverables were not explicitly detailed in the
    contract, the acceptance criteria were generally adequate to have evaluated the contractor
    deliverables against available external criteria for SDLC before acceptance. We found that
    FSA accepted deliverables even though the contractor’s process re-engineering options and
    analysis were weak; cost and benefit analysis was incomplete; and user requirements and
    functional requirements were inadequate. Clearer, more detailed requirements may have
    helped avoid or reduce, at least in part, the operational problems experienced during initial
    implementation of the system.

• 	 Overstated savings by not adjusting for all costs incurred. FSA contracted for preliminary
    planning for the COD system and paid about $7 million under prior, fixed-price task orders
    without adjusting the savings from the SIS agreement to reflect these costs. As such, the
    savings generated from the SIS agreement are overstated.

• 	 Did not ensure continuation of service and adequate audit access to the system. The COD
    SIS Work Order does not give the Department ownership to the coding for the COD system
    when the task order is completed. The agreement was for a service, which the contractor
    sub-contracted, without ensuring continuation of the service. Consequently, the


ED-OIG 	                                  A07-E0003                                   Page 1
   subcontractor retains ownership of the code, essentially the COD system, creating audit
   limitations and future contracting issues.

We recommend that FSA ensure that future contracts for system development efforts explicitly
detail requirements, as well as, specific performance measures and/or acceptance criteria; that
contracts are managed using performance measures and/or acceptance criteria included in the
contract; and that any calculation for the Modernization Partner’s share of savings under an SIS
agreement include an adjustment or amortization of all costs incurred for the system under
previous or concurrent arrangements. We also recommend that future agreements for FSA’s
modernization effort ensure audit access and continuation of services.

The Other Matters section of this report contains a discussion of an issue related to FSA’s failure
to provide clear supplemental guidance to the contractor regarding allowable travel expenses;
and lack of effective management oversight in reviewing the contractor’s monthly operations
reports.

FSA generally agreed with the findings and recommendations in Draft Report Findings 1, 2, and
5 (renumbered as Finding 4 in this report). Based on FSA’s response, we eliminated Draft
Report Finding 3. Draft Report Finding 3 dealt with the appropriateness of $24 million in
deliverable-based payments for functionality that the contractor did not deliver timely. FSA
disagreed with Draft Report Finding 4 (renumbered as Finding 3 in this report) that FSA
overstated savings and the corresponding recommendations to adjust the savings to reflect costs
under prior contracts and costs related to deliverable-based payments. FSA maintained that the
deliverable-based payments were reflected in the total cap of $57 million detailed in the
agreement’s price proposal. Although FSA provided additional information related to the cost
principle governing contractor claims for travel, the principle incorporates portions of the
Federal Travel Regulation (FTR) dealing with the maximum per diem rates – an issue identified
in the Other Matters section of this report.

We have incorporated FSA’s comments and summarized the comments and our response at the
end of each respective finding. The full text of FSA’s comments is included as an Appendix.




ED-OIG                                    A07-E0003                                    Page 2
              Audit of the Department’s Contract for the COD System 


                                              Background 


According to FSA’s Modernization Blueprint, the development of the COD system is a major
system improvement within FSA’s modernization efforts. The COD system was implemented in
April 2002 to create a single system and a common process for requesting, reporting, and
reconciling Pell Grants and Direct Loans. The COD system is used by FSA to receive school
disbursement information for student-level federal aid and validate aid payments to schools.
According to its Capital Asset Plan and Business Case, the purpose of the COD system is to
provide

• 	 A modernized platform and record for schools to originate and disburse Title IV funds, as
    well as a common process that addresses both the overlapping and individual needs of each
    of the programs (i.e., Direct Loan, Pell, and Campus-based).

• 	 Extensive customer self-service capabilities through the internet, which enables participating
    institutions of higher education to more easily reconcile Title IV data and manage funding;
    and enhances the ability to support schools in the reconciliation and substantiation processes.

Preliminary work on the COD system was done prior to creating the SIS agreement for a fixed-
price amount of about $7 million. FSA also had a fixed-price work order for the Definition Phase
– a phase that allowed the contractor to evaluate the options of converting the work order from a
fixed-price to an SIS agreement before making the decision to convert. In addition to the $7
million already paid for the COD system against prior task orders, the Department has obligated
over $134 million to the SIS agreement. Of the total SIS agreement obligations, FSA has paid
over $125 million, including $24 million for two deliverable-based payments and over $5 million
for share-in-savings – the remaining $96 million covered the operational costs as of June 2004.

Under a share-in-savings contract, a contractor assumes a risk by incurring the contract’s up-
front costs, with the plan of receiving a return from future savings. For example, if a contractor
proposes and implements a savings plan that proves to be successful, the contractor is paid a
negotiated percentage of the savings or revenue generated by its efforts. Because of the
increased risk to the Federal Government and to a contractor, SIS contracts require
knowledgeable program and contracting officials, as well as continuous vigilance on the part of
the contracting agency.

The COD system experienced operational problems during initial implementation of the system
and lower than expected user participation. Recent problems noted in our systems audit1include
promissory notes not linking to loans; student’s record or loan information changes not always
updated on COD or uploaded correctly; and the COD system not always acknowledging
acceptance or denial on a loan application to inform the school and/or student of the status of the
application. In addition, schools reported that COD system error messages were not user friendly,
but rather were confusing and difficult to resolve. These issues are reported in a separate report.
1
    Audit of Federal Student Aid’s Common Origination and Disbursement System (ACN A11-D0004).
ED-OIG 	                                       A07-E0003                                         Page 3
           Audit of the Department’s Contract for the COD System 



                                        Audit Results 


A systems and a contract audit were performed concurrently. This review was limited to
contract performance. We found that the COD system’s performance was consistent with
contract terms, which specified functional requirements but only one data processing
requirement for acceptance of the system. We also found that the contract requirements were not
tied to the contract performance goals to increase both customer and employee satisfaction.

We identified issues within the contract that, if improved in future contracting efforts, may
facilitate system development efforts by reducing the likelihood of many of the problems
experienced with the COD system recurring. Specifically, we found that FSA

• 	 Lacked adequate oversight of contract deliverables related to system development life-cycle
    (SDLC) management activities.

• 	 Overstated savings by not adjusting for all costs incurred.
• 	 Did not ensure continuation of service and adequate audit access to the system.

FSA also did not independently validate the contractor’s proposed baseline before approving it.
We were not able to validate any line item in the agreed to baseline because of differences in cost
categories between the baseline and FSA’s invoices; however, we found that the total baseline
was not significantly different than the total costs provided by FSA in support of the baseline.

The Other Matters section of this report contains a discussion that FSA did not provide clear
supplemental guidance to the contractor regarding allowable travel expenses; and lacked
effective management oversight in reviewing the contractor’s monthly operations reports.




Finding 1 – FSA Contract Requirements Were Not Tied to COD Performance
             Goals


The terms for acceptance of the system included requirements that did not tie to FSA’s stated
goals for COD, which were to increase both customer and employee satisfaction, as well as,
decrease cost. The contract specified functional requirements and included only one data
processing requirement for acceptance of the system. FSA’s acceptance of a multi-million dollar
system based on one data processing performance measure is, in our opinion, a contract
weakness. Our review was limited to the contractual requirements for acceptance of the system,
because the documents comprising the COD Work Order refer to a number of external
documents and deliverables from prior task orders, making it difficult to identify the specific
contractual requirements for the COD system beyond the acceptance criteria.
ED-OIG 	                                  A07-E0003                                    Page 4
The initial acceptance criteria included detailed functional requirements for the COD application
functionality, customer service capabilities, and operations. FSA initially determined that the
contractor provided 90 to 93 percent of the required functionality, however, the contractor
eventually provided 100 percent of the required functionality according to FSA’s documentation.
The second acceptance criterion required that the COD system meet or exceed the FSA COD
performance measurement of financial integrity, which is 80 percent of funds drawn down by
schools (meaning submitted to COD) substantiated with actual disbursement records within the
30 day regulatory requirement. The COD system has been fulfilling this data processing
requirement in accordance with contract terms. As of July 2004, the percent of funds
substantiated by COD has been right at 100 percent for the school years 2002 through 2005. For
example, the July 2004 FSA Operational Metrics indicates that the year to date drawdown
substantiated for Direct Loan was 99.8 percent. However, FSA’s use of only one performance
measure for system acceptance may have contributed to FSA’s difficulty in averting or quickly
resolving the system’s problems in other areas identified in our systems audit of COD (ACN-
A11-D0004).

According to the COD Statement of Work (SOW), dated October 2001, FSA’s goal is to
    create a Performance-Based Organization (PBO) focused on the business outcomes
    of lower operational unit costs, and higher customer and employee satisfaction.
    The Common Origination and Disbursement (COD) solution addresses this goal by
    providing a common platform and record for schools to originate and disburse Title
    IV funds, as well as a common process that addresses both the overlapping and
    individual needs of each of the programs (i.e., Direct Loan, Pell, and Campus-
    based). This consolidation and integration of the systems and infrastructures that
    currently support the programs leads to reduced unit costs for SFA2, lower costs for
    schools, a higher level of satisfaction for participants in the Title IV programs, and
    more proactive, detective, and preventive financial controls.

The system acceptance criteria were not linked to the specific goals documented in Task Order
77, Work Order 2. Incentives and penalties are inherent in SIS contracts given the payment
calculation based on reduced operational costs, which implies a more efficient process.
However, a contractor could deliver a poor performing system, such as, a system that could not
adequately process the data without manual intervention or could not interface with the required
systems. As long as that system reduced operational costs, the contractor would receive a
payment for SIS. The COD Price Proposal includes incentive payments beyond the SIS that tie
to specific system goals without any stated penalties for not achieving those goals. In addition,
use of incentive payments raises questions about whether SIS was an appropriate contracting tool
for this modernization effort.

As of June 2004, the Department had obligated over $134 million to the SIS agreement. The
Department has actually paid over $125 million to the contractor including $24 million for two
deliverable-based payments, which is discussed in Finding 3 of this report, and over $5 million
for share-in-savings. Projected operational costs and savings over the life of the contract – FY
2002 through 2011 – are expected to be over $478 million and nearly $96 million, respectively.

2
 The contract for the COD system was initiated prior to the office’s name change from Student Financial Assistance
(SFA) to Federal Student Aid (FSA).
ED-OIG                                          A07-E0003                                          Page 5
The contractor will receive the entire operational costs and its share in the expected savings is
estimated to be $53 million. With the amount of funds at stake, FSA should have included
specific contractual requirements tied to performance goals.

Recommendation
We recommend that, for future technology investments, the Chief Operating Officer for FSA
ensure

1.1 	 Future contracts explicitly state requirements and acceptance criteria, including
      requirements tied to performance goals to ensure that those goals will be achieved before
      accepting a deliverable.

FSA Comments

FSA agreed with the finding and the recommendation, but indicated that it has already
implemented the recommendation. Specifically, FSA agreed that the business case objectives
were not clearly represented in the COD contract, but stated that it has implemented procedures
and processes to better manage the acquisition process to ensure contract requirements
adequately address business case objectives. FSA added that its acquisition process has
incorporated past recommendations of both the OIG and GAO related to competition,
modernization, performance-based contracting, acquisition support to the integration efforts, as
well as the target vision; and ensuring contract deliverables reflect the business case objective.

OIG Response

While the procedures and processes cited by FSA appear to address our finding and
recommendation, they were implemented after our audit period and were not evaluated as part of
this audit




Finding 2 – FSA Lacked Adequate Oversight of Contract Deliverables Related to
             System Development Life-Cycle (SDLC) Management Activities


FSA’s contractor for deliverables related to the SDLC management of the COD system was also
responsible for designing, developing, implementing, and operating the system. Our review was
limited to the deliverables for two task orders related to SDLC management activities – process
re-engineering, cost and benefit analysis, and user and functional requirements analysis. The
contract requirements included in the task orders did not specify the level of detail needed to
satisfy the SDLC requirements for these deliverables. The Statement of Work provided that the
contractor would not use FSA’s Solution Life-Cycle (SLC) process, but would follow an SDLC
methodology. The contract does not identify the methodology for carrying out the SDLC.
Although specific SDLC requirements for the deliverables reviewed were not detailed in the task
orders, in most cases, the acceptance criteria were adequate for FSA to have evaluated the
deliverables against external criteria for these SDLC management activities before acceptance.

ED-OIG 	                                   A07-E0003                                    Page 6
FSA accepted contract deliverables that did not adequately meet the SDLC management
requirements related to process re-engineering, cost and benefit analysis, and user and functional
requirements analysis.

We reviewed the contract deliverables for two task orders related to SDLC management
activities and found that the process re-engineering options and analysis was weak; cost and
benefit analysis was incomplete; and user requirements and functional requirements were
inadequate.

        Weak Process Re-engineering Options and Analysis

        FSA contracted through Task Order 2 to estimate the value of identified opportunities by
        producing external industry metrics, which provide a benchmark against which to
        measure the SFA current environment. The acceptance criteria required an inventory of
        projects for re-engineering opportunities containing budgets and descriptions, with
        recommendations for actions to be taken.

        The Re-engineering Options and Analysis for COD, a Task Order 2 deliverable, provided
        a high-level discussion, and recommended elimination of manual processes during the re-
        engineering that were easily identifiable. The document provided high-level best
        practices for re-engineering FSA’s process for origination and disbursement of funds, but
        provided no information on how it acquired, developed, or assimilated data to arrive at
        those best practices.

        According to industry best practices, re-engineering generally approaches the problem
        from the point of view of the customer and the process. Customer views are required to
        ensure that the eventual design satisfies the users. Process views are required to remove
        current problems and focus activity on the goal of the process rather than the function, so
        that unneeded or irrelevant activity is removed. The Re-engineering Options and
        Analysis for COD did not address the customer or the process.

        Specifically, the COD re-engineering analysis did not contain an analysis of the
        Department’s mission needs, which is needed before it revises the mission-related and
        administrative processes and before making significant investments in information
        technology (IT) to be used to support those missions. The Clinger-Cohen Act3 suggests
        that the processes be revised and improved through a business process re-engineering
        effort and, once improved, investments in IT systems should be aimed at automating
        those improved processes.4 Further, the Re-engineering Options and Analysis for COD
        did not document that the contractor quantitatively benchmarked agency process
        performance against comparable processes and organizations in the public or private
        sectors in terms of cost, speed, productivity, and quality of outputs and outcomes, as
        required by the Clinger-Cohen Act.5



3
  Previously referred to as the Information Technology Management Reform Act of 1996, Division E of Public Law

104-106, 110 Stat. 679 (1996). 

4
  The Clinger-Cohen Act, Section 5123 (5).

5
  The Clinger-Cohen Act, Section 5123 (4).

ED-OIG                                         A07-E0003                                         Page 7
     The documentation to support COD as part of a work process redesign does not provide
     the substance necessary to meet the requirements for revising administrative processes
     and performing work process redesign before making significant investments in
     information technology. Prior to investing in information technology, an analysis for
     revising and redesigning work processes is needed. This will help ensure that the
     investment will achieve greater effectiveness and efficiency in the organization’s
     programs and operations.

     Incomplete Cost and Benefit Analysis

     FSA contracted for a cost and benefit analysis under Task Orders 2 and 19. Task Order 2
     directed the contractor to estimate the value of identified opportunities by producing
     external industry metrics, which provide a benchmark against which to measure the SFA
     current environment. The acceptance criteria required an inventory of opportunities
     projects containing budgets and descriptions, with recommendations for actions to be
     taken. Task Order 19 directed the contractor to provide “a presentation that clearly
     explains the rationale behind selecting a COD solution over other potential options.” The
     acceptance criteria included in the task order required that the document “contain a list of
     the significant re-engineering options selected … and explain the rationale for
     eliminating or selecting specific options”; and “describe the expected impact of COD in
     terms of the SFA goals of Increased Customer Service, Decreased Costs, and Increased
     Employee Satisfaction.”

     The Re-engineering Options and Analysis, states that FSA evaluated six options
     including acquiring a commercial off-the-shelf software package, custom system
     development, maintaining status quo, and variations thereof. However, it did not provide
     any detailed analysis of these options including any costs or valued benefits. Also, there
     was no analysis of the risk for contracting for a service versus custom system
     development. Task Order 19 required a presentation explaining the rationale for
     selecting the COD solution over other potential options rather than a detailed analysis of
     all options, including costs and benefits. The wording in the task order required an
     explanation for selecting the COD solution, which could have contributed to the omission
     of detailed cost and benefit information for other options in the deliverable.

     The analysis appears to have preceded the user and functional requirements analyses.
     Information on user and functional requirements is needed to identify the most promising
     options and to provide accurate cost and benefit information for assessing the options. A
     complete and accurate cost and benefit analysis is necessary to demonstrate that the
     investment chosen clearly provides the most efficient allocation of total agency resources,
     and to assure that the right investment decision was made.

     Inadequate User and Functional Requirements Analyses

     FSA contracted, through Task Order 19, to develop more detail to the COD functional
     requirements provided in Task Order 2, which assessed the current environment. Task
     Order 19 included requirements to build on previous COD deliverables that listed high
     level COD “To-Be” functional requirements. The acceptance criteria states that the task
     order deliverable will describe “the COD “To-Be” Functional Requirements [and] … will

ED-OIG                                  A07-E0003                                   Page 8
          specifically address new requirements in the areas of Performance-Based Funding,
          Subsidiary Ledger processing, and Campus-Based Student-Level reporting.”

          Although institutions input the data used by FSA to manage the Pell Grant and Direct
          Loan programs, the user requirements were from high-level process owners at the
          Department responsible for carrying out critical business requirements. We found no
          evidence that the contractor identified all potential users let alone consulted with
          participating institutions in developing the user requirements. According to an FSA
          official, FSA understood user requirements from a historical perspective given its
          previous dealings with end users of the legacy systems replaced by the COD system. As
          such, FSA did not view the lack of input from institutions as a problem in determining
          user requirements.

          Although the contractor documented system requirements and indicated that it assessed
          compatibility to the current departmental environment and technology drivers, a
          functional enterprise architecture was not available at the time of this assessment. In
          September 2002, we reported that although both the Department and FSA have made
          progress in laying the groundwork for their enterprise architectures, critical elements
          need to be completed; specifically, the architectures need to be integrated, and data
          standardization characteristics and techniques need to be fully implemented. 6 According
          to an independent contractor used by FSA to review the COD system requirements, the
          COD system operational problems experienced during its initial implementation were
          due, in part, to unclear requirements and software design defects. An FSA official
          acknowledged that user and functional requirement documentation was limited.

The task orders contracting for process re-engineering, cost and benefit analysis, and user and
functional requirements did not include requirements and acceptance criteria that were specific
enough to satisfy the requirements related to SDLC management for the deliverables reviewed.
However, the acceptance criteria were general enough for FSA to have evaluated the deliverables
against available external criteria included in best practices documents, such as, information
from the National Institute of Standards and Technology, and the Carnegie Mellon Software
Engineering Institute. FSA accepted contract deliverables that did not provide adequate process
re-engineering, cost and benefit analysis, and user and functional requirements. Had FSA 1)
explicitly specified the SDLC requirements in the task orders; and/or 2) adequately reviewed the
contractor’s deliverables documenting SDLC management before accepting the deliverables,
many of the weaknesses identified in this review and our systems audit (ACN A11-D0004) may
have been corrected and operational problems experienced in the initial implementation of the
system may have been avoided.

Recommendations
We recommend that, for future technology investments, the Chief Operating Officer for FSA
ensure




6
    ED-OIG/A07-C0001, Audit of Enterprise Architecture, dated September 2002.
ED-OIG                                          A07-E0003                             Page 9
2.1 	 Contracts for SDLC explicitly detail requirements and specific acceptance criteria requiring
      deliverables to be consistent with the terms of the contract.

2.2 	 Effective oversight of personnel involved in evaluating SDLC contract deliverables.

FSA Comments

FSA agreed with the finding and the recommendations. FSA agreed that oversight of the
contractor’s system development management efforts related to the COD system needed
improvement, especially during the initial phase task orders. FSA stated that it improved
oversight in the detailed requirements, design, development, test, and implementation and
maintenance stages.

In addition, although not directly in response to Finding 2, FSA stated that it no longer relies on
vendors who will perform development and operational contracts for support in calculating
investment cost. Instead, FSA awarded two contracts to small businesses in the spring of 2004
for quantitative and qualitative analysis support to assist with baseline cost, investment and
related analysis, where the complexity of analysis exceeds the skills of FSA’s own employees.
These vendors are precluded under the agreement from working on any vendor team engaged to
perform work under operations or development contracts.

FSA indicated that it has already addressed the first recommendation under this finding by
requiring vendors to follow a standard, proven SDLC methodology for system development
acquisitions. FSA added that subsequent to the COD contract, it implemented a number of steps
to enhance contract oversight including enhanced training for its Contracting Officer’s
Representatives. Specifically, FSA stated that it had initiated a process to re-engineer the
performance management training for those responsible for FSA contracts, including those
contracts with IT systems being developed under an acceptable SDLC. The re-engineered
curriculum includes training on earned value management (EVM), performance management,
contract management, and project management. Going forward, FSA committed to continuing
to strengthen oversight efforts relative to its contractors.

OIG Response

While FSA’s comments appear to address our finding and recommendations, the corrective
actions were implemented after our audit period and were not evaluated as part of this audit.
FSA must still provide adequate contract oversight – evaluating contract deliverables to
determine accuracy and completeness, and whether the documentation provides FSA with the
information necessary to make decisions related to a modernization or system development
project.




ED-OIG 	                                  A07-E0003                                    Page 10
Finding 3 – FSA Overstated Savings by Not Adjusting for All Costs Incurred


FSA contracted for preliminary planning of the COD system under prior, fixed-price task orders
without factoring the cost of these task orders into the SIS calculation. Had FSA adjusted the
savings for the cost of preliminary planning and the cost of the system, or amortized the cost
over the life of the agreement, the savings would have been lower than calculated.

Using an SIS contracting tool, a contractor assumes a risk by incurring the contract’s up-front
costs, with the plan of receiving a return from future savings. The SIS payment generally
provides a greater return on investment to compensate the contractor for assuming the up-front
risk. The SIS calculation compares the cost of operating the legacy systems to the cost of
operating the new or modernized system as the performance measure for determining how
successfully the contractor has designed the system. However, in this contract, FSA paid for
preliminary planning for the system under prior task orders.

The contractor performed preliminary planning related to the COD system, under fixed-price
Task Orders # 2, 6, and 19 and received almost $7 million for that work. However, the $7
million was not included in the SIS calculation even though the deliverables directly related to
the total cost of the system. FSA used a fixed-price work order under the SIS Task Order to
evaluate whether the contractor could make a better return on its investment by converting the
fixed-price to an SIS. The initial Work Order #2 under the SIS Task Order #77, dated October
2001, included a fixed-price for the Definition Stage – a stage that allowed the contractor to
evaluate the options of converting the Work Order from a fixed-price to an SIS agreement before
making the decision to convert. We question whether FSA should have paid the contractor to
evaluate whether an SIS contract was advantageous to the contractor.

FSA has paid the contractor over $5 million in share-in-savings payments calculated as the
contractor’s share in the reduction of operational costs after migrating operations from the legacy
systems to the COD system. Though it may appear that the contractor reduced costs and
contributed to savings under the SIS agreement for the COD system, those savings are overstated
because the calculation for the savings has not been adjusted to reflect the $7 million spent on
planning for the system. If FSA had adjusted the savings for the entire cost of the system, or
amortized the cost over the life of the agreement, the savings would have been lower than
calculated.

Recommendations
We recommend that the Chief Operating Officer for FSA




ED-OIG                                    A07-E0003                                   Page 11
3.1 	 Establish appropriate control procedures to assure that the contractor’s share of savings
      under an SIS agreement does not include savings that are attributable to a previous or
      concurrent fixed-price arrangement or another SIS agreement.

3.2 	 Ensure that all future SIS agreements include an adjustment for the costs, such as pre-
      development costs that are directly related to the project.

3.3 	 Include SDLC costs when reporting the total cost of the COD system.

FSA Comments

FSA disagreed with the finding and the recommendations in the draft report that SIS payments
be adjusted to reflect $7 million paid for preliminary planning work and $24 million in
deliverable-based payments. FSA stated that the terms of the contract did not permit the
adjustments recommended and that any change would require the agreement of the contractor or
payment of compensation.

FSA added that the fact that “the government spent funds under different contracts to study (or
even to begin development of a system) has no bearing on its obligations under the separate and
distinct Share-in-Savings contract.” FSA agreed that, based on recently issued FAR guidance
and GAO recommendations, such costs should be included in the baseline for Share-in-Savings
arrangements. FSA added that it would ensure that any future Share-in-Savings arrangements
are written to properly include theses costs as part of the overall Share-in-Savings calculation.

FSA disagreed with a statement in our draft report that FSA did not consider $24 million in
deliverable-based payments in the SIS calculation, which resulted in the savings being
overstated. FSA stated that the payments were made based on meeting a milestone, indicating
that they were not intended to reflect savings. FSA added that section 3.3 of the price proposal,
entitled Modernization Partner Cap, discusses the sharing of savings and concludes with the
following statement: "The monetary cap will be $57 million (including deliverable-based
payments)."

OIG Response

Based on FSA’s comments, we revised our finding and recommendations.

Based on a validation of FSA’s response, we are satisfied that the $24 million in deliverable-
based payments are reflected in the savings as part of the monetary cap of $57 million and have
modified our finding and recommendation, accordingly. However, by including the deliverable-
based payments in the cap, FSA in essence established a cap of $33 million in savings and a
minimum contract payment of $24 million to be paid regardless of the savings realized from the
COD system. Establishing a minimum contract payment amount through deliverable-based
payments in the agreement contradicts the concept of an SIS agreement and raises questions
about whether SIS was an appropriate contracting tool for this modernization effort.

Although the agreement does not provide a mechanism to adjust the SIS calculation for costs
incurred under previous task orders, such costs are necessary for the development of a system
and should be recognized in the total cost of the system/service. Whether FSA can amend the

ED-OIG 	                                  A07-E0003                                    Page 12
COD SIS agreement or not, it should include the $7 million cost of system development efforts
in reporting the total cost of the COD system/service. In its comments, FSA stated that, based on
recently issued FAR guidance and GAO recommendations, such costs should be included in the
baseline for SIS arrangements. FSA added that it would ensure that any future SIS arrangements
are written to properly include theses costs. However, SDLC costs should not be included in the
baseline for SIS arrangements, but rather the savings calculation should be adjusted to amortize
the costs of SDLC management, such as pre-development costs that are directly related to the
project. We modified our recommendation concerning the inclusion of SDLC costs.


Finding 4 – FSA did not Ensure Continuation of Service and Adequate Audit
           Access to the System

The COD SIS Work Order, as written, does not give the Department ownership of the coding for
the COD system when the task order is completed. The SIS agreement was for a service, which
the contractor subcontracted, without ensuring continuation of the service. As such, the
subcontractor retains ownership of the code, essentially the COD system, which has created audit
limitations.

The subcontractor retains the ownership to the system code – effectively the COD system
ownership – even after the task order is completed. Because the contractor subcontracted the
system development, the Department has no direct contract with the entity retaining ownership
of the system. As such, following completion of the SIS Work Order the Department would
have to enter a new service contract with the contractor or subcontractor; negotiate a separate
contract to buy the code; or replace the COD system. FSA decided to replace the COD system
when that contract expires in 2006. FSA is currently in the process of contracting for a system to
replace COD under the Front-End Business Integration (FEBI) contract.7 FSA paid the
contractor a total of $24 million in deliverable-based payments, essentially paying for the cost of
developing the system, and the subcontractor was due to be paid for development of the system
code through its subcontract with the contractor. Because FSA contracted for a service rather
than a system, it is now in the process of replacing that service/system.

Office of Inspector General (OIG) auditors were not able to perform some of the planned audit
work on the audit of the COD system because of access limitations. According to the FSA
Director for the Acquisition Planning and Contract Management Group, the COD SIS Work
Order did not explicitly include provisions requiring audit access nor did it require the contractor
to include those provisions in all subcontracts for the COD system. An independent contractor
tasked by FSA to perform an independent verification and validation of the COD system design
and implementation recommended that, for future releases, FSA consider purchasing the
software instead of a proprietary service; thereby providing full access to all components of the
COD system, at any and all sites of contract performance, including system code.



7
 The FEBI effort includes Applications Processing, which consists of Central Processing System (CPS), FAFSA On
The Web, EDExpress, Student Aid Awareness, EDPubs/UPS and the Public Inquiry Contract (PIC) and the Federal
Student Aid Information Center as well as Common Origination and Disbursement (COD).
ED-OIG                                         A07-E0003                                        Page 13
Recommendations
We recommend that the Chief Operating Officer for FSA ensure that future agreements

4.1 	 Include provisions to ensure continuation of service and the conveyance of data rights
      clauses to appropriate subcontractors.

4.2 	 Provide for audit access at all sites of contract performance and require that the contractor
      include the same provision in all subcontracts.

FSA Comments

FSA indicated agreement with the finding and the recommendations. FSA acknowledged that
the OIG’s systems auditors were initially unable to perform all of their audit procedures relative
to the COD system because of system access limitations placed on them by the subcontractor.
However, it was FSA’s understanding that following FSA discussions with the contractor and the
subcontractor, the requested systems access was granted to the OIG auditors, which enabled
them to complete their audit procedures as intended. FSA added that it uses various authorities
provided under FAR to ensure continuation of services under existing SIS contracts, and that it
will make efforts to ensure that appropriate audit clauses are added to existing arrangements at
the time of renewal, or whenever appropriate.

OIG Response

While OIG auditors were eventually provided access to complete similar audit procedures in a
subsequent audit, the audit team for the COD system audit was not able to complete its audit
procedures. FSA’s commitment to ensure that appropriate audit clauses are added to existing
arrangements should prevent audit access problems in the future.




ED-OIG 	                                   A07-E0003                                    Page 14
           Audit of the Department’s Contract for the COD System 



                                        Other Matters 


FSA did not provide clear supplemental guidance to the contractor regarding allowable travel
expenses; nor did it provide effective management oversight in reviewing the contractor’s
monthly operations reports. As a result, the contractor included questionable charges for travel
in its operational invoices.

Federal Travel Regulations (FTR) contained in 41 Code of Federal Regulations (C.F.R.)
implement the statutory requirements for travel by Federal civilian employees and those
authorized to travel at Government expense. Based on our review of monthly operations reports,
we found that the contractor’s claims were not prepared in accordance with regulations.
Specifically, we found 1) lodging in excess of the General Services Administration (GSA) rate;
2) per diem in excess of the GSA rate; 3) claims for reimbursement not submitted timely; and 4)
miscellaneous expenses charged under lodging. In some instances, we were unable to reconcile
the supporting documentation provided for travel expenses with the contractor’s spreadsheet
submitted for reimbursement. FSA should provide adequate oversight in reviewing the
contractor’s invoices to identify reimbursable travel expenses and question charges that are not
reimbursable.

FSA Comments

FSA maintained that the contractor did not include questionable charges for travel and provided
additional information related to the cost principle governing contractor claims for travel. FSA
stated that cost principle FAR 31.205-46 governs travel expenses. Specifically, FSA stated that
the cost principle incorporates
        the portions of the FTR dealing with the maximum per diem rates, the definitions of
        lodging, meals, and incidental expenses, and the regulatory coverage dealing with special
        or unusual situations…. There are special situations that are recognized in the FTR that
        may require expenses in excess of the per diem rate. If the situation arises infrequently, a
        corporate officer may approve without government review. If frequent, an advance
        agreement with the government contracting officer is necessary.

OIG Response

Since the cost principle incorporates portions of the FTR dealing with the maximum per diem
rates and we identified claims in excess of those rates, we did not modify this section of the
report. While there are recognized exceptions allowing expenses in excess of the per diem rate,
the cost principle requires that a written justification for use of the higher amounts be approved
by an officer of the contractor's organization or designee to ensure that the authority is properly
administered and controlled to prevent abuse. A copy of the written justification should be
attached to the contractor’s invoice for the related travel expenses.



ED-OIG                                     A07-E0003                                    Page 15
           Audit of the Department’s Contract for the COD System 



                        Objectives, Scope, and Methodology 


The objectives of our audit were to
• 	 Determine whether the COD system’s performance is consistent with the contract terms for
    system acceptance;
• 	 Assess the effectiveness of contract oversight by the contracting officer’s representative and
    the contracting officer over the contractor’s system development life-cycle (SDLC)
    management efforts to ensure that products and services are delivered and contract terms are
    met;
• 	 Determine if work related to the SIS may have been done under prior task orders;
• 	 Evaluate issues related to ownership of the COD system code; and
• 	 Review the contractor invoices for its share of savings.

To accomplish our objectives, we reviewed applicable FSA policies and procedures, as well as
laws, regulations, and agency guidelines addressing system modernization and contracting. We
interviewed officials in FSA including the contracting officer, contracting officer’s
representative, and program management. We obtained and reviewed FSA’s Modernization
Blueprint, the BPA for FSA’s Modernization Partner (Modernization Partner Agreement), and
task orders and work order issued under the BPA that specifically provided COD system
deliverables including SDLC documentation. We identified and reviewed the deliverables for
two task orders related to SDLC. We also obtained and reviewed all contractor invoices for the
COD system, including invoices for deliverables under the previous fixed-price task order, share-
in-savings, and system operations as of June 2004. In addition, we reviewed SDLC best
practices documents and prior OIG audit reports, along with Government Accountability Office
reports, applicable to this area.

We obtained access to COD system operational reports and were provided system performance
information from our system audit (ACN A11-D0004), done concurrently with this audit – the
results of the system audit will be reported in a separate audit report. We reviewed the system
operational reports to determine whether system performance was consistent with the
requirements included in the SIS Work Order for data processing. To meet our objectives, we
did not rely on electronic data from the Department.

We conducted work at the Department’s FSA offices in Washington, D.C. and our OIG office in
Kansas City, MO, during the period November 2003 to June 2004. We held an exit conference
with Department and FSA officials on September 30, 2004. Our audit was performed in
accordance with generally accepted government auditing standards appropriate to the scope of
the review.



ED-OIG 	                                  A07-E0003                                   Page 16
           Audit of the Department’s Contract for the COD System 



                           Statement on Internal Controls 


As part of our review, we gained an understanding of the Department’s and FSA’s internal
control structure over the Modernization Partner Agreement, specifically, the SIS agreement for
the COD system. We also reviewed the policies, procedures, and practices applicable to the
scope of the review. For the purpose of this report, we assessed and classified the significant
internal controls related to the Department’s SIS agreement for the COD system into the
following categories:

       •   Acceptance of deliverables

       •   Procedures to evaluate contract performance

       •   Contract payments

       •   Compliance with laws and regulations

Because of inherent limitations, a study and evaluation made for the limited purpose described
above would not necessarily disclose all material weaknesses in the internal controls. However,
our assessment identified internal control weaknesses as set out in the Audit Results and Other
Matters sections of this report.




ED-OIG                                   A07-E0003                                  Page 17
         APPENDIX: Auditee Comments




ED-OIG           A07-E0003            Page 18 

                                          FEDERAL
                                          STUDENT AID
                                                                                 13


                               CHIEF OPERATING OFFICER

TO:           Richard J. Dowd
              Regional Inspector General for Audit
              Office of the Inspector General

FROM:         Theresa S.  Shaw~
              Chief Operating Officer

SUBJECT:      DRAFT AUDIT REPORT - "Audit of the Department's Contract for the
              Common Origination and Disbursement (COD) System," dated October
              22,2004, (Control No. ED-OIG/A07-E0003)

Thank you for providing us with an opportunity to respond to the Office of the Inspector
General's (DIG) draft audit report, "Audit of the Department's Contract for the COD
System," Contfol Number ED-OIG/A07-E0003, dated October 22, 2004. We are pleased
the report noted that the COD system is fulfilling its data processing requirements in
accordance with the contract temlS for this system. We believe we have made significant
progress in resolving issues regarding the COD System contract. The attached report
provides a response to each finding and recommendation.

Overall, we agree with many of the observations made by the DIG relative to the
Department's contract for the COD system. We agree that contractor oversight during
the initial task orders needed improvement and steps were taken to enhance our oversight.
We implemented an improved methodology for system development and addressed many
of the shortcomings initially identified.

However, there are certain issues that were identified by the DIG where FSA does not
concur. Specifically, while some operational problems were identified during the initial
implementation of the COD system, we do not consider them to be unusual, and we are
generally pleased with the system's performance to date. The COD system has
significantly reduced processing time and now handles 100% of Pell and Direct Loan
funds. Since going live, the system has processed more than 58 million disbursements
totaling more than $42 billion. In addition, according to the tenns of the contract, FSA
has realized cost savings during Fiscal Year (FY) 2003 and FY 2004 totaling $13.7
million. Finally, using the American Customer Satisfaction Index (ACSI) survey as our
tool for measuring customer satisfaction with COD, we established a baseline measure of
66 for COD customer satisfaction during FY 2003. In FY 2004, the overall COD
customer satisfaction scores increased by 10 percent.

We take exception to the OIG's finding that FSA "paid a deliverable·based payment that
did not meet requirements." At the time FSA made the payments in question, the
contractor had delivered (and FSA had put into production) a system containing more
                         830 First Street, NE, Washington, D.C 20202
                                        1-8DO-4-FEO-AIO
                                     www.studenraid.ed.gov
than 90 percent of the required functionality and was working to deliver the remaining
functionality. The payments made to the contractor excluded a "hold-back" amount to
encourage completion and delivery of this additional functionality.

We also disagree that FSA overstated savings by not adjusting for all costs relating to this
Share-in-Savings arrangement. The costs cited by the OIG relate to early task orders,
which were not part of the Share-in-Savings contract, and, therefore, were properly
excluded. We do agree, however, that based on recently issued Federal Acquisition
Regulation (FAR) guidance and Government Accountability Office (GAO)
recommendations, such costs should be included in the baseline for Share-in-Savings
arrangements. We will ensure that any future Share-in-Savings arrangements
contemplated by FSA are written to properly include theses costs as part of the overall
Share-in-Savings calculation.

Thank you again for the opportunity to review and comment on this report.

Attachment

cc:    Patrick J. Howard




                                             2

Response to Draft Audit Report, "Audit of the Department's Contract for the COD
System," Control No. ED-OIGIA07-E0003.



Finding I - FSA Contract Requirements \Ve re Not Tied to COD Performance Goals

FSA's Response:

FSA agrees that business case objectives were not clearly represented in the COD
contract FSA has implemented procedures and processes to better manage the
acquisition process to ensure contract requirements adequately address business case
objectives. FSA's improved funct ion aligns with the Government Accountability
Office 's (GAO) acqu isition framework. Specifically, FSA's acquisition process has
incorporated past recommendations of both the DIG and GAO for the following contract
areas:

   •     Competition;
   •     Modernization;
   •     Perfonnance-based contracting;
   •     Acquisition support to the integration efforts as well as the target vision; and
   •     Ensuring contract deliverables reflect business case objective .

FSA has made significant progress in deve loping the necessary long-tenn procedures and
processes to ensure continuous improvement for the acquisit ion function. In 2003, FSA
developed an enterprise-wide procurement plan to align the acquisition function as well
as contract management with the FSA target vision . This pl an is a comprehensive
examination of mission need, acquisition timing and opportunities for consolidation.
FSA is one of the few federal civilian organizations to develop such a comprehensive
plan. Complementing the plan is an enterprise market research plan supported by both
strategic and tactical market research effo rts.

To support acquisition planning and related market research efforts, FSA is developing a
procurement-planning database . The database will allow FSA to plan th e sequencing of
its acquis itions with fl exibility and efficiency. Additionally. FSA is building a cost
library, which will allow it to refine its approach to business case development,
independent government cost estimates ( ICE's) and investment decisions. Finally, the
acquisition function is reflected in the annual plan; ensuring specific perfonnance goals
for procurement are tied to FSA and departmental goals.

As a result of our own analyses and market research, FSA reassessed its requirements for
integrat ion/modernization vendors. In the spring of2004, FSA issued a sol icitation for
an agreement intended to, in part, replace our modernization partner agreement. This
effort resulted in the award of a new agreement, known as the Integration Leadership
Support Contract (ILSC), to Grant-Thornton (GT) and a broad team of supporting
vendors. Under this agreement, GT and its team members are excluded from perfonning
operations or development contracts.
Response to Draft Audit Report, "Audit of the Department's Contract for the COD
System," Control No . ED-OIG/A 07-E0003.



Likewise, FSA no longer relies on vendors who will perform development and
operational contracts for support in calculating investment cost. Instead, FSA awarded
two contracts to small businesses in the spring of2004 for quantitative and qualitative
(Q&Q) analysis support. Where the complexity of analysis exceeds the skills ofFSA's
own employees, the Q&Q vendors are engaged to assist with baseline cost, investment
and related analysis . Like the ILSC vendor, the Q&Q vendors are precluded under the
agreement from working on any vendor team engaged to perform work under operations
or development contracts.

Additionally, to improve our overall management of the acquisition function, we have
taken several steps to strengthen internal controls to ensure business case objectives are
reflected in our contracts. In response to the Chief Operating Officer's (COO's) request
for assistance in improving internal controls, FSA's procurement executive established a
special Contract Review Board (CRE) in January 2003. The CRE provides department­
wide quality assurance on procurement activities. Similarly, we have established a
Project Management Office for improved contract monitoring and to identify the best
perfonnance metrics for use throughout the organization.

OIG Recommendation

We recommend that, for future teclmology investments, the Chief Operating Officer for
FSA ensure:

1.1 	   Future contracts explicitly state requirements and acceptance criteria, including
        requirements tied to performance goals to ensure that those goals will be achieved
        before accepting a deliverable.

FSA' s Response:

FSA requires that contracts (for completion efforts) include appropriate requirements and
acceptance criteria tied to contract deliverables, which are reflective of the business case
objective pertaining to each contract.

Finding 2 - FSA Lacked Adequate Oversigbt of Contract Deliverables Related to
System Development Life-Cycle (SDLC) Management Activities


FSA's Response:

FSA agrees that oversight of the contractor's (Accenture) system development
management efforts relative to the COD system - especially during the initial phase task
orders for the project needed improvement.

However, steps were taken in the detailed requirements, design, development, test, and
implementation and maintenance stages to improve oversight. Specifically, the



                                              2

Response to Draft Audit Report, "Audit of the Department's Contract for the COD
System," Control No. ED-OIG/A07-E0003.



contractor completely revamped its methodology used for startup (which was specified in
the contract) to make it more robust. The new methodology that was implemented was
more appropriate for a significant development project such as COD. It includes clear
sign-off milestones, phase containment and specified outcome. COD is currently being
operated and maintained under an Accenture SDLC methodology that is assessed at the
Software Engineering institute's Capability Maturity Model Integration (CMMi) Level 4.
The CMMi framework is a widely recogn ized standard for assess ing and improving
software development processes.

While we acknowledge that certain operational problems occurred during the initial
implementation of the COD system, we believe this is a relatively normal condition in a
major system development project and are generally pleased with the system's
performance to date.

Significant milestones relative to the COD system include:

        • 	 The successful conversion of mill ions of records from legacy systems to the COD 

            system or COD data archive; 

        • 	 The processing of more than 42 million disbursement records totaling in excess of 

            S58 billion; 

        • 	 COD is handling 100% of Pell and Direct Loan funds;
        • 	 Substantiation in excess of 99% of school draw downs with actual disbursement 

            data; 

        • 	 lmplementation of the industry standard Common Record, which is currently 

            being processed by more than 75% of all schools; 

        • 	 Submiss ion of 50% of Direct Loan promissory notes electronically; and
        • 	 Reduction in processing time from days to hours.

Additionally, since this contract was negotiated, FSA has implemented a number of steps
to enhance oversight of its contractors including enhanced training for Contracting
Officer's Representatives (see response to recommendation 2.2 below.) Going forward,
FSA will continue to work to further strengthen oversight efforts relative to its
contractors.

DIG Recommendations

We recommend that, fo r future technology investments, the Chief Operating Officer for
FSA ensure:

2.1 	      Contracts for SDLC explicitly detail requ irements and specific acceptance criteria
           requiring deliverables to be consistent with the terms of the contract.

2.2 	      Effective oversight of personnel involved in evaluating SOLC contract
           deliverables.
                                                                                                  •


                                                3

Response to Draft Audit Report, "Audit of the Department's Contract for the COD
System," Control No. ED-OIG/A07-E0003.




FSA's Response:

2.1 	   For system development acquisitions where an SDLC methodology is
        appropriate, FSA requires vendors to follow a standard, proven methodology such
        as CMMi or ISO 9001.

2.2 	   We agree with this recommendation. Ensuring the professional capability of our
        Contracting Officer's Representative (COR) workforce is essential. It is
        specifically critical for staff responsible for evaluating contract deliverables
        produced under a contract requiring an acceptable SDLC methodology. To this
        end, FSA has initiated a process to re-engineer the performance management
        training for those responsible for FSA contracts, including those contracts with IT
        systems being developed under an acceptable SOLe. The re-engineered
        cuniculum includes training on earned value management (EVM), performance
        management, contract management, and project management. The EVM training,
        in particular, will assist program management staff to better monitor and evaluate
        contract deliverables related to the SDLC methodology. This training effort will
        be an on-going professional development process that will provide for beginner
        classes as well as refresher and update classes for existing professional CORs and
        others to keep current with evolving technology and contract management
        procedures.

Finding 3 - FSA Paid a Deliverable-based Payment for a System tbat Did Not Meet
            Requirements

FSA did, in fact, make payments totaling 511.160 million to the contractor (Accenture)
prior to their having met 100% of the acceptance criteria for the COD system. The initial
deliverable-based payment of 512 million was scheduled in the contract's price proposal
for October 2002. According to that schedule, FSA made the payment determination five
months after the COD system initial release implementation date of April 29, 2002. COD
contract management requested an evaluation of implemented COD requirements against
the contractual acceptance criteria by FSA's COD development managers. The
development managers deemed the COD system to be in substantial compliance with the
acceptance criteria. FSA was fully aware at the time of the payments that a small portion
(7%) of the acceptance criteria had not yet been met. However, the use of this system in
production mode and over an extended period of time constituted constructive acceptance
of this system and obligated FSA to make the payments set forth in the COD contract.

Management considered these factors when determining the timing of payments to
Accenture and concluded that the multiple payment approach ultimately used was the
appropriate course of action and would help facilitate completion of the remaining items
pertaining to this system.




                                             4
Response to Draft Audit Report, "Audit of the Department's Contract for the COD
System," Control No, ED-OlG/A07-E0003.



The guidance FSA properly followed to manage the situation described in the finding
concerning nonconforming supplies is addressed in the FAR at 46.407, which states, in
part:

    (a) The contracting officer should reject supplies or services not conforming in all
        respects to contract requirements (see 46.102). In those instances where deviation
        from this policy is found to be in the government's interest, such supplies or
        services may be accepted only as authorized in this section.

    (b) The contracting officer ordinarily must give the contractor an opportunity to
        correct or replace nonconforming supplies or services when this can be
        accomplished within the required delivery schedule. Unless the contract specifies
        otherwise (as may be the case in some cost-reimbursement contracts), correction
        or replacement must be without additional cost to the government. Paragraph
        (e)(2) of the clause at 52.246-2, Inspection of Supplies-Fix ed-Price, reserves to
        the government the right to charge the contractor the cost of government
        reinspection and retests because of prior rejection.

   (c)(I)ln situations not covered by paragraph (b) of this section, the contracting
      officer ordinarily must reject supplies or services when the nonconformance is
      critical or major or the supplies or services are otherwise incomplete. However,
      there may be circumstances (e.g., reasons of economy or urgency) when the
      contracting officer determines acceptance or conditional acceptance of supplies or
      services is in the best interest of the government [emphasis added]. The
      contracting officer must make this determination based upon­

     (i) 	     Advice of the technical activity that the item is safe to use and will perform
               its intended purpose;

     (ii) 	    Information regarding the nature and extent of the nonconformance or
               otherwise incomplete supplies or services;

     (iii) 	   A request from the contractor for acceptance of the nonconforming or
               otherwise incomplete supplies or services (iffeasible);

     (iv) 	    A recommendation for acceptance, conditional acceptance, or rejection, with
               supporting rationale; and

     (v) 	 The contract adjustment considered appropriate, including any adjustment
           offered by the contractor.

In the case of the COD system, FSA conditionally accepted the system in accordance
with (c)(l) above. Payment following conditional acceptance such as occurred here is
covered in FAR 46.407(f) as follows:




                                               5
Response to Draft Audit Report, "Audit of the Department's Contract for the COD
System," Control No. ED-OIG/A07-E0003.



     (f) 	 When supplies or services are accepted with critical or major
           nonconformances as authorized in paragraph (c) of this section, the
           contracting officer must modify the contract to provide for an equitable price
           reduction or other consideration. In the case of conditional acceptance,
           amounts withheld from payments generally should be at least sufficient to
           cover the estimated cost and related profit to correct deficiencies and complete
           unfinished work [emphasis added] . The contracting officer must document
           in the contract file the basis for the amounts withheld. For services, the
           contracting officer can consider identifying the value of the individual work
           requirements or tasks (subdivisions) that may be subject to price or fee
           reduction. This value may be used to determine an equitable adjustment for
           nonconforming services. However, when supplies or services involving
           minor nonconformances are accepted, the contract need not be modified
           unless it appears that the savings to the contractor in fabricating the
           nonconforming supplies or performing the nonconforming services will
           exceed the cost to the Government of processing the modification.

As required under the prevailing regulation, FSA withheld an amount sufficient to cover
the deficiencies noted during inspection. Thereafter, in accordance with the regulation,
the vendor corrected the deficiencies and delivered the balance of the functionality in two
subsequent releases. On delivery, inspection and acceptance, FSA made payment after
each release was accepted. Again, FSA's actions were entirely in accordance with the
regulation.

While there are several examples found in court and board decisions that support FSA's
actions in regard to conditional acceptance and the making of partial payment for the
COD system, the Veterans Board of Contract Appeals (V ABCA) No.6 t 25 02-2 BCA P
32,003, Appeal of Fisher Imaging Corporation, is particularly relevant as it demonstrates
what would have happened had FSA acted as the OIG appears to prefer.

Here, the V A received a piece of medical equipment (an EPX Electrophysiology
Preferred System) and after installation and inspection the V A found some 16
deficiencies with the equipment. The vendor proceeded to correct the deficiencies while
the VA made payment for 80% of the contract price and proceeded to perform hundreds
of medical procedures with the equipment. After four failed attempts to correct the
deficiencies the Contracting Officer (CO) terminated the contract demanding with his
determination return of the monies paid (80% of the contract price).

The V ABCA overruled the CO and dismissed the government's demand for repayment of
80% of the contract price as well as demands for re-procurement cost.

Like the VA case, FSA partially accepted and partially paid for the COD system, putting
it into production with 93% of functionality delivered. In production, the system
delivered significant benefits to FSA and the customers it serves. Only after subsequent




                                             6

Response to Draft Audit Report, "Audit of the Department's Contract for the COD
System," Control No. ED-OIG/A07-EOO03.


releases brought the COD system into 100% compliance with contract tenns did FSA
make full payment. Unlike VA, FSA followed the dictates of the prevailing regulation
and was able to manage this implementation to success.

OIG Recommendations

Because contract perfonnance measures are only effective when they are used to manage
the contract, we continue to recommend that the Chief Operating Officer for FSA:

3.1 	   Ensure that contracting officials manage contracts using perfonnance measures
        included in the contract; and

3.2 	   Ensure that FSA does not pay contractor invoices, including invoices for incentive
        fees or deliverable-based payments, when performance measures or acceptance
        criteria are not met.

FSA's Response:

3.1 	   FSA agrees with this recommendation. Standard operating procedures are and
        will cont inue to be in complete compliance with this recommendation. FSA's
        COD management use reports on system performance measures and data on a
        daily basis to manage the contract. Performance data is reported on daily
        production reporting as well as on a report presented during a weekly project
        status review.

3.2 	   FSA agrees with this recorrunendation and follows the Department's standard
        contract performance management guidance, "OeFO: 2- 108 Contract Monitoring
        for Program Officials," dated September 16, 2004, to manage vendor performance
        including monitoring performance measures.

Finding 4 - FSA Overstated Savings by Not Adjusting for AU Costs Incurred

FSA disagrees with th is finding because the costs that the O IG cites were obligated under
separate contracts (Task Orders 2 and 19) than the Share-in-S avings contract. The fact
that the government spent funds under different contracts to st udy (or even to begin
development of a system) has no bearing on its obligations under the separate and distinct
Share-in-Savings contract.

OIG Recommendations

We recommend that the Chief Operating Officer for FSA:

4.1 	   Adjust the S IS payments to reflect the $7 million cost of the system development
        efforts included in previous fixed-price agreements or amortize the cost over the
        life of the agreement, adjusting the savings accordingly.


                                              7

Response to Draft Audit Report, "Audit of the Department's Contract for the COD
System," Control No. ED-OIG/A07 -E0003.




4.2 	   Adjust the SIS payments to reflect the $24 million cost of the system or amortize
        the cost over the life of the agreement, adjusting the savings accordingly.

4.3 	   Establish appropriate control procedures to assure that the contractor's share of
        savings under an SIS agreement does not include savings that are attributable to a
        previous or concurrent fixed-price arrangement or another SIS agreement.

FSA's Respo nse:

4.1 	   Unfortunately, the tenns of the instant contract do not provide for the government
        to make a unilateral adjustment to the SIS payments as prescribed by the OIG.
        Because there is no basis for making these adjustments, and because there would
        have been a significant negative financial impact on the vendor, we are confident
        that they would not agree to such a change.

        Specifically, the tenns of the COD system order provide for a specific way of
        calculating the savings generated by the order. Specifically, the order states:

        "The Modernization Partner's share of the monthly savings will be calculated as
        follows:

        I. 	 Monthly Baseline Costs will be calculated by applying the applicable
             processing year ramp up percentages to the Fiscal Year Baseline Costs (as
             adjusted annually) from Table 2.

        2. 	 Total monthly operations costs that are included in the SIS/SIR calculation.
             This total excludes those items listed in section 3.2.

        3. 	 Monthly savings will be calculated by subtracting the COD Operations costs
             at month end from the monthly COD Baseline Costs.

        4. 	 The Modernization Partner's share will be calculated by applying the
             applicable Fiscal Year Modernization Partner Share-of-Results Percentage
             from Table to the calculated monthly savings."

        The tenns of this contract are lUlambiguous and bind the government as well as
        the vendor.

        The government has the right to make unilateral changes to contracts only as set
        forth in the changes clause FAR 52.243-1. This clause serves four major
        purposes:




                                              8

Response to Draft Audit Report, "Audit of the Department's Contract for the COD
System," Control No. ED-OIG/A07-E0003.



        I. 	 To provide operating flexibility to the govenunent by providing for unilateral
             changes in work to accommodate technological advances and changes in the
             government's needs;

        2. 	 To provide the contractor an avenue through which to propose changes to
             work;

        3. 	 To provide the govenunent authority to procure additional services within the
             general scope of the contract; and

        4. 	 To provide a mechanism through which the contractor can process claims.

        While the govenunent has the sovereign right to make changes to its
        requirements, the vendor has a right to be compensated for any changes. The
        government does not have the right to make unilateral changes to the payment
        terms of a contract as is suggested. Nor does it have the right to consider the
        compensation to a contractor under a separate contract when making decisions
        regarding pricing on another contract.

        While the facts as stated are not disputed, the government has no authority to do
        as the OIG recommends .

4.2 	   The facts and discussion regarding the government's authority above in
        recommendation 4.1 apply here as well.

        There are two distinguishing factors between the costs identified in
        recommendations 4.1 and 4.2. These are: 1) the costs were paid on the task order
        where savings accrue and 2) the costs are discussed separately from the SIS
        payments.

        The fact that these costs were accrued under the instant contract and the payments
        were made based on meeting a milestone indicate that they were not intended to
        reflect savings. The most significant indicator of the intent of the parties is the
        discussion in the price proposal section 3.3 entitled "Modernization Partner Cap."
        This section discusses the sharing of savings and concludes with the following
        statement: "The monetary cap will be $57 million (including deliverable-based
        payments)." Clearly this indicates that once savings (as calculated plus the
        payments made on the deliverable basis) reach $57 million, no addit ional savings
        will be shared.

        Absent the plain language of the order which indicates the intent of the party, as
        previously discussed in response to recommendation 4.1, the government has no
        authority to make a unilateral change to the payment terms of this or any contract.




                                              9

Response to Draft Audit Report, "Audit of the Department's Contract for the COD
System," Control No, ED-OIG/A07 -E0003,



        Although the OIG may not agree that the payment terms reflect what it believes
        should be reflected, the terms of the contract are only factors to be considered in
        the interpretation of the contract unless an ambiguity exists. In this case, no such
        ambiguity has been identified and therefore the recommendation cannot be acted
        upon.

4.3 	   There is proposed FAR language (see FAR Case 2003 -008, Share-In-Savings
        contracting; published at 69 FR 40514, July 2,2004) concerning SIS contracting
        that FSA believes will be helpful once official. Additionally, the GAO has
        developed some particularly useful guidance in a series of reports which FSA
        relies on presently, and will continue to rely on in the future for any SIS
        initiative. This guidance includes two reports and one letter:

           • 	 Acquisition Management: Agencies Can Improve Training on New
               Initiatives, GAO-03-2SI (Washington, D,C,: Jan, 15, 2003),
               http://www .gao. gov/new. items/d03281.pdf

           • 	 Contract Management: Commercial Use of Share-in-Savings Contracting,
               GAO-03-327 (Washington, D,C,: Jan, 31,2003).
               hnp:/lwww.gao.gov/new.items/d03327.pdf

           • 	 Contract Management: OFPP Policy Regarding Share-in-Savings
               Contracting Pursuant to the E-Government Act 0[2002 (Washington,
               D.C.: Mar 24, 2003)
               http://www . gao. govInew .item s/ d03 552r. pd f

        The new FAR guidance and the GAO recommendations are insufficient to ensure
        FSA's business interests are protected in selecting any contracting method.
        Therefore, FSA also relies on the investment management process (predominately
        found in Office of Management and Budget (OMS) Circular A-II), review and
        analysis of various acquisition strategies, carefully supported with business cases
        and acquisition plans and the commitment of management.

        Since September 2003, FSA has not utilized the SIS approach for any
        procurement it was contemplating. However, this approach as well as many
        others remain options available on a business case-by-business case basis.

Finding 5 - FSA Did Not Ensure Continuation of Service and Adequate Audit
Access to the System

FSA's Response:

We understand that the OIG's systems auditors were initially unable to perfoml all of
their audit procedures relative to the COD system because of system access limitations
placed on them by the subcontractor (TSYS). It is our understanding, that following FSA


                                              10 

Response to Draft Audit Report, "Audit of the Department's Contract for the COD
System," Control No. ED-OiG/A07-E0003.



discussions with the Accenture and TSYS, the requested systems access was granted to
the OIG auditors, which enabled them to complete their audit procedures as intended.

OIG Recommendations

We recommend that the Chief Operating Officer for FSA ensure that future agreements:

5.1 	    Include provisions to ensure continuation of service.

5.2 	    Provide for audit access and require that the contractor include the same provision
         in all subcontracts.

FSA's Response:

5.1 	    FSA utilizes various authorities provided under FAR Parts 6.3, 52.237·3, 16.5 and
         17.1, to ensure continuation of services provisioned under existing SIS contracts.

5.2 	    Efforts will also be made to ensure that appropriate audit clauses are added to
         existing arrangements at the time of renewal, or whenever appropriate.

Other Matters

FSA did not provide clear supplemental guidance to the contractor regarding allowable
travel expenses; nor did it provide effective management oversight in reviewing the
contractor's monthly operations reports.

FSA's Response:

We do not believe the contractor included questionable charges for travel. In fact,
Federal Travel Regulation (FTR) does not dictate what contractors are allowed regarding
payment for lodging and meals and incidental expenses (M&IE). The cost principle,
FAR 31.205-46, governs the allowability of such expenses. The cost principle does
incorporate the portions of the FIR dealing with the maximum per diem rates, the
definitions of lodging, meals, and incidental expenses, and the regulatory coverage
dealing with special or unusual situations. In most circumstances, the maximum per
diem rate is the threshold ofallowability.

An excerpt of FAR 31.205-46 follows:

        (2) Except as provided in subparagraph (a)(3) of this subsection, costs incurred for
        lodging, meals, and incidental expenses (as defined in the regulations cited in
        (a)(2)(i) through (iii) of this subparagraph) shall be considered to be reasonable and
        allowable only to the extent that they do not exceed on a daily basis the maximum
        per diem rates in effect at the time of travel as set forth in the-­




                                                II 

Response to Draft Audit Report, "Audit of the Department's Contract for the COD
System," Control No. ED-OIG/A07-EOO03.



     (i) Federal Travel Regulation, prescribed by the General Services Administration,
     for travel in the contiguous United States, available on a subscription basis from the
     Superintendent of Documents, U.S. Government Printing Office, Washington, DC
     20402, Stock No. 922-002-00000-2;

Expenses in excess of the per diem are not necessarily unallowable. There are special
situations that are recognized in the FTR that may require expenses in excess of the per
diem rate. If the situation arises infrequently, a corporate officer may approve without
government review. If frequent, an advance agreement with the government contracting
officer is necessary.

An excerpt of FAR 31.205-46 follows:

     (3) In special or unusual situations, actual costs in excess of the above-referenced
     maximum per diem rates are alJowable provided that such amounts do not exceed
     the higher amounts authorized for Federal civilian employees as permitted in the
     regulations referenced in (a)(2)(i), (ii), or (iii) of this subsection. For such higher
     amounts to be allowable, all of the following conditions must be met:

     (i) One of the conditions warranting approval of the actual expense method, as set
     forth in the regulations referred in (a)(2)(i), (ii), or (iii) of this subsection, must
     exist.

     (ii) A wriuenjustification for use of the higher amounts must be approved by an
     officer of the contractor's organization or designee to ensure that the authority is
     properly administered and controlled to prevent abuse.

     (iii) Ifit becomes necessary to exercise the authority to use the higher actual
     expense method repetitively or on a continuing basis in a particular area, the
     contractor must obtain advance approval from the contracting officer.

The conditions warranting actual expenses (up to 300 percent of the per diem rate) are
found in FTR 301-11.300 (Excerpt follows):

§301-11.300 When is actual expense reimbursement warranted?

When:

(a) Lodging and/or meals are procured at a prearranged place such as a hotel where a
meeting, conference or training session is held;




                                               12 

Response to Draft Audit Report, "Audit of the Department's Contract for the COD
System," Control No. EO-OIG/A07 -E0003.




(b) Costs have escalated because of special events (e.g., missile launching periods,
sporting events, World's Fair, conventions, natural disasters); lodging and meal expenses
within prescribed allowances cannot be obtained nearby; and costs to commute to/from
the nearby location consume most or all of the savings achieved from occupying less
expensive lodging;

(c) Because of mission requirements; or

(d) Any other reason approved within your agency.




                                             13