oversight

Review the FSA Modernization Partner Agreement to (1) determine the adequacy of the performance indicators established to measure contractor performance, (2) assess the agreement's compliance with laws and regulations, and (3) determine the accuracy of the costs incurred for the products and services provided.

Published by the Department of Education, Office of Inspector General on 2002-11-20.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

       Audit of FSA’s Modernization Partner Agreement





                                   FINAL AUDIT REPORT 

                                     ED-OIG/A07-B0008 

                                       November 2002 





Our mission is to promote the efficiency,                 U.S. Department of Education
effectiveness, and integrity of the                          Office of Inspector General
Department’s programs and operations.                      Kansas City, Missouri Office
                                        NOTICE 

            Statements that managerial practices need improvements, as well as other 

                         conclusions and recommendations in this report 

          represent the opinions of the Office of Inspector General. Determinations of 

corrective action to be taken will be made by the appropriate Department of Education officials. 


         In accordance with Freedom of Information Act (5 U.S.C. § 552), reports 

           issued by the Office of Inspector General are available, if requested, to 

   members of the press and general public to the extent information contained therein is not 

                              subject to exemptions in the Act 

                       UNITED STATES DEPARTMENT OF EDUCATION

                                    OFFICE OF INSPECTOR GENERAL




MEMORANDUM
                                                 NOV 2 02002
TO: 	         Theresa S. Shaw
              Chief Operating Officer
              Federal Student Aid

              Jack Martin 

              Chief Financial Officer 

              Office of the Chief Financial Officer 



FROM:          He len Lew   ~/h-- LtJ
               Acting Assistant Inspector General for Auclit


SUBJECT: 	 FINAL AUDIT REPORT 

           Audit of FSA' s Modernization Partner Agreement 

           Control No. ED-OIG/A07-B0008 



Attached is our subject final report presenting our findings and recommendations resulting from
our review ofFSA's Modernization Partner Agreement. The report identifies areas for
improvement and provides recommendations. We received your comments, which basically
concurred with the findings in our draft audit report.

Please provide the Supervisor, Post Audit Group, Office of Chief Financial Officer and the
Office of Inspector General with quarterly status reports on promised corrective actions until all
such actions have been completed or continued follow-up is unnecessary.

In accordance with the Freedom of Information Act (5 U.S.C. §552), reports issued by the Office
of Inspector General are available, if requested, to members of the press and general public to the
extent information contained therein is not subject to exemptions in the Act.

We appreciate the cooperation given us in the review . Should you have any questions
concerning this report, please call William Allen at (816) 268-0501 . Please refer to the above
audit control number in all correspondence relating to this report.


Attachment
          Audit of FSA's Modernization Partner Agreement 



                             Table of Contents



Executive Summary ………………………………………………………………………………1 


Background ……………………………………………………………………………………… 3 


Audit Results ……………………………………………………………………………...…..…..5 


    Finding No. 1 - Task Orders Include Inadequate Performance Measures….……………..5 


    Finding No. 2 - FSA Has Not Reviewed the Modernization Partner Agreement to 

                     Ensure that FSA Receives the Best Value………………………………8 


    Finding No. 3 - The Baseline FSA Used for Its Share-in-Savings Task Order Is 

                     Overstated...……………………………………………………………10 


Other Matters ……………………………………………………………………………………13 


Objectives, Scope, and Methodology …………………………………………………………...15 


Statement on Management Controls …………………………………………………………….17 


Appendix ………………………………………………………………………………………...18 

                 Audit of FSA's Modernization Partner Agreement 



                                    Executive Summary 



The Modernization Partner Agreement is a Blanket Purchase Agreement between the Department of
Education (Department) and Accenture (ED-99-DO-0002). Federal Student Aid (FSA) uses the
Modernization Partner Agreement to implement its Modernization Blueprint. As of August 9, 2002,
the Modernization Partner Agreement obligated nearly $244 million under 112 task orders, four of
which are share-in-savings task orders.

We reviewed the FSA Modernization Partner Agreement to (1) determine the adequacy of the
performance indicators established to measure contractor performance, (2) assess the agreement’s
compliance with laws and regulations, and (3) determine the accuracy of the costs incurred for the
products and services provided. We found that FSA payments to the Modernization Partner were
accurate and in accordance with the terms of the authorizing task orders. However, we found that

• 	 Contract performance measures were inadequate. Of the 47 task orders reviewed, 36 included
    deliverables with inadequate performance measures because those measures could not be used
    effectively to measure the usability of the deliverables or to determine that the contractor’s
    performance was in accordance with contract terms. We recommend that FSA ensure that future
    performance-based contracts or task orders include effective performance measures to evaluate
    the contractor’s performance.

• 	 The required review of the agreement to ensure continued best value was not completed. FSA
    has not reviewed the Modernization Partner Agreement to ensure that the agreement continues to
    represent the best value. We recommend that FSA ensure that the annual review of the
    agreement is completed as required and that the review include a comparison of costs under the
    agreement to an outside market in determining whether the agreement continues to provide the
    best value for contracted services.

• 	 The share-in-savings task order baseline was overstated. In August 2001, we reported that the
    baseline used to measure the FSA Modernization Partner’s performance and to calculate
    payments under a share-in-savings task order was overstated. An overstatement creates a larger
    contractor payment than is actually earned. The baseline was overstated because it did not
    reflect savings from a prior, separate FSA initiative. Although FSA disputed the amount that the
    baseline was overstated, it agreed that our recommendations represent good business practice
    and agreed to implement them in future share-in-savings agreements. We continue to
   recommend that FSA use the most current and relevant data available when it establishes a
   baseline for a share-in-savings contract or task order.

The Department generally agreed with our findings. We have incorporated their comments, where
appropriate, and have summarized the comments and OIG’s response at the end of each respective
finding. The full text of the Department’s comments is included as an Appendix.
                 Audit of FSA's Modernization Partner Agreement 



                                          Background 



The Higher Education Amendments of 1998 established a Performance-Based Organization (PBO) –
a discrete management unit responsible for managing the operational functions supporting the
programs authorized under Title IV of the Higher Education Act of 1965, as amended (HEA). The
Amendments gave the PBO independent control of its budget allocations and expenditures,
personnel decisions and processes, procurements, and other administrative and management
functions. The responsibilities of the PBO included integrating the information systems supporting
the Federal student financial assistance programs; implementing an open, common, integrated
system for the delivery of student financial assistance under Title IV; and developing and
maintaining a student financial assistance system that contains complete, accurate, and timely data to
ensure program integrity. Federal Student Aid (FSA), the PBO, developed the Modernization
Blueprint as its plan for changes necessary to meet modernization goals.

Using GSA’s schedule of contractors, the Department of Education (Department) competed its
requirements for a “Modernization Partner” to assist in implementing the Department’s
Modernization Blueprint. In September 1999, the Department issued a Blanket Purchase
Agreement (BPA: ED-99-DO-0002) with Accenture (formerly known as Andersen Consulting) as
the FSA Modernization Partner. The BPA’s scope of work indicates that the overall intent of the
Modernization Partner Agreement is to assist in implementing the Department’s Modernization
Blueprint.

The Blueprint explains how and when FSA will modernize and includes the FSA Modernization
Business Requirements, a Functional and Technical Architecture, as well as a Sequencing Plan to
migrate from FSA’s current working environment to the target architecture. FSA released the first
Modernization Blueprint on September 30, 1999; the second is dated July 15, 2000; the third was
completed in 2001; and the fourth edition of the Blueprint was completed in March 2002. The
Department has not yet approved the last two updates to the Blueprint for public release.

According to the Modernization Partner Agreement, the planned outcome of the Modernization
Blueprint is to reduce and retire redundant data and systems, through a planned and systematic
process of transitioning to the modernized system and the elimination of waste and inefficiencies in
manageable steps. The Modernization Partner is responsible and accountable for the on-time quality
delivery of projects within the Blueprint and other modernization projects that FSA may require in
the future. The Modernization Partner is required to accomplish two primary outcomes:

•   Manage the execution of the Modernization Blueprint
•   Develop, test, deploy, and maintain information systems

Under the BPA, work is to be authorized by task orders, with each task order under the BPA
considered a separate award. All task orders are to include performance objectives that support
FSA’s Modernization objectives, as well as performance measurements to be used to evaluate the
contractor’s performance. As of August 9, 2002, the agreement included nearly $244 million
obligated under 112 task orders, including four share-in-savings task orders: Central Data System
(CDS), Title IV – Wide Area Network (TIVWAN), E-Servicing, and Common Originating
Disbursements (COD).
                 Audit of FSA's Modernization Partner Agreement 



                                          Audit Results 



We reviewed the FSA Modernization Partner Agreement to (1) determine the adequacy of the
performance indicators established to measure contractor performance, (2) assess the agreement’s
compliance with laws and regulations, and (3) determine the accuracy of the costs incurred for the
products and services provided. We found that

• 	 Performance measures in 36 of the 47 task orders reviewed were not adequate to assess whether
    contractor performance was in accordance with contract terms and to assure the performance,
    reliability, and usability of all deliverables;

• 	 FSA has not completed an annual review of the Modernization Partner Agreement to ensure that
    the agreement continues to represent the best value for products and services; and

• 	 The baseline used to measure the contractor’s performance and calculate contractor payments in
    the share-in-savings task order was overstated.

Except as discussed in this report, nothing came to our attention to indicate that the contract was not
administered in accordance with laws and regulations. However, we had some concerns about
whether share-in-savings pricing is fully authorized under the pricing provisions of the GSA
Schedule 70 contract. We plan to continue reviewing this share-in-savings issue in a subsequent
audit. The Other Matters section of this report contains a discussion of a system control weakness
not specifically related to the Modernization Partner Agreement and a general discussion of our
concerns with the share-in-savings pricing issue.



Finding No. 1 – Task Orders Include Inadequate Performance Measures


We reviewed 47 performance-based task orders that included 452 deliverables, issued under the FSA
Modernization Partner Agreement. We found that 36 of the 47 task orders reviewed included 230
deliverables with inadequate performance measures. The performance measures were inadequate
because the measures did not provide sufficient quantifiable or qualitative information to determine
if the contractor’s performance was in accordance with the terms of the contract and that the
deliverables were usable and reliable.
Section 142(c)(1) of the HEA requires the Chief Operating Officer to use performance-based
contracts to the extent practicable. It states

          The Chief Operating Officer shall, to the extent practicable, maximize the use of
          performance-based servicing contracts . . . to achieve cost savings and improve
          service.

Accordingly, the Modernization Partner Agreement states

          All Task Orders shall be performance based. Each Task Order shall include
          performance objectives that reflect and support the Modernization objectives of
          [FSA] for that Task Order.

          Each Task Order shall include performance measurements that will be used to
          evaluate the contractor’s performance under the Task Order and determine if the
          required outcomes have been achieved.

The Modernization Partner Agreement includes a Quality Assurance Plan that describes the quality
review methods used to evaluate the Modernization Partner’s system projects. Under the
Modernization Partner Agreement, the Modernization Partner is required to develop indicators to
measure the Modernization Blueprint’s accomplishments and to ensure that all deliverables
contribute to FSA’s performance goals.

The Clinger-Cohen Act1 requires that agencies provide a process to furnish a means for measuring
performance. Section 5122(b)(6) of the Clinger-Cohen Act states

          The process of an executive agency shall . . . provide the means for senior
          management personnel of the executive agency to obtain timely information
          regarding the progress of an investment in an information system . . . on an
          independently verifiable basis, in terms of cost, capability of the system to meet
          specified requirements, timeliness, and quality.

Under Federal Acquisition Regulation (FAR) § 37.601

          Performance-based contracting methods are intended to ensure that required
          performance quality levels are achieved and that total payment is related to the degree
          that services performed meet contract standards.

1
    Previously referred to as the Information Technology Management Reform Act of 1996, Division E of
    Public Law 104-106, 110 Stat. 679 (1996).
Among other requirements for performance-based task orders, FAR § 37.602-1(b) states

        When preparing statements of work, agencies shall, to the maximum extent
        practicable . . . (1) Describe the work in terms of “what” is to be the required output
        rather than either “how” the work is to be accomplished or the number of hours to be
        provided . . . [and] (2) Enable assessment of work performance against measurable
        performance standards . . . .

We found that 230 of the 452 deliverables reviewed under the Modernization Partner agreement
(over 50 percent) included inadequate performance measures. For example, we found that

• 	 The only acceptance criteria for 35 of the deliverables reviewed were the on-time delivery of the
    products or services and their acceptance by the user. These criteria are not adequate
    performance standards because they do not provide for testing or include product-specific
    indicators that effectively measure the contractor’s performance or the reliability and usability of
    the deliverables.

• 	 The only acceptance criterion for 11 of the deliverables reviewed was that the deliverable is
    delivered and accepted by the user. Several of these deliverables were manuals of instructions
    for performing new procedures. The task orders did not include performance measures to
    evaluate the usability of the instructions.

The absence of adequate performance indicators limits FSA’s ability to determine that contractor
performance is in accordance with contract terms or that the deliverables are useable and reliable. In
addition, without adequate performance measures, FSA lacks assurance that the contractor-provided
deliverables contributed to FSA’s performance goals.

Recommendation

1.1 	   For future performance-based contracts, we recommend that the Chief Operating Officer for
        FSA ensure that the contract or task order includes performance measures to

        • Evaluate the contractor’s performance throughout the performance period.

        • Test the performance, reliability, and usability of all deliverables.

        • Assure that all deliverables contribute to FSA’s performance goals.
Department Comments

The Department concurred with the basic finding and recommendations in the audit report. The
Department stated that it recognized the need to develop performance measures that focus on
outcome and that it had implemented, or planned to implement, actions to develop effective and
quantifiable performance measures. However, the Department indicated that it did not agree that the
report’s recommendations should apply to all deliverables. The Department stated that performance
measures were appropriate for systems development and maintenance, but not for other deliverables
like status reports or plans.

OIG Response

We commend the Department’s initiation of actions to improve performance measures and reiterate
that performance measures are a necessary part of performance-based contracting regardless of the
nature of the deliverable. We disagree with the suggestion that performance measures are not
appropriate for all deliverables. For example, a status report would be of little value if it were not
timely, accurate, and useful for implementing the larger modernization effort. Any effective status
report or plan, if necessary to the modernization effort, could be evaluated according to specific and
appropriate performance measures related to content, accuracy, timeliness, cost-effectiveness, and/or
usefulness.



Finding No. 2 – FSA Has Not Reviewed the Modernization Partner Agreement to
                Ensure that FSA Receives the Best Value


FSA has not reviewed the Modernization Partner Agreement to assess whether the agreement
represents the best value for contracted services. FSA is required to conduct an annual review to
determine that the Modernization Partner Agreement continues to represent the best value for
products and services.

• 	 The Modernization Partner’s Schedule 70 contract states, “When establishing BPAs ordering
    offices shall . . . [r]eview BPAs periodically. Such reviews shall be conducted at least annually.
    The purpose of the review is to determine whether the BPA still represents the best value. . . .”

• 	 The Modernization Partner Agreement states, “The Contracting Officer will review this BPA on
    an annual basis to ensure the competitive nature of the services to be provided are consistent with
    the market place.”
According to an FSA official, FSA’s review of the Modernization Partner Agreement was limited to
assessing progress in its modernization effort. FSA’s focus was on the task orders awarded and the
forecast of initiatives that were going to be included in subsequent task orders. FSA has not
compared the pricing for deliverables under the Modernization Partner Agreement to an independent
marketing source, which would be necessary to make a determination regarding the competitive
nature of the services provided and whether the BPA continues to represent the best value.

The nature of the work being performed by the Modernization Partner reinforces the need for the
annual review. The Modernization Partner is the exclusive, prime contractor for FSA’s
modernization effort to carry out its statutory tasks of improving service, reducing costs and
integrating information systems. Under the agreement, the Modernization Partner is tasked to

•   Perform modernization management;
•   Complete and maintain the Modernization Blueprint;
•   Provide contract management;
•   Develop, integrate, test, deploy, and maintain information systems; and
•   Enforce the use of standards and technology tools.

As a result, the Modernization Partner may propose, develop, and then seek to implement a project
under the modernization effort.

The Office of Federal Procurement Policy’s Policy Letter 92-1, “Inherently Governmental
Functions,” provides instructive guidance:

       When they are aware that contractor advice, opinions, recommendations, ideas,
       reports, analyses, and other work products are to be considered in the course of their
       official duties, all Federal Government officials are to ensure that they exercise
       independent judgment and critically examine these products.

                                  * * * * * * *

       It is the policy of the Executive Branch to ensure that Government action is taken as a
       result of informed, independent judgments made by Government officials who are
       ultimate [sic] accountable to the President. When the Government uses service
       contracts, such informed, independent judgment is ensured by . . . providing greater
       scrutiny and an appropriate enhanced degree of management oversight . . . when
       contracting for functions that are not inherently governmental but closely support the
       performance of inherently governmental functions . . . .
Given the relationship of the Modernization Partner to FSA and the scope of duties under the
agreement, FSA should be providing significant scrutiny and an enhanced degree of management
oversight as it uses the services of the Modernization Partner to carry out its statutory
responsibilities. A properly conducted annual review should help FSA ensure that it receives the
best value for products and services under the Modernization Partner Agreement.

Recommendation

We recommend that the Chief Operating Officer for FSA ensure that

2.1 	   FSA conduct the required annual review of the Modernization Partner Agreement, including
        a cost and benefit and/or external market analyses, to assure that FSA receives the best value
        for products received.

Department Comments

The Department concurred with the finding and recommendation in the audit report. The
Department stated that, in September 2002, it initiated a review to determine if the agreement
continues to represent the best value for products and services. The Department also stated that
because of the breadth and complexity of the agreement, it has not determined when that review will
be completed.



Finding No. 3 – The Baseline FSA Used for Its Share-in-Savings Task Order Is
                Overstated


During the course of our review, we reported that FSA needed to use more accurate baseline
information and maintain documentation that showed that it considered alternatives to the share-in-
savings type contract. Accurate baseline information is crucial to the success of share-in-savings
contracting. We also reported that FSA did not provide documentation supporting a consideration of
alternatives to the share-in-savings task order for the Central Data System (CDS) simplification.

Under the CDS share-in-savings task order (Task Order No. 26), the Modernization Partner incurred
up-front expenses in return for a potentially higher payoff, based on how well it streamlined CDS
operations and reduced associated costs. The Modernization Partner’s performance is measured
through the reduction in system operational cost after the elimination of the CDS. The payment to
the Modernization Partner is calculated by comparing the operational costs for CDS and its migrated
functions after elimination to the baseline of the operational costs before the elimination.
We found that, even though FSA’s payments to the Modernization Partner were made in accordance
with the terms of the contract, the $20.3 million baseline used to measure the contractor’s
performance and to calculate contractor payments for the share-in-savings task order was overstated
by about $5.4 million. The overstatement in the baseline information occurred because FSA did not
use the most current information available, at the time, to establish the CDS operational cost
baseline. The established baseline did not reflect the reduction in CDS costs that resulted from
FSA’s previous decision to migrate CDS’s data to the Virtual Data Center (VDC). Since the CDS
data was migrated to the VDC in November 1999, at least five months of cost data related to the
VDC was available when the share-in-savings task order was signed in July 2000.2

The Contracting Officer for the share-in-savings task order acknowledged that the baseline did not
reflect the savings generated by the migration of CDS’s data to the VDC. The Contracting Officer
stated that FSA adjusted for the omission of the VDC savings by negotiating a reduced percentage
for the Modernization Partner’s share of the future savings. However, since FSA did not take the
available data into account, there is no assurance that the adjustment to the contractor’s percentage
of the share of savings adequately adjusted for savings that resulted from the migration of data to the
VDC.

We reported our findings to FSA during our audit and provided the recommendations that are
included in this report. FSA’s response indicated that it agreed that our recommendations represent
good business practices and that it would implement the recommendations in future share-in-savings
contracts. FSA did not dispute that the baseline was overstated but disagreed with our calculation of
the amount of the baseline overstatement. According to FSA’s response the baseline was overstated
by about $1.3 million compared to our calculation of about $5.4 million. The response stated that
the costs of continuing to provide the functionality for the CDS functions migrated to the VDC
should offset the savings from that decision. We reviewed FSA’s response and additional cost
information and see no reason to change our recommendations.

We also reported that FSA did not provide documentation supporting a consideration of alternatives
to the share-in-savings task order for the CDS simplification. FSA should have performed a cost and
benefit analysis, including an analysis of alternatives for system continuation, in-house elimination,
share-in-savings task order or fixed-price for contractor-assisted elimination, or expiration of the
existing contract. Such an analysis could have provided decision-makers with valuable information
on cost and performance to use in support for the share-in-savings task order or an alternative
selection. Though FSA responded that it had considered alternatives to the share-in-savings task
order, FSA did not state that it documented its consideration, which was consistent with our review.
FSA agreed that such analysis represents good business practices, and that it would ensure that
alternatives to share in saving arrangements are documented as part of the business case and contract
files.

2
    Because of delayed receipt of invoices, FSA generally receives the complete monthly operational cost data
    about two months after the costs are incurred. FSA had complete operational cost data up through at least
    April 2000.
Recommendations

We continue to recommend that the Chief Operating Officer for FSA ensure that

3.1 	   The most current and relevant information is used to develop baselines used to determine
        payments under share-in-savings task orders and contracts.

3.2 	   Share-in-savings task orders and contracts provide a mechanism for adjustment (such as a
        contract clause), when the established baseline is found to be substantially different than the
        actual baseline.

3.3 	   Documentation is maintained in the contract file to demonstrate that FSA considered
        alternatives to share-in-savings type contracts.

Department Comments

The Department concurred with the finding and recommendations in the audit report. Although the
Department agreed that the baseline was overstated, it did not agree with our calculation of the
amount of the overstatement. However, the Department stated that rather than debate the issue it
would like to implement a process by which the OIG is involved in the negotiations of future share-
in-savings contracts, to help ensure an accurate baseline calculation and avoid possible
overstatements.

OIG Response

We commend the Department in its desire to improve its process for share-in-savings and will
evaluate any request for assistance in terms of our workload and independence standards.
                 Audit of FSA's Modernization Partner Agreement 



                                          Other Matters 



During the course of our review, we reported a system control weakness over the obligation of
funds. Specifically, contracting officers were able to obligate contract funds in excess of their
delegated authority. In addition, we have some general concerns about the legality of issuing share-
in-savings task orders under the Modernization Partner Agreement.


Inadequate Controls over the Obligation of Funds

During our audit, we discovered that contracting officers were able to override the Department’s
Contracting and Purchasing Support System (CPSS) controls and obligate contract funds in excess
of their delegated authority. We found that contracting officers provided a justification statement on
the system and were permitted to obligate funds in excess of their delegated authority. Although
CPSS can generate a report of contracting officers’ obligations exceeding the delegated authority,
the Department was experiencing problems in generating this report.

Under FAR § 1.602-1(a), “Contracting officers may bind the Government only to the extent of the
authority delegated to them.” The ability to bypass a system control can render that control
ineffective and allow a contracting officer to potentially bind the government to obligations over the
contracting officer’s limit of authority. As a result, the Department does not have the necessary
management controls to safeguard contract funds, and contracting officers could be liable for
obligations authorized in excess of their authority. We reported this issue to FSA in May 2001.

Department Comments

The Department concurred with the finding and stated that, to monitor and prevent future
occurrences of this type, the Department’s Contracts and Purchasing Operations (CPO) is
performing periodic reviews of procurement obligations to ensure that procurement obligations are
made by Department officials within their delegated warrant authority.

OIG Response

Although we did not test CPO’s new monitoring efforts, they should address our concerns for
strengthening controls over the obligation of funds.
Share-in-Savings Contract Pricing Conflict with the Modernization Partner Agreement

Pricing for the share-in-savings contract appears to be in conflict with the hourly rate schedule
included in the Modernization Partner Agreement. Under a share-in-savings contract, the
Modernization Partner incurs up-front expenses in return for a potentially higher payoff, calculated
as a percentage share of the savings resulting from the contractor’s work. However, the
Modernization Partner Agreement uses GSA’s schedule of contractors and provides for the pricing
of tasks using an hourly rate. While the GSA contract also authorizes and encourages the use of
fixed price or incentive prices, the contract also says that these alternatives are to be based on the
scheduled labor rates. As such, we will evaluate the use of share-in-savings pricing under the GSA
Schedule 70 contract in a subsequent audit.
                     Audit of FSA's Modernization Partner Agreement 



                               Objectives, Scope, and Methodology 



The objectives of our audit were to

• 	 Evaluate the performance indicators established to measure contractor performance to determine
    the adequacy of those measures in assessing whether contractor performance is in accordance
    with contract terms.

• 	 Assess the Modernization Partner Agreement’s compliance with laws and regulations.

• 	 Determine the accuracy and reasonableness3 of the costs incurred for the products and services
    provided.

To accomplish our objectives, we reviewed applicable FSA policies and procedures, as well as laws,
regulations, and agency guidelines addressing system modernization and contracting. We reviewed
prior OIG audit reports, along with General Accounting Office reports, applicable to this area. We
obtained and reviewed FSA’s Modernization Blueprint, the BPA for FSA’s Modernization Partner
(Modernization Partner Agreement), and task orders issued under the BPA. Our audit focused on
the objectives listed above, but at this time, we plan to perform additional work and to issue
subsequent audit reports concerning FSA’s Modernization Partner Agreement. This includes a
review of the legality of issuing share-in-savings task orders under the Modernization Partner
Agreement given the mechanism used for pricing – a percentage share of the resulting savings as
opposed to the Modernization Partner Agreement’s pricing schedule.

To assess whether performance indicators for measuring contractor performance were included, we
reviewed a judgmental sample of 47 task orders from the 67 task orders in effect as of May 2001.
The 47 task orders reviewed provided for 452 deliverables under the BPA. The specific task orders
reviewed were task orders 1 through 23, 25, 26, 28 through 35, 37, 38, 39, 41, 42, 43, 44, 48, 49, 50,
52, 57, 58, and 67. We did not review other task orders because they either had not been issued or

3
    Although our initial objectives included addressing the reasonableness of costs incurred, we did not conduct
    tests related to that portion of the objective because it would have required a subjective comparison of like
    products and/or services between contractors. Therefore, we did not determine the reasonableness of
    contractor prices for the products and/or services included in the task order, only that the costs were within
    the terms of the contract and that contractor invoices were supported by appropriate documentation,
    reviewed, and approved.
were being modified at the time of our request. For the 47 judgmentally selected task orders, we
reviewed invoices for accuracy of payments made to Accenture, analyzing supporting
documentation for the invoices, and ensuring that the invoices were consistent with the authorizing
task order.

We analyzed baseline cost data and assumptions used to support the share-in-savings task order for
Central Data System simplification. We interviewed personnel from various areas within FSA,
including the Contracting Officer and Contracting Officer Representative for the BPA with
Accenture, as well as, personnel in FSA Channels (business units), FSA’s Office of Chief Financial
Officer, and the Department’s Contracts and Purchasing Operations.

We conducted work at FSA’s offices in Washington, DC, and our OIG office in Kansas City, MO,
during the period January 2001 to February 2002. During our audit, we issued two Action
Memorandums to FSA:

• 	 SFA Action Memorandum No. 01-05, dated May 31, 2001, reported that contracting officers
    were able to obligate contract funds in excess of their delegated authority and to override the
    Contracting and Purchasing Support System (CPSS) controls.

• 	 SFA Action Memorandum No. 01-06, dated August 1, 2001, reported that FSA needed to use
    more accurate baseline information and maintain documentation that showed that it considered
    alternatives to the share-in-savings type contract.

We held an exit conference with FSA officials on September 17, 2002. Our audit was performed in
accordance with generally accepted government auditing standards appropriate to the scope of the
review described above.
                 Audit of FSA's Modernization Partner Agreement 



                          Statement on Management Controls 



As part of our review, we gained an understanding of the Department’s management control
structure over the Modernization Partner Agreement. We also reviewed the policies, procedures,
and practices applicable to the scope of the review. For the purpose of this report, we assessed and
classified the significant management controls related to the Department’s Modernization Partner
Agreement into the following categories:

       •   Award of task orders
       •   Acceptance of deliverables
       •   Contract payments
       •   Compliance with laws and regulations
       •   Procedures to evaluate contract performance

Because of inherent limitations, a study and evaluation made for the limited purpose described above
would not necessarily disclose all material weaknesses in the management controls. However, our
assessment identified management control weaknesses as set out in the Audit Results and Other
Matters sections of this report.
              APPENDIX: Auditee Comments on the Draft Report




ED-OIG/A07-B0008                                               Page 18 

                UNITED STATES DEPARTMENT OF EDUCATION 



                                                    NOV - 6"j , 




MEMORANDUM 



TO:                       Thomas A. Carter
                          Assistant Inspector General for Audit Services

FROM:            ~l'1 Theresa S. Shaw       -~                                    r?~
                 {V ­ Chief Operating Officer  '/
                      Federal Student Aid

                           Jack Martin                      Lj;}~/)Li~
                           Chief Financial Officer         . '/
                           Office of the Chief Fina ialOfficer

SUBJECT:                  Response to Draft Audit Report-Audit ofFSA 's Modernization
                          Partner Agreement, ACN A07-B0008

Thank you for providing us with an opportunity to review and respond to your draft report
entitled, Audit ofFSA 's Modernization Partner Agreement (Audit Control Number A07-B0008).

We were pleased that your draft audit report states that the contract is generally being
administered in accordance with federal laws and regulations, and that FSA's payments to the
Modernization Partner have been accurate and in accordance with the terms of the authorizing
task orders. However, your report also states that your audit found the following: I) the task
orders included inadequate performance measures; 2) FSA had not reviewed the Modernization
Partner Agreement to ensure that FSA receives the best value; and 3) that the baseline FSA used
for the Central Data System (CDS) share-in-savings task order was overstated.

As you know, FSA previously responded to the recommendations your office made during the
course of the audit regarding the use of the most current and relevant information to develop
baselines in determining payments under share-in-savings task orders and contracts. While we
agree the baseline used was overstated, we cannot agree with your calculation to indicate by what
amount. However, rather than continue to debate the amount by which the baseline was
overstated, we would like to implement a process by which the OIG is involved in the
negotiations of our future share-in-savings contracts to help ensure an accurate baseline
calculation, and avoid possible overstatements.




         Our mission is to ensure equal access to education and to promote educational excetlence throughout the nation.
In addition, FSA is now contracting services from independent consulting firms to conduct
independent reviews of baseline information to further ensure that the baselines developed for
future modernization projects are reasonable and accurate.

The concept of share-in-savings contracts is a new and innovative approach to government
contracting that provides the government with increased flexibilities, and the potential for
tremendous opportunities for savings. The Department is one of the first government agencies to
experiment with this type of contract. Negotiating these types of contracts can be challenging
because they are so new to the government's way of doing business, and require explicit and
measurable beginning baselines and target performance criteria to establish that savings have, in
fact, been realized. We acknowledge that FSA is learning valuable lessons as it works to refine
the contract process for future negotiations. However, we are pleased to report that we have
implemented, or are well on our way to implementing, many of the recommendations contained
in this audit report.

Enclosure 1 contains the Department's response to each finding. Again, we appreciate the
opportunity to review and comment on the draft report.




Enclosures

cc: 	 Pat Howard, OIG
      Glenn Perry, OCFO
      Candace Hardesty, FSA
      Victoria Bateman, FSA




                                                   2

Audit ofFSA's Modernization Partner Agreement
(A07-B0008)


Finding No.1 - Task Orders Include Inadequate Performance Measures

We concur with this finding and recommendations, with one minor exception: We
believe that Recommendation 1.1 should be amended as follows: As appropriate, test the
performance, reliability, and usability ofall deliverables. The recommendations are
intended to address system development and maintenance deliverables, but they don't
apply to other types of de1iverab1es. Therefore, "all de1iverab1es" does not apply. For
example, how would we measure the "performance" of a status report? Why would we
want to evaluate the contractor's performance throughout the performance period if
he/she was only preparing a report or a plan for us?

FSA recognizes that it needs to develop performance measures that focus on outcomes.
We have implemented, or plan to implement, the following actions to develop effective
and quantifiable performance measures:

    1. 	 Acquisitions and Contracts Performance staff attended a three-day training
         session on October 1,3, and 4 on the "Seven Steps to Performance Based
         Acquisition. "

    2. 	 The project manager will attend a training program this fall on performance­
         based contracting.

    3. 	 We will amend the Accenture Blanket Purchase Agreement to require the use of
         FSA's Solution Lifecycle process on all applicable task orders. The Solution
         Lifecycle process has entrance and exit criteria for each phase of system
         development.

    4. 	 We will require performance metrics in all new proposals.

    5. 	 We will measure performance throughout the contract to further improve results.

Finding No.2 -FSA Has Not Reviewed the Modernization Partner Agreement To
Ensure that FSA Receives the Best Value

We concur with this finding, and the recommendation that FSA conduct an annual review
ofthe Modernization Partner Agreement, including a cost and benefit and/or external
market analyses, to assure that FSA receives the best value for products received.

In September 2002, FSA began a review of the Blanket Purchase Agreement (BPA) to
determine if the Accenture agreement continues to represent the best value for products
and services. The breadth and complexity of the agreement through the tasks which have
been issued to date are vast and while this review will be done with urgency, it has not
Audit of FSA' s Modernization Partner Agreement
(A07-B0008)

yet been determined when that effort will be completed. However, prior to the award of
any future task orders under the BP A, more comprehensive efforts will be undertaken to
obtain services that are of best value to the Department and FSA.

Finding No. 3 -The Baseline FSA Used for Its Share-in-Savings Task Order Is
Overstated

As we mentioned during the audit, we concur with this finding and the recommendations,
but cannot come to agreement with the amount overstated. However, rather than
continue to debate the amount by which the baseline was overstated, we would like to
propose and implement a process for moving forward on future share-in-savings
contracts.

For all future share-in-savings contracts, we would like to involve the OIG at the
beginning of negotiations to review the calculation to help ensure an accurate baseline
and avoid possible overstatements.

Other Matters--Inadequate Controls over the Obligations of Funds

The OCFO Contracts and Purchasing Operations (CPO) Director discussed the incident
with the FSA contracting officer. The contracting officer was advised that she had
exceeded her authority and that she was not authorized to do so. To monitor for the
prevention of future occurrences of this type, CPO performs periodic reviews of
procurement obligations recorded in EDCAPS to ensure that procurement obligations are
made by Department officials within their delegated warrant authority. CPO will also
continue to review individual authorities periodically to ensure that they are in
accordance with the requirements of FAR Subparts 1. 6-Career Development,
Contracting Authority, and Responsibilities.
                   REPORT DISTRIBUTION SCHEDULE 

                          Audit Control Number A07-B0008 


Action Officials                                                   No. of Copies

Theresa S. Shaw                                                           1
Chief Operating Officer
Federal Student Aid
U.S. Department of Education
830 1st Street, N.E.
Washington, DC 20202

Jack Martin                                                               1
Chief Financial Officer
FOB 6 Room 4E313
400 Maryland Avenue SW
Washington, DC 20202

Other ED Officials/Staff (electronic copy)

William D. Hansen, Deputy Secretary                                       1
John Danielson, Chief of Staff                                            1
Eugene Hickok, Under Secretary                                            1
Craig B. Luigart, Chief Information Officer                               1
Stephen Hawald, Chief Information Officer, FSA                            1
Dawn Dawson, FSA, Audit Liaison Officer                                   1
Kent Talbert, Deputy General Counsel, Office of General Counsel           1
Thomas P. Skelly, Director, Budget Service                                1
John Gibbons, Director, Communications                                    1
Clay Boothby, Acting DAS, Legislation and Congressional Affairs           1
Laurie M. Rich, AS, Intergovernmental and Interagency Affairs             1
Philip Maestri, Director, Fin’l Improve. & Post Audit Opns, OCFO          1
Michelle Douglas and Carolyn Adams, OGC (Correspondence Control)          1
L’Wanda Rosemond, General Operations Team                                 1
Charles Miller, Post Audit Group, OCFO                                    1
Headquarters and Regional Audit Managers                                  1 each