oversight

Department's Processes for Validating the EDUCATE Contractor's Performance

Published by the Department of Education, Office of Inspector General on 2011-05-31.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

     Department’s Processes for Validating the EDUCATE 

                 Contractor’s Performance





                                 FINAL AUDIT REPORT





                                    ED-OIG/A19K0007

                                        May 2011



Our mission is to promote the                          U.S. Department of Education
efficiency, effectiveness, and                         Office of Inspector General
integrity of the Department's                          Washington, DC
programs and operations.
                     NOTICE
Statements that managerial practices need improvements, as well as
other conclusions and recommendations in this report represent the
opinions of the Office of Inspector General. Determinations of
corrective action to be taken will be made by the appropriate
Department of Education officials.

In accordance with the Freedom of Information Act (5 U.S.C. § 552),
reports issued by the Office of Inspector General are available to
members of the press and general public to the extent information
contained therein is not subject to exemptions in the Act.
                                    UNITED STATES DEPARTMENT OF EDUCATION
                                           OFFICE OF INSPECTOR GENERAL

                                                                                                                     AUDIT SERVICES
                                                            May 31, 2011

Memorandum
TO:                  Danny Harris
                     Chief Information Officer
                     Office of the Chief Information Officer

                     Thomas Skelly
                     Delegated to Perform Functions and Duties of the Chief Financial Officer
                     Office of the Chief Financial Officer

FROM:                Keith West /s/
                     Assistant Inspector General for Audit

SUBJECT:             Final Audit Report
                     Department’s Processes for Validating the EDUCATE Contractor’s Performance
                     Control Number ED-OIG/A19K0007

Attached is the final audit report that covers the results of our review to determine whether the
Department had adequate controls in place to validate the EDUCATE contractor’s performance
prior to authorizing payment of invoices. An electronic copy has been provided to your Audit
Liaison Officers (ALO). We received the combined comments from the Office of the Chief
Information Officer and Office of the Chief Financial Officer and the corrective action plan for
each of the recommendations in our draft report.

Corrective actions proposed (resolution phase) and implemented (closure phase) by your office
will be monitored and tracked through the Department’s Audit Accountability and Resolution
Tracking System (AARTS). Department policy requires that you develop a final corrective
action plan (CAP) for our review in the automated system within 30 days of the issuance of this
report. The CAP should set forth the specific action items and targeted completion dates
necessary to implement final corrective actions on the findings and recommendations contained
in this final audit report.

In accordance with the Inspector General Act of 1978, as amended, the Office of Inspector
General is required to report to Congress twice a year on the audits that remain unresolved after
6 months from the date of issuance.

In accordance with the Freedom of Information Act (5 U.S.C. §552), reports issued by the Office
of Inspector General are available to members of the press and general public to the extent
information contained therein is not subject to exemptions in the Act.


 The Department of Education's mission is to promote student achievement and preparation for global competitiveness by fostering educational
                                                   excellence and ensuring equal access.
Messrs. Harris and Skelly
Page 2 of 2

We appreciate the cooperation given to us during this review. If you have any questions, please
call Michele Weaver-Dugan at (202) 245-6941.

Enclosure

cc:    Deborah Coleman, ALO, Office of the Chief Information Officer
       Roscoe Price, ALO, Office of the Chief Financial Officer, Contracts and Acquisition
          Management
                                              TABLE OF CONTENTS



                                                                                                                                 Page

EXECUTIVE SUMMARY ...........................................................................................................1


BACKGROUND ............................................................................................................................4


AUDIT RESULTS .........................................................................................................................6


           FINDING NO. 1 – The Department Needs to Improve Processes for 

                           Validation of SLA Performance ....................................................6


           FINDING NO. 2 – The Department Needs to Ensure SLAMs Have
                           Appropriate Technical Expertise to Monitor Assigned
                           EDUCATE Contract SLAs ...........................................................10


           FINDING NO. 3 – The Department Needs to Improve Processes for 

                           Validation of Chargeback Reports ...............................................13


           FINDING NO. 4 – The Department Needs to Modify its SLA Framework
                           to Effectively Encourage Improvements in
                           Contractor Performance ...............................................................16


OBJECTIVE, SCOPE, AND METHODOLOGY ....................................................................20


ENCLOSURE 1 – Summary of Selected SLAs


ENCLOSURE 2 – Acronyms/Abbreviations Used in this Report

ENCLOSURE 3 – Department Response to Draft Report
Final Audit Report
ED-OIG/A19K0007                                                                       Page 1 of 22



                                        EXECUTIVE SUMMARY



The Education Department Utility for Communications, Applications, and Technology
Environment (EDUCATE) contract established a Contractor-Owned Contractor-Operated
Information Technology (IT) service model for the Department of Education (Department) under
which the EDUCATE contractor provides the total IT platform and infrastructure to support
Department employees in meeting the Department’s mission. The contract was awarded in
September 2007 as a 10-year, performance-based, indefinite delivery/indefinite quantity contract
with fixed unit prices.

The Department monitors and evaluates the contractor-provided IT services through a Service
Level Agreement (SLA) 1 framework that applies incentives or disincentives if performance falls
above or below established standards. The EDUCATE contractor bills the Department monthly
for the provided services through the use of chargeback reports, which list unit-based hardware
and software profiles for end users, as well as charges for help desk support services, e-mail
services, telecommunications usage, and printers. Data from the Department’s Contract and
Purchasing Support System indicated the EDUCATE contractor had been paid $134 million
from November 2007 through December 2010.

The objective of our audit was to determine whether the Department had adequate controls in
place to validate the EDUCATE contractor’s performance prior to authorizing payment of
invoices. Overall, our audit found that the Department does not have adequate controls in place
for validating contractor performance. Specifically, we found that the Department’s validation
processes do not provide independent assurance of contractor performance or assurance of the
quality of the data being relied upon to assess performance. In addition, the Department has not
always assigned personnel with the appropriate qualifications to validate contractor-submitted
SLA performance data. We also determined the Department does not always use independent,
accurate, or complete data to validate contractor-prepared chargeback reports, provides limited
time for review of these reports by affected parties, and duplicates effort during the chargeback
report validation process. Finally, we found that the current SLA framework is ineffective in
encouraging the EDUCATE contractor to improve performance. As a result, the Department
does not have assurance that the EDUCATE contractor is performing as required, will improve
performance when necessary, and is being paid appropriately for the level of service provided.

To correct the weaknesses identified, we recommend that the Chief Information Officer and
Chief Financial Officer, among other things:

       •	 Review all SLAs and identify possible sources of independent supporting data to be used
          for SLA performance validation.
       •	 Implement procedures to periodically test underlying performance data in the contractor’s
          systems for accuracy, especially data that is being relied upon for SLA validation
          purposes.


1
    SLAs formally define the contractor’s required level of performance.
Final Audit Report
ED-OIG/A19K0007                                                                       Page 2 of 22

   •	 Formally establish and implement validation procedures, to include the identification of
      appropriate supporting documentation to be used for validation, for each SLA.
   •	 Identify the necessary knowledge, skills, and abilities required for oversight of each SLA
      and staff positions accordingly.
   •	 Determine whether an independent data source is available for use in validating

      chargeback reports.

   •	 Establish and implement written procedures to reflect the intended chargeback report
      validation process. The procedures should reflect the current IT environment, identify
      roles and responsibilities of officials and offices involved in the chargeback report
      validation process, ensure sufficient time is available for effective review of chargeback
      reports, and clearly define data sources to be used for validation purposes.
   •	 Reevaluate the allocation of incentives and disincentives to ensure they are appropriate
      for achieving the desired results of the SLA framework, in accordance with guidelines
      and best practices for performance-based contracting.
   •	 Establish and implement guidelines for routine reviews and evaluations of the

      appropriateness of the allocation of incentives and disincentives.


In its response to the draft audit report, the Office of the Chief Information Officer (OCIO) and
Office of the Chief Financial Officer (OCFO) concurred with 12 of the 14 recommendations.
OCIO/OCFO did not concur with draft recommendation 1.4 related to duplicative validation
tasks performed by the Independent Verification and Validation (IV&V) contractor and Service
Level Agreement Monitors (SLAMs). OCIO/OCFO noted that duplication was intended.
OCIO/OCFO also did not concur with draft recommendation 3.1 related to the identification of
independent data sources for validation of chargeback reports, stating that the recommendation
would create a redundant source of chargeback data and would be contrary to good business
practices. In response to the Department’s comments, we have slightly modified
recommendation 1.4. In addition, we noted that the Department’s response to draft
recommendation 4.2 was inconsistent with what was intended by the recommendation. As a
result, we have slightly modified recommendation 4.2 to more clearly define its intent.

Overall, OCIO/OCFO stated that the report provided valuable insight and accurately identified
several areas of needed improvement, but also stated that limited consideration was given to
OCIO/OCFO efforts made during the past year to address specific issues presented in the report.
During our fieldwork, we engaged in many discussions with applicable Department officials and
staff, to include senior OCIO management, SLAMs, Contracts and Acquisition Management
representatives, Program Office staff with responsibility for chargeback report validation, the
Chargeback Program Manager, as well as IV&V contractor representatives, to gain an
understanding of the Department’s processes for validating contractor performance and
chargeback reports. We also prepared process flow charts that were reviewed by appropriate
OCIO staff, reviewed applicable documented processes, and reviewed corrective actions taken in
response to prior related Office of Inspector General (OIG) work. While the scope of our review
covered the period November 2007 through March 2010, we also inquired about any changes
OCIO made to validation processes between April 2010 and December 2010 when we conducted
our fieldwork.
Final Audit Report
ED-OIG/A19K0007                                                                     Page 3 of 22

Subsequent to fieldwork completion, OIG solicited additional comments and information from
OCIO in response to our proposed findings and recommendations. During the exit conference
conducted on December 7, 2010, OCIO officials expressed their belief that the necessary
processes to validate SLA performance were in place and the chargeback process had been
revamped within the past year. In a related discussion, OIG requested that OCIO provide any
specific additional information that should be considered in preparation of our draft report. In
response, OCIO officials stated they would provide information relating to significant process
updates. However, on December 12, 2010, OCIO officials responded via email that they had no
additional comments at that time.

Other than the slight modifications to draft recommendations 1.4 and 4.2 previously mentioned,
we have not made any additional changes to our findings or recommendations. OCIO/OCFO’s
comments are summarized at the end of each applicable finding. The full text of OCIO/OCFO’s
response is included as Enclosure 3 to this report.
Final Audit Report
ED-OIG/A19K0007                                                                                 Page 4 of 22



                                           BACKGROUND



The Education Department Utility for Communications, Applications, and Technology
Environment (EDUCATE) contract established a Contractor-Owned Contractor-Operated
(COCO) Information Technology (IT) service model for the Department of Education
(Department) under which the EDUCATE contractor provides the total IT platform and
infrastructure to support Department employees in meeting the Department’s mission. This
includes items such as desktop and printer services, help desk support, data center operations,
e-mail, network, disaster recovery, and other special services. The contract was awarded in
September 2007 as a 10-year, performance-based, indefinite delivery/indefinite quantity contract
with fixed unit prices.

The Department monitors and evaluates the contractor-provided IT services through a Service
Level Agreement (SLA) framework that applies incentives or disincentives if performance falls
above or below established standards. The EDUCATE contractor bills the Department monthly
for the provided services through the use of chargeback reports, which list unit-based hardware
and software profiles for end users, as well as charges for help desk support services, e-mail
services, telecommunications usage, and printers. All charges, including incentives and
disincentives, are reflected in the monthly invoices provided to the Department. Data from the
Department’s Contract and Purchasing Support System (CPSS) indicated the EDUCATE
contractor had been paid $134 million from November 2007 through December 2010.

SLA Validation

The Department’s process to validate SLA performance includes multiple Department officials
and the support of an Independent Verification & Validation (IV&V) contractor. The
Department’s Contract Line Item Number (CLIN) owners are responsible for evaluating the
distinct services and deliverables under the EDUCATE contract. Within this role, CLIN owners
oversee performance of all of the SLAs within their CLIN. 2 To assist CLIN owners with their
responsibilities, the Department uses Service Level Agreement Monitors (SLAMs) to monitor
the contractor’s performance under 36 3 SLAs on a daily basis. The SLAMs’ responsibilities
include validating monthly reports for their respective SLAs and determining whether or not the
EDUCATE contractor met or missed the SLAs’ performance standard for the month. In some
instances a CLIN owner may also serve as a SLAM. The total number of SLAMs varied during
our fieldwork; however, as of April 2010 there were nine SLAMs.

The IV&V contractor’s responsibilities include assisting the Department in reviewing the
EDUCATE contractor’s performance. Within this role, the IV&V contractor conducts
independent reviews of the SLA reports and analyzes the reasonableness of corresponding
supporting documentation. The IV&V contractor submits monthly reports summarizing the

2
  EDUCATE CLINs consist of: Desktop Services; Helpdesk Support Services; Systems/Data Center Services;

E-mail Services; Network, Telecommunication, and Multimedia Services; Disaster Recovery Services; Special

Services; and Printer Services.

3
  Total number of SLAs as of March 1, 2009.

Final Audit Report
ED-OIG/A19K0007                                                                      Page 5 of 22

EDUCATE contractor’s performance during the prior month to the Department’s contract
administration team.

SLA performance data originates primarily from three data sources. The first source is the
Operational Process Application Suite System (OPAS), which is the EDUCATE contractor’s
system for creating, tracking, and retaining help desk tickets created from customer calls. The
second source is from network and security management tools used to monitor and administer
the network and track intrusions or vulnerabilities. The third source is customer satisfaction
surveys completed by Department users.

Chargeback Validation

The EDUCATE contractor bills the Department monthly through the use of chargeback reports,
which list unit-based hardware and software profiles for end users, as well as charges for help
desk support services, e-mail services, telecommunications usage, and printers. The chargeback
reports are the basis for the unit-based costs on the monthly invoices. The chargeback approach
is designed to ensure that cost allocations are distributed to Principal Offices (PO) based on
usage.

On the 1st business day of the month, the EDUCATE contractor extracts data from its systems
related to all usage-based items and services from the prior month. The EDUCATE contractor
has 2 weeks to prepare and edit the chargeback reports for delivery to the Department. By the
16th day of each month, the EDUCATE contractor provides the chargeback reports, listing the IT
assets assigned to each PO, to the Department’s Chargeback Program Manager (PM).

The Chargeback Program Management Office (PMO), consisting of 3 SLAMs, conducts a
preliminary review of the chargeback reports for the Department’s 24 POs. The PMO follows a
27-point checklist developed by the Department to validate the chargeback reports. The PMO
has 5 calendar days to review the reports and provide feedback to the Chargeback PM.
According to the Chargeback PM, she reviews the PMO comments and performs a final cursory
review of the reports for additional errors. The Chargeback PM sends feedback on the reports
back to the EDUCATE contractor as needed.

According to the Chargeback PM, once all changes have been made, she notifies the Contracting
Officer’s Representative (COR) to accept the chargeback reports as a deliverable. The
Chargeback PM distributes the applicable chargeback reports to the assigned Principal Office
Coordinators (POCs) via e-mail on the 20th day of the month. The POCs have 3 business days to
review the chargeback reports and provide requests for changes through OPAS. Corrections
identified by the POC reviews are reflected in the subsequent month’s reports as long as they are
received by the EDUCATE contractor within the 3rd business day after the 20th of the month.
Final Audit Report
ED-OIG/A19K0007                                                                                          Page 6 of 22



                                               AUDIT RESULTS



Overall, our audit found that the Department does not have adequate controls in place to validate
the EDUCATE contractor’s performance. We found that the Department’s processes do not
provide independent assurance of contractor performance or assurance of the quality of the data
being relied upon to assess performance. In addition, the Department has not always assigned
personnel with the appropriate qualifications to validate contractor-submitted SLA performance
data. We also found the Department needs to improve processes for validation of chargeback
reports of unit-based expenses. Specifically, the Department does not always use independent,
accurate, or complete data to validate chargeback reports, provides limited time for affected
parties to review the reports, and duplicates effort during the validation process. Finally, we
found that the current SLA framework is ineffective in encouraging the EDUCATE contractor to
improve performance. As a result, the Department does not have assurance that the EDUCATE
contractor is performing as required, will improve performance when necessary, and is being
paid appropriately for the level of service provided.


FINDING NO. 1 - The Department Needs to Improve Processes for Validation of
                Service Level Agreement Performance.

The Department does not have adequate processes in place to validate SLA performance.
Specifically, the Department’s validation processes do not provide independent assurance of
contractor performance or assurance of the quality of the data being relied upon to assess
performance.

Lack of Independent Assurance

We noted that the Department relies on the EDUCATE contractor’s data and contractor-
developed queries 4 in its SLA validation processes. We reviewed the processes used by 6 of 9
SLAMs (66.7 percent) responsible for validation of 7 of 36 SLAs (19 percent) to validate the
completeness of the contractor’s SLA reports and to follow-up on identified anomalies. 5 Our
review revealed a lack of independence within the processes. Contractor data is essentially being
used to validate contractor data, as follows:

           •	 The six SLAMs stated that any anomalies within the contractor’s SLA reports are
              researched using the corresponding contractor data.
           •	 Three of the six SLAMs stated that they generate their own reports from the
              contractor’s OPAS using contractor-provided queries. The SLAMs then compare the
              results of the queries to the contractor’s SLA reports to determine if any differences
              exist.



4
    A query is used to extract information from a database in a readable format according to the user’s request.
5
    See Enclosure 1 for summary of selected SLAs.
Final Audit Report
ED-OIG/A19K0007                                                                             Page 7 of 22

We found that the IV&V contractor processes appear to be duplicative of those performed by the
selected Department SLAMs. Specifically, an IV&V official stated that they generate reports
from the EDUCATE contractor’s OPAS for ticket-based SLAs. The reports are then compared
to those developed by the EDUCATE contractor and provided to the SLAMs, similar to the
process employed by at least three of the SLAMs noted above.

Lack of Assurance of Data Quality

The data sources for the seven SLAs we reviewed included four OPAS-based SLAs, two system
tool-based SLAs, and one survey-based SLA. Overall we found the processes implemented by
the Department in its performance validation for each of the seven SLAs did not include steps to
gain assurance that the data used as a basis for performance reporting from any of the three data
sources were accurate.

We reviewed other activities performed by the Department that may have provided assurance of
the quality of performance data. This included the Department’s response to a prior Office of
Inspector General (OIG) report and work conducted by its IV&V contractor. We determined
neither of these activities provided assurance of the accuracy of the EDUCATE contractor’s
performance data, as further described below.

The Office of the Chief Information Officer’s (OCIO) corrective actions taken in response to
ED-OIG/L19K0001 6 included: 1) assessing the data elements used for data collection for each
SLA to assure accuracy; and 2) assessing the accuracy of the underlying data. With regard to the
first item, according to Department officials, the Department examined OPAS queries used by
SLAMs and concluded the queries provided data necessary for SLAMs to assess SLA
performance. It did not determine whether other data elements should also have been included.
With respect to the second item, OCIO officials stated they did not look at the accuracy of data
but rather reviewed sampled tickets to ensure they contained sufficient detail for use in
performance evaluation. In response to an OIG request, OCIO was unable to provide the
documentation relating to either of the above processes, to include items such as the number and
types of records selected, the analysis conducted, and related results.

The IV&V contract applicable to EDUCATE requires a weekly review of the quality of help
desk tickets. To perform this review, the IV&V contractor samples 150 closed help desk tickets
and reviews related data based on criteria from the EDUCATE contractor’s Helpdesk
Procedures. The IV&V review consists of nine criteria and includes checks of items such as key
fields not being left blank, appropriate use of the incident description field, and confirmation of
closure with the customer. We noted that between April 2009 and March 2010, the IV&V
contractor documented problems with the overall quality of help desk tickets stored in OPAS.
For example, the March 2010 IV&V report identified data quality problems with 49 of the 150
tickets sampled (33 percent). Specific problem areas included:

            •	 Sampled tickets (25 percent) did not contain enough information to show how or
               if the problem was resolved; and


6
 Untimely Resolution of Issues Impacting Performance Validation and Payment Calculations Under the EDUCATE
Contract (L19K0001), dated March 9, 2010.
Final Audit Report
ED-OIG/A19K0007                                                                        Page 8 of 22

              •	 Sampled tickets (37 percent) did not document if the user was satisfied with the
                 results and agreed the ticket could be closed.

As part of the corrective action in response to the aforementioned OIG report, OCIO officials
also indicated they attempted to independently replicate the results of the contractor-provided
queries. However, these officials indicated that they were unable to develop queries that
returned the same results as those of the EDUCATE contractor.

In conducting this audit, OIG also attempted to independently replicate the results of the
contractor-provided queries. We generated a report for 1 of the 7 selected SLAs (14 percent)
using the query generated by the EDUCATE contractor for the March 2010 reporting period. 7
We found that our report included 7,077 items compared to the 3,514 items in the contractor’s
SLA report. We followed-up on this discrepancy with the responsible SLAM, who stated the
contractor’s query extracts OPAS data that should be attributed to other SLAs and that the
contractor manually revises the data returned by the query to prepare the final SLA report. The
SLAM stated he relies on the contractor’s finalized report for validation because the task of
replicating the contractor’s processes are too complicated and time consuming.

The Government Accountability Office (GAO) “Standards for Internal Control in the Federal
Government” states that internal control is an integral component of an organization’s
management that provides reasonable assurance that objectives are being achieved, to include the
effectiveness and efficiency of operations. It also serves as the first line of defense in
safeguarding assets and preventing and detecting errors and fraud, and helps government
program managers achieve desired results through effective stewardship of public resources.

The EDUCATE Contract Monitoring Plan states that contractor performance is monitored
through the established SLA framework. We found that the SLA framework establishes the SLA
owner, the frequency at which data is captured, and the source of the data supporting the SLA
performance. However, the framework does not require processes to validate the accuracy of the
performance data, take into consideration the independence of the data sources being used to
validate performance, or specify validation procedures. The Department’s SLAMs subsequently
developed their own procedures relating to SLA validation and did not include processes to test
data accuracy. In related discussions during our audit, SLAMs did not appear to be concerned
with the lack of independent data.

The SLA validation process also lacked adequate management oversight from the inception of
the contract. While performance under the EDUCATE contract began in November 2007,
Department management did not begin examining the validation process until December 2009,
after approximately $84 million had already been paid to the EDUCATE contractor. However,
as noted above, these efforts were unsuccessful in establishing data quality and did not address
the issue of independent validation sources.

By not obtaining independent supporting data to validate the contractor’s performance, the
Department does not have assurance that the EDUCATE contractor is accurately reporting on its
performance and is receiving proper payment for the level of performance provided. Due to the
inherent conflict that exists with the EDUCATE contractor reporting out on its own performance,

7
    GN-5, Time to Resolve
Final Audit Report
ED-OIG/A19K0007                                                                             Page 9 of 22

the Department should obtain independent assurance from outside sources whenever possible.
Although issues with OPAS data quality have been identified, SLAMs continue to rely on this
data as the sole source of supporting documentation for certain SLA validation processes.
Duplicative processes performed by SLAMs and the IV&V contractor result in the inefficient
utilization of human resources.

Recommendations

We recommend that the Chief Information Officer (CIO) and Chief Financial Officer (CFO)

1.1	    Review all SLAs and identify possible sources of independent supporting data to be used
        for SLA performance validation.

1.2	    Implement procedures to periodically test underlying performance data in the contractor’s
        systems for accuracy, especially data that is being relied upon for SLA validation
        purposes.

1.3	    Formally establish and implement validation procedures, to include the identification of
        appropriate supporting documentation to be used for validation, for each SLA.

1.4	    Review SLA validation tasks performed by the IV&V contractor against those performed
        by SLAMs and eliminate identified duplicative efforts, where appropriate, to ensure the
        most effective use of resources.

Department Comments

OCIO and OCFO generally concurred with the recommendations and provided a corrective
action plan for each, with the exception of draft recommendation 1.4, related to duplicative
validation tasks performed by the IV&V contractor and SLAMs. OCIO noted the necessity of
independence of the IV&V contractor and that duplication is intended. In response to draft
recommendation 1.1, OCIO added that the EDUCATE contract specifically calls for the
collection of much of the specific data that is used to assess the respective SLAs and that it
invests resources to assess audit trails of the data sufficient to ensure its reliance and attest to its
authenticity.

OIG Response

The IV&V contract’s Performance Work Statement defined the following tasks to be performed
by the IV&V contractor:

        Task 2 – Management support service – provide management support to include review
        of work products and/or deliverables for accuracy, completeness, and quality.

        Task 5 – SLA support – provide management support in OCIO in overseeing the
        EDUCATE contract in evaluating service provider performance as stated in the SLAs and
        reported by the service provider. This shall include attending the daily SLA meetings
        and reviewing raw data that is input into the SLA.
Final Audit Report
ED-OIG/A19K0007                                                                       Page 10 of 22

As provided above, the IV&V contract is not clear as to whether the IV&V contractor is required
to duplicate SLAM tasks. Nor is it clear that the IV&V contractor must duplicate such tasks in
order to maintain independence and perform its management and SLA support functions as
defined in the contract. Further, as noted in the finding, only some of the SLAMs were
performing tasks duplicative of IV&V tasks. This raises a question as to whether the Department
truly intended for such tasks to be duplicative. We found that due to a lack of documented SLA
validation procedures, SLAMs developed their own processes. This raises the possibility that
some SLAMs unknowingly implemented procedures that were duplicative of those already being
performed by the IV&V. Reviewing and eliminating unnecessary duplicative tasks could help
ensure that the Department’s resources are more efficiently and effectively utilized. We have
modified the associated draft recommendation slightly to reflect that while some tasks performed
by IV&V staff and Department staff may need to be duplicated, others may not call for such
action and should be considered for elimination from one of the respective parties’ assigned
duties.

While OCIO notes that resources are invested to ensure data provided by the EDUCATE
contractor are accurate and reliable, we found that processes cited to provide assurance over
query and data quality were limited and could not be substantiated, as detailed in our finding.

During the exit conference conducted on December 7, 2010, OCIO officials expressed their
belief that the necessary processes to validate SLA performance had been implemented since we
conducted our audit work. OIG proceeded to solicit additional comments and information from
OCIO in response to our proposed findings and recommendations, to determine whether any
additional processes had been implemented that we were unaware of and to ensure proper
consideration was given to them during the preparation of our draft report. No additional
information was provided at the meeting. OIG granted additional time for OCIO to provide
information prior to the issuance of the draft report. In response, OCIO officials stated they
would provide information relating to significant process updates. However, on
December 12, 2010, OCIO officials responded via email that they had no additional comments at
that time.


FINDING NO. 2 – The Department Needs to Ensure SLAMs Have Appropriate Technical
                Expertise to Monitor Assigned EDUCATE Contract SLAs.

During the period covered by our review, the Department inappropriately staffed SLA validation
positions with employees lacking the technical expertise to effectively perform their roles and
responsibilities. We found that 2 of the 6 SLAMs included in our review (33 percent) did not
have the technical background necessary to effectively monitor their assigned performance areas.

   •	 One individual was assigned as the SLAM responsible for monitoring network
      performance SLAs in September 2007. During discussions regarding his responsibilities
      as a SLAM, this individual stated that he did not possess the technical knowledge needed
      to perform this function. Although he had previous experience with aspects of network
      contract management, for example serving as a COR for several telecommunications and
      multimedia service contracts, he believed he was not knowledgeable about the technical
      aspects of network monitoring. We subsequently reviewed a listing of SLAM
      qualifications that OCIO provided as justification for this individual’s assignment to the
Final Audit Report
ED-OIG/A19K0007                                                                                      Page 11 of 22

        SLAM position. The qualifications noted for this individual consisted of certifications as
        a Professional Property Manager and COR. The noted qualifications were not aligned
        with the responsibilities of monitoring network performance SLAs. This individual
        asked for reassignment in November 2009 and was subsequently reassigned to monitor
        the EDUCATE SLA relating to property management, a better fit for his skill set.

    •	 The second individual was originally assigned to the role of Network,
       Telecommunications, and Multimedia Services CLIN owner in September 2007. Before
       being assigned to work with the network CLIN, he worked with OCIO’s Investment
       Acquisitions Management Group and Project Management Office. As CLIN owner, he
       was responsible for overseeing three SLAMs, who were supposed to have the technical
       expertise to directly monitor SLA performance. As such, the CLIN owner stated that he
       relied on the SLAMs as technical experts. However, after the transfer of the SLAM
       discussed immediately above and the retirement of another SLAM under his supervision
       in December 2009, the CLIN owner was left with increased responsibilities. He stated
       that because of the staffing limitations he assumed some SLAM responsibilities to
       monitor and validate the contractor’s performance without sufficient technical expertise
       or time needed to do so effectively. According to the listing of qualifications provided by
       OCIO, this individual possesses a Government Financial Manager certification and
       Information Technology Infrastructure Library (ITIL) v3 foundation, 8 neither of which
       appear to relate to the responsibilities of monitoring network performance SLAs.

        The CLIN owner stated he began presenting his staffing concerns to management in
        December 2009, and was assigned two additional staff between May and June 2010 to
        assist in monitoring contractor performance. According to the CLIN owner, the SLAMs
        assigned to monitor the network performance SLAs do not have the technical background
        necessary to effectively perform their duties. The CLIN owner was subsequently
        reassigned to a position outside of EDUCATE contract performance monitoring in
        October 2010. In addition, one of the two SLAMs added in 2010 is no longer in that role.

GAO “Standards for Internal Control in the Federal Government” states that “All personnel need
to possess and maintain a level of competence that allows them to accomplish their assigned
duties.”

We determined that the Department did not define the knowledge, skills, and abilities needed for
positions that monitor and validate SLAs. Instead, the Department used existing staff to fill the
SLAM positions without always effectively considering whether the individuals were
appropriately trained or experienced to perform the work. One SLAM stated that he was
assigned to monitor network SLAs because his previous job position was terminated with the
onset of the EDUCATE contract. The CLIN owner noted above took on more responsibilities
than he was qualified for and capable of doing because of staff turnover.



8
 The Foundation Level is the entry level certification which offers candidates a general awareness of key elements,
concepts and terminology used in the ITIL v3 Service Lifecycle, including the linkages between lifecycle stages, the
processes used and their contribution to Service Management practices.
Final Audit Report
ED-OIG/A19K0007                                                                       Page 12 of 22

Individuals assigned to oversee contractor performance may be unable to do so effectively
without the appropriate training or experience. This can result in the Department paying for a
level of service that it is not receiving.

Recommendations

We recommend that the CIO

2.1	   Identify the necessary knowledge, skills, and abilities required for oversight of each SLA
       and CLIN, and staff positions accordingly.

2.2	   Provide the necessary training opportunities for current staff lacking the requisite
       technical skills to perform their assigned functions.

Department Comments

OCIO concurred with the recommendations and provided a corrective action plan for each
recommendation. OCIO agreed that not all SLAMs possess the requisite technical expertise
needed to perform the required functions for each of the SLAs. However, OCIO stated that it is
confident that the majority of SLAMs possess the requisite skills, knowledge, and experience to
carry out their duties, such as general IT, analytical, and project management expertise. Further,
according to OCIO, in November 2010, it significantly restructured resources and
responsibilities in order to strengthen skills and abilities relating to specific SLAs.

OIG Response

While OIG acknowledges that SLAMs and CLIN owners reviewed as part of this audit likely
possessed general IT knowledge, skills, and abilities, our finding focused on the lack of
specialized technical expertise of some SLAM and CLIN owners that impacted their ability to
effectively perform duties within the areas they were assigned. OCIO’s response appears to
highlight the general and fundamental IT and project management skills of the assigned SLAMs
and CLIN owners and indicates that OCIO has taken steps to address the issues noted in our
finding by reallocating resources in November 2010. However, at the time of the noted
restructuring, OCIO still had not established the appropriate knowledge, skills, and abilities for
all SLAM and CLIN owners. As evidenced in the finding, OCIO had previously assigned
SLAMs and CLIN owners without having defined the necessary knowledge, skills, and abilities
required of the positions, which resulted in employees being assigned to positions with roles and
responsibilities they were not technically qualified to perform.
Final Audit Report
ED-OIG/A19K0007                                                                         Page 13 of 22

FINDING NO. 3 – The Department Needs to Improve Processes for Validation of
                Chargeback Reports.

We found that the Department needs to improve its processes related to validation of chargeback
reports. Specifically, the Department does not always use independent, accurate, and complete
data for reconciling IT assets in the contractor-provided chargeback reports, provides limited
time for affected parties to review the reports, and duplicates effort during the validation process.
The validation process is further hindered by the fact that the Department has not established
contractor performance metrics related to the completeness and accuracy of chargeback reports.

Lack of Independent, Accurate, and Complete Data

According to the Chargeback PM, POs are not required to independently maintain an inventory
of PO equipment. Rather, the EDUCATE contractor uses data from the Asset Management
Portal to update an OCIO web tool on a daily basis, and POCs can use this contractor-provided
data, along with the contractor-provided annual inventory listing, for validating the contractor-
prepared chargeback reports. We found 2 of the 5 POCs (40 percent) included in our review
maintained their own inventory of equipment, while the remaining 3 used contractor data to
validate chargeback reports. Additionally, 3 of the 5 POCs (60 percent) noted that the contractor
data is not always accurate as discrepancies have been found between the contractor’s asset
management system and the contractor’s data in OPAS.

Three out of 5 POCs (60 percent) also stated that they are not notified of all additions, moves,
installs, and changes to user profiles submitted to the EDUCATE help desk, as not all requests
require POC approval. Therefore POCs do not have complete records for validating charges for
such requests on the chargeback reports.

Limited Time for Review and Duplication of Effort

Department officials have limited time to review chargeback reports. Specifically, 2 of the 5
selected POCs (40 percent) expressed concern that they had insufficient time to review the
chargeback reports. We noted that the EDUCATE contractor has 2 weeks to generate and
prepare the chargeback reports prior to submission to the Department. Once submitted to the
Department, the Chargeback PMO review team has 5 calendar days to complete a preliminary
review of the reports and recommend acceptance as a deliverable. After the chargeback reports
are formally accepted, each POC receives the applicable report for review. POCs are allotted 3
business days for their review. Our review of chargeback reports from October 2009 through
May 2010 noted that the 5 selected POCs averaged 611 user hardware/software profiles for
validation per month. This ranged from an average of 1,479 in Federal Student Aid (FSA) to
233 in OCIO.

We also determined that numerous processes completed by the Chargeback PMO were
duplicated by the POCs during their reviews. We noted that the Chargeback PMO team follows
an internally generated 27-point review checklist during its validation process to include such
items as matching summary totals, proper formatting, and completeness of data. Our discussion
with the POCs noted that, while their processes vary due to a lack of formal policies and
procedures, their reconciliation processes include at least 8 of the 27 items (29.6 percent) on the
Chargeback PMO’s checklist.
Final Audit Report
ED-OIG/A19K0007                                                                        Page 14 of 22

GAO “Standards for Internal Control in the Federal Government” states that control activities are
an integral part of an entity’s accountability for the stewardship of government resources and
help ensure that actions are taken to address risks. It also states control activities, including
verification, and the creation and maintenance of related records, helps to ensure that all
transactions are completely and accurately recorded.

We determined that there are no written policies or procedures in place for reviewing chargeback
reports. The Department’s most current version of Handbook OM-05 “Property Management
Manual” is dated December 31, 2002, and does not reflect the Department’s current COCO
environment regarding IT equipment and related services. We noted the Department is
developing a Handbook for Property Management (Handbook) to supersede its Property
Management Manual. The most recent draft of the Handbook was dated December 13, 2010,
and included limited language reflecting the COCO environment. However, the Handbook
appears to exclude equipment and services provided under the EDUCATE contract and therefore
does not include any related procedures concerning the chargeback validation process.

According to the Chargeback PM, the Department provided instructions to the POCs for
reconciling chargeback reports. However, the instruction document provided was prepared by
the EDUCATE contractor and explains how to report errors found in the chargeback reports, not
how to validate the chargeback reports. The Chargeback PM also felt that the annual physical
inventory listing provided by the contractor was sufficient for providing independent data to
reconcile against.

We also determined there are no clear roles or responsibilities communicated regarding the
chargeback review process. Two of the 5 POCs (40 percent) interviewed were unaware of the
review process being performed by the PMO review team and therefore unaware that the review
process was being duplicated.

The Department negotiated the timeframes for the chargeback reporting process and agreed that
the EDUCATE contractor would have until the 16th of each month to submit the chargeback
reports and that the 23rd of each month would be the deadline for any changes to be made to the
inventory. After this date, the information is used by the contractor to prepare the report due the
following month. It appears this schedule may not allow all affected parties the ability to
perform an adequate review of the chargeback reports. In addition, per the EDUCATE
Chargeback Plan dated March 2, 2010, the Department agreed to not include incentives or
disincentives defined in the EDUCATE contract as part of the chargeback process in order to
keep the process as uncomplicated as possible and easily understandable to POs.

As a result, the Department lacks assurance that the chargeback reports contain accurate and
complete information, that related billings are appropriate, and that its review processes are
efficient and consistent. Without an independent data source to validate against, the Department
is essentially comparing data to the same data used in compiling the reports, providing little
value to the Department. As a result of POCs duplicating certain tasks performed by the
Chargeback PMO team, the Department is not utilizing human resources efficiently.

By not providing an adequate amount of time for review of chargeback reports, errors and
needed corrections may not be identified. PMO staff have noted that they need to work several
hours of overtime each month in order to complete their reviews within the allotted timeframes.
Final Audit Report
ED-OIG/A19K0007                                                                                      Page 15 of 22

We obtained the results of the chargeback report validations performed by the five POCs
included in our review from October 2009 through May 2010. An average of 3,053 assets was
assigned to the 5 POCs each month. The POC reviews identified a total average of 38 potential
overcharges (1.2 percent) and 42 other errors 9 (1.4 percent) in the chargeback reports each
month. It is likely that additional problems with the chargeback reports could be identified by
the POC reviews given additional time and independent data to validate against. Because the
reports are officially accepted as deliverables before the POCs have had a chance to review them,
the risk of acceptance of inaccurate reports and resulting improper payments is increased.
Corrections identified by the POCs will not be captured timely and may not be communicated to
the contractor.

Additionally, without established performance metrics, the contractor has no incentive for
ensuring that the chargeback reports it prepares are substantially accurate and complete, to
include making needed corrections as a result of Department feedback. We reviewed the results
of chargeback report reviews performed by the PMO from August 2009 through September
2010. Error rates noted by the PMO during those months ranged from 13.95 percent to
2.31 percent, with an average monthly error rate of 5.83 percent. When asked about the types of
errors being noted, the Chargeback PM stated they include duplicate charges, incorrect charges
for software installation and basic help desk calls, as well as math errors. We noted that only
35 percent of the chargeback reports were deemed acceptable by the PMO as originally
submitted. Twelve percent of the reports were not deemed acceptable until the third revision was
submitted.

Requiring an expected level of performance from the contractor would assist in reducing the
level of effort needed in the Department’s validation efforts. While informal performance
metrics have been established within the past year, there appears to be no penalty if the metrics
are not met. We reviewed contractor performance against the established informal chargeback
report metrics established from November 2009 through September 2010. The contractor was
noted as meeting the metrics for only one of the months during that time period. However, no
penalty was assessed to the contractor. The Chargeback PM stated that formal rejection of the
reports is not realistically an option as it would set the chargeback process further behind.
Report deficiencies are communicated informally to the contractor to stay within the allotted
review timeframes.

Recommendations

We recommend that the CIO and CFO

3.1	    Determine whether an independent data source is available for use in validating
        chargeback reports.

3.2	    Establish and implement written procedures to reflect the intended chargeback report
        validation process. The procedures should reflect the current IT environment, identify
        roles and responsibilities of officials and offices involved in the chargeback report


9
  Other errors include incorrect spelling of user names, equipment assigned to the wrong user, and equipment listed
in the wrong location within a PO, none of which impact billing to the Department.
Final Audit Report
ED-OIG/A19K0007                                                                       Page 16 of 22

       validation process, ensure sufficient time is available for effective review of the
       chargeback reports, and clearly define data sources to be used for validation purposes.

3.3	   Formalize performance metrics for chargeback reports and hold the contractor
       accountable for submission of quality reports.

Department Comments

OCIO/OCFO concurred with the recommendations, with the exception of draft recommendation
3.1 relating to the determination of independent data sources for use in validating chargeback
reports. OCIO/OCFO stated that EDUCATE already contains a system of record for asset
management that serves as the official asset inventory database. Draft recommendation 3.1
would thus create a redundant source of chargeback data and would be contrary to good business
practices, requiring continual, recurring reconciliation and diminishing the value of the official
system of record. OCIO/OCFO provided a corrective action plan for the remaining
recommendations.

OIG Response

As noted in the finding, POCs rely on contractor-provided data and an annual contractor-
provided asset inventory listing to validate contractor-prepared chargeback reports. As a result,
the Department is essentially comparing data to the same data used in compiling the reports,
providing little value to the Department. As also stated in the finding, 3 of the 5 POCs
(60 percent) noted that the contractor data is not always accurate as discrepancies have been
found between the contractor’s asset management system and the contractor’s data in OPAS.
Draft recommendation 3.1 does not require the Department to create a duplicative system to
manage IT assets, but rather to determine whether any alternative, independent data sources are
available for use in validation to provide greater assurance that the Department is being
appropriately billed.


FINDING NO. 4 - The Department Needs to Modify its SLA Framework to Effectively
                Encourage Improvements in Contractor Performance.

The EDUCATE contract’s SLA framework is ineffective in encouraging improvements in
contractor performance and has not been effectively modified to focus on noted areas of
substandard contractor performance. While the contract includes a percentage pool for
incentives and disincentives tied to SLA performance, they are spread among numerous SLAs,
resulting in diluted incentives and disincentives, which have little impact on the EDUCATE
contractor’s performance.

The EDUCATE SLA framework provides the performance standards applied during the
Department’s performance validation processes and identifies related incentives and
disincentives. According to the SLA framework as of March 1, 2009, 8 percent of the monthly
contract value shall be at risk through the use of disincentives. The 8 percent was proposed by
the contractor and accepted by the Department. The Department has unilateral control over the
allocation of the 8 percent among the 36 SLAs. Each quarter, the Department has the option to
reallocate the disincentive weightings among the SLAs. Under the current SLA framework, the
Final Audit Report
ED-OIG/A19K0007                                                                     Page 17 of 22

8 percent in total disincentives is allocated across all 36 SLAs with a minimum disincentive of
0.13 percent and maximum of 0.30 percent. If a particular SLA has a disincentive invoked in
one month due to missing the performance standard, and the performance standard is missed
again in the next month, then the disincentive percentage is increased by 1.08. This is continued
until the SLA performance standard is achieved or until the disincentive percentage is doubled.
As a result, the maximum disincentive for ongoing substandard performance is 0.60 percent.
Between March 2009 and February 2010, the EDUCATE contractor had disincentives applied to
approximately 7 SLAs (19 percent) per month. During this same time period the EDUCATE
contractor missed five SLA performance standards for five or more consecutive months. See
Table 1.1 below.

               Table 1.1 – SLAs Missed for Five or More Consecutive Months

                                       Number of Consecutive           Percentage of
                                       Months Between March            Months Where
                                       2009 and February 2010          Performance
                   SLA
                                         Where Performance               Standard
                                             Standards                    Missed
                                         Were Not Achieved             Consecutively
       DS-1: IMAC (Install, Move,                 5                         42
       Add, and Change)
       GN-3: Customer Survey                        9                         75
       GN-5: Time to Resolve                       12                        100
       NS-4: Video                                  6                         50
       Teleconferencing
       Availability
       SP-3: Enterprise                             8                         67
       Vulnerability

According to the Contracting Officer, the disincentives in the SLA framework are modified
based on the need to influence the EDUCATE contractor’s performance. Modification 47, dated
November 26, 2009, revised the SLA framework by reallocating the disincentives and incentives
among the 36 SLAs. However, the rationale for the reallocation was not documented by the
Department in official contract files. A review of the SLA framework found that the range of
disincentives allocated to each SLA remained unchanged, with a minimum disincentive of 0.13
percent and a maximum disincentive of 0.30 percent. Further review of the five SLAs noted
above determined the Department did not materially revise related disincentives through
Modification 47. See Table 1.2 below.
Final Audit Report
ED-OIG/A19K0007                                                                     Page 18 of 22

              Table 1.2 – Modification 47 Impact on Selected SLAs

                                                           Disincentive          Change In
                                      Disincentive
                                                           Percentage           Disincentive
              SLA                  Percentage Before
                                                              After                Value
                                    Modification 47
                                                          Modification 47
  DS-1: IMAC                             0.25%                0.27%                 0.02%
  GN-3: Customer Survey                  0.23%                0.23%                 0.00%
  GN-5: Time to Resolve                  0.27%                0.25%                -0.02%
  NS-4: Video                            0.23%                0.23%                 0.00%
  Teleconferencing Availability
  SP-3: Enterprise Vulnerability         0.23%                 0.23%               0.00%
  TOTAL                                  1.21%                 1.21%               0.00%

Federal Acquisition Regulation (FAR) Subpart 16.401(a) states that incentive contracts are
designed to obtain specific acquisition objectives by including appropriate incentive
arrangements designed to (i) motivate contractor efforts that might not otherwise be emphasized;
and (ii) discourage contractor inefficiency and waste.

FAR Subpart 16.402-2 states

       To the maximum extent practicable, positive and negative performance incentives shall
       be considered in connection with service contracts for performance of objectively
       measurable tasks when quality of performance is critical and incentives are likely to
       motivate the contractor.

       Technical performance incentives may be particularly appropriate in major systems
       contracts, both in development (when performance objectives are known and the
       fabrication of prototypes for test and evaluation is required) and in production (if
       improved performance is attainable and highly desirable to the Government).

Office of Federal Procurement Policy (OFPP) “A Guide to Best Practices for Performance-Based
Service Contracting” dated October 1998 states

       Where negative incentives are used, the deduction should represent as close as possible
       the value of the service lost. This amount is usually computed by determining the
       percentage of contract costs associated with each task. For example, if a given task
       represents 10 percent of the contract costs, then 10 percent will be the potential maximum
       deduction in the event of task failure.

We determined the implementation of an ineffective framework likely occurred because the
Department did not effectively plan for or effectively implement the performance incentives as
part of this contract. Further, there are no formal policies and procedures for evaluating the
appropriateness of the SLA framework. There is also no supporting documentation that provides
the rationale for selecting the past and present SLA framework, which could assist the
Department in determining the appropriateness of future actions.
Final Audit Report
ED-OIG/A19K0007                                                                       Page 19 of 22

The Department’s methodology for allocating incentives and disincentives limits its ability to
effectively motivate and manage the EDUCATE contractor’s performance. As evidenced above,
although the contractor repeatedly did not achieve the SLA performance standards for multiple
SLAs during the scope of our review, the resulting impact of the disincentives on the amount
paid to the contractor was negligible. More effective use of incentives and disincentives could
encourage the contractor to improve performance and do so in a timely manner.

During the course of the audit, we reviewed the incentive plan for a similar performance-based
IT contract. We found that the FSA Virtual Data Center (VDC) contract included an incentive
plan that appeared to provide a more appropriate performance structure than the EDUCATE
SLA framework. Specifically, the FSA VDC contract's incentive plan included significant and
material incentives and disincentives that should discourage poor contractor performance. For
example, the VDC contract provides for a maximum of 41 percent in total disincentives to be
applied in any given month, compared to the 8 percent maximum in total monthly disincentives
that can be applied under the EDUCATE contract. We noted that the 8 percent in disincentives
that can be applied across all performance measures under the EDUCATE contract is the same
amount that can be applied to a single performance measure under the VDC contract.

Recommendations

We recommend that the CIO and CFO

4.1	   Reevaluate the allocation of incentives and disincentives to ensure they are effective in
       achieving the desired results of the SLA framework, based upon guidelines and best
       practices for performance-based contracting.

4.2	   Establish and implement guidelines for routine reviews and evaluations of the
       appropriateness of the allocation of incentives and disincentives.

4.3	   Ensure that decisions about reallocating incentives and disincentives are adequately
       documented.

4.4	   Review other IT contracts utilizing incentive plans to identify possible best practices that
       could be established for the EDUCATE contract.

Department Comments

OCIO/OCFO concurred with the recommendations and provided a corrective action plan for
each of the recommendations.

OIG Response

OIG found that the Department’s response to recommendation 4.2 was inconsistent with our
recommendation. The Department’s corrective action plan focused on evaluating the current
process for monitoring SLAs. However, recommendation 4.2 relates to routine reviews and
evaluations of the appropriateness of the SLA framework— specifically the allocation of
contract incentives and disincentives. As a result, we have slightly modified recommendation
4.2 to more clearly define its intent.
Final Audit Report
ED-OIG/A19K0007                                                                                     Page 20 of 22



                        OBJECTIVE, SCOPE, AND METHODOLOGY



The objective of our audit was to determine whether the Department had adequate controls in
place to validate the EDUCATE contractor’s performance prior to authorizing payment of
invoices. Specifically, we assessed whether the Department had adequate processes in place to
validate contractor-submitted (1) SLA performance data and calculations, and (2) chargebacks of
unit-based expenses. To accomplish our objectives, we gained an understanding of internal
control applicable to the SLA validation and chargeback report validation processes. In addition,
we obtained an understanding of controls related to the EDUCATE contract’s SLA framework
because it serves as the fundamental basis for the Department’s assessment of contractor
performance. We reviewed applicable laws and regulations, Department policies and
procedures, relevant EDUCATE and IV&V contract documentation, GAO “Standards for
Internal Control in the Federal Government,” OFPP’s “A Guide to Best Practices for
Performance-Based Service Contracting,” and prior reports issued by OIG related to the
EDUCATE contract. Specific information on the scope and methodology applicable to each of
the key areas reviewed during our audit are presented below.

SLA Validation
We judgmentally selected for review 7 of the 36 SLAs (19 percent) included in the EDUCATE

contract based on issues identified by the IV&V contractor, significance of each SLA to the 

Department, areas deemed high-risk by the audit team, coverage across multiple SLA categories,

and the audit team’s technical competency to effectively assess related processes.

[See Enclosure 1 for selected SLAs.]


The scope of our review included analysis of the processes in place to validate SLA performance 

between November 2007 and March 2010. To assess the effectiveness of the Department’s

processes to validate contractor-submitted SLA performance data and calculations we:


       1.	 Interviewed the six SLAMs assigned to monitor the seven selected SLAs 10 regarding the
           duties they were assigned to perform, their applicable qualifications, and the processes
           they used to review and validate the SLA performance data provided by the contractor.
       2.	 Reviewed policies and procedures related to the validation processes for each of the
           selected SLAs.
       3.	 Created flow charts of the processes for each SLA and obtained concurrence from the
           applicable SLAM that our understanding of processes in place was accurate.
       4.	 Conducted discussions with IV&V officials regarding their role and responsibilities in the
           validation process and reviewed relevant IV&V contract documentation.
       5.	 Reviewed monthly IV&V contractor reports from April 2009 through March 2010 related
           to help desk ticket quality reviews.
       6.	 Independently validated the results of a process used by the Department to validate
           performance of one judgmentally selected SLA. Specifically, GN-5, Time to Resolve,
           was selected for validation because of its significance to Department operations. This


10
     One of the SLAMs interviewed was assigned responsibility for monitoring two of the selected SLAs.
Final Audit Report
ED-OIG/A19K0007                                                                       Page 21 of 22

       SLA measures the effectiveness of the contractor in resolving IT incidents that impact
       Department employees.
   7.	 Interviewed OCIO officials regarding items such as the Department’s efforts to gain
       assurance of the accuracy and completeness of the EDUCATE contractor’s performance
       data and the qualification criteria used to assign SLAMs.

Chargeback Report Validation
We judgmentally selected to review the chargeback validation processes employed by the 5
largest of the Department’s 24 POs (21 percent) based on the total number of assigned
workstations as of June 2009. These POs accounted for 3,532 of 5,882 (60 percent) of the
workstations billed to the Department by the EDUCATE contractor through the chargeback
reports. The scope of our review included analysis of the processes in place to validate
chargeback reports between November 2007 and February 2010.

We interviewed the POCs in charge of validating the chargeback reports regarding the duties
they were assigned to perform, any guidance they had received related to validating chargeback
reports, and the process used to review and reconcile the chargeback data provided by the
contractor. We also interviewed OCIO officials charged with responsibility for the initial review
of the chargeback reports and recommendation of acceptance of the contractor’s deliverables.

We reviewed documentation provided by the OCIO officials outlining the process followed and
areas included in the review and validation of the reports, as well as data provided related to the
errors noted during OCIO reviews in conjunction with any related performance metrics. We also
obtained and analyzed documentation provided by selected POCs that identified items they
considered errors in chargeback reports reviewed.

SLA Framework
We obtained and analyzed contractual documentation identifying the individual SLAs, related
performance standards, and potential impact of performance on contractor payments. We
judgmentally selected the period between March 2009 and February 2010 for review of monthly
invoices, as this was the most current 12-month period as of the start of our review. We
analyzed the disincentives to determine the average number of disincentives applied per month
as well as the number of SLAs that had disincentives applied over consecutive months. We
interviewed OCIO and OCFO officials to obtain an understanding of the SLA framework and
their role in establishing and modifying the framework.

Because there is no assurance that the judgmental samples used in this audit are representative of
their respective universe, the results should not be projected over the unsampled SLAs and
chargeback validation processes.

Use of computer-processed data for the audit was generally limited to contractor and Department
developed reports that supported performance and billing. This included monthly SLA reports,
IV&V monthly reports, chargeback reports, and monthly invoices. We did not specifically
assess the reliability of the computer-processed data. Rather, we assessed the Department’s
processes to validate this data prior to invoice payment. As such, the computer-processed data
itself did not have a material effect on the findings, conclusions or recommendations. Control
deficiencies noted with data validation processes are identified in the audit findings.
Final Audit Report
ED-OIG/A19K0007                                                                       Page 22 of 22

We conducted fieldwork at Department offices in Washington, D.C., from April 2010 through
December 2010. We provided our audit results to OCIO staff during an exit conference
conducted on December 7, 2010. We conducted this performance audit in accordance with
generally accepted government auditing standards. Those standards require that we plan and
perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our
findings and conclusions based on our audit objectives. We believe that the evidence obtained
provides a reasonable basis for our findings and conclusions based on our audit objectives.
                                                                               Enclosure 1

                             Summary of Selected SLAs

SLA Number   SLA Name                SLA Description
DS-1         IMAC                    Measures the average time to complete IMAC requests
                                     relative to a specific user for hardware, software,
                                     telecommunications, and any cabling items included
                                     under the EDUCATE contract.
HD-1         Disable User Accounts   Measures the time for deletion, suspension, and changes
                                     of access authorizations and codes for Department or
                                     contractor user accounts.
HD-4         First Call Resolution   Measures the ability of the EDUCATE Help Desk to
                                     complete “resolvable” incidents for hardware, software
                                     and system components within the desktop environment
                                     that impact customers.
GN-3         Customer Survey         Measures whether the services received from the
                                     EDUCATE contractor were satisfactory.
GN-5         Time to Resolve         Measures the time it takes for the contractor to resolve
                                     incidents for existing hardware, software, and
                                     telecommunications components within the Department
                                     that impact employees.
NS-2         Network Latency         Measures the amount of time it takes for data to travel
                                     from an end-user across the EDUCATE contractor’s
                                     network to all EDUCATE locations.
SP-1         Event and Incident      Measures the start/stop time to complete event
             Response                notification, incident notification, time to containment,
                                     and resolution.
                                                                            Enclosure 2

                      Acronyms/Abbreviations Used in this Report


ALO          Audit Liaison Officer

CAP          Corrective Action Plan

CFO          Chief Financial Officer

CIO          Chief Information Officer

CLIN         Contract Line Item Number

COCO         Contractor-Owned Contractor-Operated

COR          Contracting Officer’s Representative

CPSS         Contract and Purchasing Support System

Department   U.S. Department of Education

EDUCATE      Education Department Utility for Communications, Applications, and Technology
             Environment

FAR          Federal Acquisition Regulation

FSA          Federal Student Aid

GAO          Government Accountability Office

Handbook     Draft Handbook for Property Management, OM 05, dated December 13, 2010

IMAC         Install, Move, Add, and Change

IT           Information Technology

ITIL         Information Technology Infrastructure Library

IV&V         Independent Verification and Validation

OCIO         Office of the Chief Information Officer

OFPP         Office of Federal Procurement Policy

OIG          Office of Inspector General
OPAS   Operational Process Application Suite System

PM     Program Manager

PMO    Program Management Office

PO     Principal Office

POC    Principal Office Coordinator

SLA    Service Level Agreement

SLAM   Service Level Agreement Monitor

VDC    Virtual Data Center
                                                                                                    Enclosure 3


                  UNITED STATES DEPARTMENT OF EDUCATION

                                 WASHINGTON, D.C. 20202·,       ___




                                                         April 11, 2011


MEMORANDUM


TO:               Michele Weaver·Dugan, Director
                  Operations Internal Audit Team




FROM:             Danny A. Harris, Ph.D.
                  Chief Information offlce     -f;!J
                  Thomas p, Skelly                             ! /"'-J'''''/'/
                  Delegated to Perform Functions and Duties of the Chi f Financial Officer
                  Office of the Chief Financial Officer


SUBJECT:          Response to Draft Audit Report
                  Department's Processes for Validating the EDUCATE Contractor's Performance
                  Control Number ED-OIG/A19K0007




Thank you for providing the subject Draft Audit Report entitled "Department's Processes for
Validating the EDUCATE Contractor's Performance". We appreciate the diligence and expertise
provided by your team in conducting this extensive work. Your Draft Audit Report provides
valuable insight into the effectiveness of the EDUCATE contract performance, processes and
measurement practices and accurately identifies several areas of needed improvement. It is
apparent, however, that limited consideration was given to the extraordinary strides taken by
this office during the past year to address many of the specific issues presented in your report.
Much of the report addresses the reliance upon data provided by the EDUCATE contractor that
is used by the Department to measure performance.


However, the report often fails to note improvements that have been made or contractual and
other restrictions associated with the management of this data, including minimizing duplicative
processes, prudent fiscal management, system security limitations negating the use of multiple
data providers, and adherence to industry best practices. OCIO appreciates the insight and
attention provided by this report and is looking forward to working closely with your office to
manage the recommendations that you have identified. However, it is also important to note
that many of the management improvements implemented prior to the beginning of the audit
are presently demonstrating significant improvement in the management of the EDUCATE
contract.


The following is OCIO's responses to the Findings and associated Recommendations provided.
OCIO will address each recommendation as stipulated in the plan provided, and as agreed upon
by your office.




 Our mtuton   to en ure equal access to <!ducQto'on and to promote educational excellence throughout the Nallon.
FINDING No.1 - The Department Needs to Improve Processes for Validation of Service level
Agreement Performance.


1.1 Review all SLAs and identify possible source of independent supporting data to be used
    for SLA performance validation.


    Proposed Corrective Action -


    OCIO and OCFO concur with this recommendation and will identify possible sources of
    independent SLA supporting data where possible. It is important to note, however, that the
    EDUCATE contract specifically calls for the collection, retention and provision of much of the
    specific data that is used to assess the respective SLAs. In these circumstances, it is
    necessary to collect and evaluate this specific data to evaluate SlA performance, regardless
    of the fact that the EDUCATE contractor was responsible for collection and maintenance of
    the data. For this reason, OCIO invests resources to assess audit trails of the data sufficient
    to ensure its reliance and attest to its authenticity.


    In addition to identifying possible sources of independent SLA measurement data as
    recommended in your draft report, OCIO and OCFO shall establish written policies and
    procedures to ensure that available audit trails are maintained and evaluated by routinely
    sampling data to validate its accuracy and integrity.


    OCIO shall identify possible sources of independent SLA supporting data where possible and
    establish OCIO Policies and Procedures to be used to validate data authenticity and integrity
    by June I, 2011.


1.2 Implement procedures to periodically test underlying performance data in the
    contractor's systems for accuracy, especially data that is being relied upon for SLA
    validation purposes.


    Proposed Corrective Action-


    OCIO and OCFO concur with this recommendation and, as stated in response to the previous
    recommendation, will implement written procedures to periodically test underlying
    performance data in the contractor's system for accuracy, especially data that is being relied
    upon for SlA validation purposes. It should be noted, however, that OCIO presently reviews
    a significant sampling of all OPAS records and conducts independent tests of other hardware
    and software applications related to data used for SLA measurement.


    OCIO and OCFO shall implement procedures to periodically test underlying performance
    data in the contractor's system for accuracy, especially data that is being relied upon for SLA
    validation purposes by June 1, 2011.


1.3 Formally establish and implement validation procedures, to include the identification of 

    appropriate supporting documentation to be used for validation, for each SLA. 



    Proposed Corrective Action-
   OCIO and OCFO concur with this recommendation and will establish and implement written
   validation procedures, to indude the identification of appropriate supporting
   documentation to be used for validation, for each SLA. Please be aware however, that
   specific instruction used to measure SLAs presently exists as shared agreements within the
   EDUCATE contract. These are strictly adhered to by the SLA monitors (SLAMs) in conducting
   periodic evaluations and shall be considered in the design of the established validation
   procedures.


1.4 	Review SLA validation tasks performed by the IV&V contractor against those performed
   by SLAMs and eliminate identified duplicative efforts.


   Proposed Corrective Action -


   OCIO and OCFO do not concur with this recommendation on the basis of the necessity of
   independence of the IV&V contractor. It is imperative that the analytical products of the
   IV&V contractor be used as assessment tools to determine the validity of the services
   provided by the EDUCATE contractor, as well as the processes used by SLAMs to measure
   performance. The use of IV&V products as 5LA evaluation tools rather than independent
   management evaluation tools is contrary to the independent nature and underlying intent
   of the IV&V service provided. IV&V reports are generally duplicative of SLAM reports by
   intention, and primarily used to assess the accuracy and appropriateness of the SLA, the
   SLAM processes, and of the trends associated with these measures. In all cases, IV&V
    services are intentionally held independent of the contract management processes and free
   of influence of contract or management designs.


FINDING No.2 - The Department Needs to Ensure SLAMs Have Appropriate Technical
Expertise to Monitor Assigned EDUCATE Contract SLAs.


2.1 Identify the necessary knowledge, skills. and abilities required for oversight of each SLA
    and CliN. and staff positions accordingly.


    Proposed Corrective Action -


    OCIO concurs that not all SLAMs possess the requisite technical expertise needed to
    appropriately perform the required functions for each of the SLAs, and will identify the
    necessary knowledge, skills, and abilities required for oversight of each SLA and ClIN, and
    shall staff positions accordingly and as applicable within human resource and budget
    constraints. OCIO shall establish guidelines identifying appropriate knowledge, skills and
    abilities by June 1, 2011.


    Though aCID concurs that some SLAMs may lack suitable technical expertise to adequately
    perform the necessary functions of the position, OCIO is confident that the majority do
    possess the requisite skills, knowledge and experience to carry out the necessary oversight
    duties. It warrants notice that resource and responsibilities were significantly restructured
    in November of 2010 as aCID determined the need to strengthen skills and abilities relating
    to specific SLAs. Although each (LIN owner and SLAM perform unique functions, general IT,
    analytical, and Project Management expertise are necessary skills relating to management
   and oversight of SLAM functions. OCIO presently ensures that fundamental skills,
   knowledge and experience are possessed by the current CliN owners and SlA monitors
   including the following:


   •    Ability to apply advanced IT principles, standards, and practices sufficient to accomplish
        assignments such as: develop and interpret strategies and policies governing the
        planning and delivery of IT services throughout the agency;
   •    Provide expert technical advice, guidance, and recommendations to management and
        other technical specialists on critical IT issues;
   •    Apply new developments to previously unsolvable problems and make decisions or
        recommendations that significantly influence important agency IT policies or programs;
    •   Apply interrelationships of multiple IT specialties, the agency's IT architecture, new IT
        developments and applications, emerging technologies and their applications, IT
        security concepts, standards, and methods;
    •   Use Project Management principles, methods, and practices including developing plans
        and schedules, estimating resource requirements, defining milestones and deliverables,
        monitoring activities, and evaluating and reporting on accomplishments; and
    •   Oral and written communications techniques.


2.2 	Provide the necessary training opportunities for current staff lacking the requisite
    technical skills to perform their assigned functions.


    Proposed Corrective Action -


    OCIO concurs with this recommendation. Training has and will continue to be identified,
    recommended, and made available to all OCIO staff responsible for the management and
    oversight of the EDUCATE contract.


    OCIO shall prepare a list of the requisite knowledge, skills, and abilities required for
    oversight of each SLA and ClIN and ensure appropriate training is indentjfied and available
    for inclusion on the Individual Development Plans (lOPs) for each SLAM and ClIN owner.
    The list for recommended training for SLAMs and CliN owners to include on lOPs shall be
    completed by December 31, 2011.


FINDING No.3 - the Department Needs to Improve Processes for Validation of Chargeback
Reports.


3.1 Determine whether an independent data source is available for use in validating
    chargeback reports.


    Proposed Corrective Action -


    OCIO and OCFO do not concur with this recommendation. The Asset Management Portal
    (AMP) is the EOUCATE system of record for asset management and serves as the official
    asset inventory database. OCIO and OCFO believe that maintaining a redundant source of
    charge back data would equate to the use of a "Cuff Record" and would be contrary to good
    business practices, requiring continual, recurring reconciliation and diminishing the value of
   the official system of record. OC10 and OCFO assert that keeping a cuff system would not
   improve the validation of chargeback reports or information, and that creating such an
   independent data source would add an additional layer of necessary reconciliation and risk
   to the business process.


3.2 Establish and implement written procedures to reflect the intended chargeback report
   validation process. The procedures should reflect the current IT environment, identify
   roles and responsibilities of officials and offices involved in the chargeback report
   validation process, ensure sufficient time is available for effective review of the
    chargeback reports, and clearly defined data sources to be used for validation proposes.


   Proposed Corrective Action -


    0(10 and OCFO concur with this recommendation. OC10 and OCFO shall develop and
    implement a Departmental ACS Directive governing the appropriate processes to be used to
    validate the chargeback reports, and identify the roles and responsibilities of officials and
    offices involved in these processes. The ACS directive shall establish a schedule of events to
    ensure adequate time is given to allow program offices to validate the chargeback reports
    and shall clearly define the data sources to be used for such validation. The ACS Directive
    shall be drafted, approved, and published by December 31, 2011.


3.3 Formalize performance metrics for chargeback reports and hold the contractor
    accountable for submission of quality reports.


    Proposed Corrective Action -


    OC10 and OCFO concur with this recommendation. OCIO and OCFO shall create
    performance metrics to be used in validating the accuracy of the information received on
    the monthly chargeback reports as well as a method of tracking discrepancies identified by
    POs to ensure that the information is corrected on the following month's report. OCIO and
    OCFO shall complete these performance metrics by September 30, 2011.


FINDING No.4 - The Department Needs to Modify its SLA Framework to Effectively Encourage
Improvements in Contractor Performance.


4.1 Reevaluate the allocation of incentives and disincentives to ensure they are effective in
    achieving the desired results of the SLA framework, based upon guidelines and best
    practices for performance·based contracting.


    Proposed Corrective Action -


    OCIO and OCFO concur with this recommendation and have initiated Contract Modifications
    78 and 80 for the purpose of reallocating and maximizing the incentives and disincentives
    allowable to improve effectiveness of the 5LA framework. Decisions made followed
    discussions with other Federal Agencies to address best practices relating to the use of
    incentives and disincentives in managing large IT contracts.


    Modifications 78 and 80 shall be agreed to by the EDUCATE Contractor by May 31, 2011.
4.2 	Establish and implement guidelines for routine reviews and evaluations of the
   appropriateness of the SLA framework.


   Proposed Corrective Action-


   OCIO and OCFO concur with this recommendation. The CIO has initiated a team to evaluate
    the effectiveness of each of the SlA review processes. The team is evaluating the current
    processes used to monitor the various SLAs, shall validate that monitoring actions are
   complete and properly documented, and shall finalize their analysis by providing
    recommend improvements to the SLA monitoring process. OCiO shall provide SLA Review
    Guidelines to all CliN owners and SLAMs by June 30, 2011.


4.3 	Ensure that decisions about reallocating disincentives and incentives are adequately
    documented.


    Proposed Corrective Action-


    OCIO and OCFO concur with this recommendation and shall ensure that the EDUCATE
    contract management staff are made aware of the importance of fully documenting all
    considerations, intentions and logic used to support decisions regarding changes and use of
    EDUCATE contract incentives and disincentives. OCIO and OCFO leadership shall meet with
    EDUCATE management staff to ensure that all parties are aware of the necessity for
    adequate preparation and retention of such supporting documentation by April 30, 2011.


4.4 	Review other IT contracts utilizing Incentive plans to identify possible best practices that
    could be established for the EDUCATE contact.


    Proposed Corrective Action-


    OCIO and OCFO concur with this recommendation and have initiated discussions regarding
    best practices related to incentive plans used by other federal agencies in managing large IT
    contracts. aCID and OCFO shall continue to pursue best practices relating to incentive plans
    with additional agencies and document our findings. Results of discussions with other
    Federal agencies related to incentive plans shall be documented by May 31, 2011.


We sincerely appreciate the expertise and commitment to quality that has been provided by
DIG in conducting this audit. aCID and OCFO will quickly address all issues as presented in this
response, once agreement to our proposed actions is received. I look forward to continued
partnership with you and your staff as we work together to improve the Department's
management of the EDUCATE contract.


If you have any questions, please do not hesitate to contact Greg Robison of my staff 

at 202-245-7187.