oversight

Florida Department of Education Control Over State Assessment Scoring.

Published by the Department of Education, Office of Inspector General on 2009-09-30.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

      Florida Department of Education Control Over State
                     Assessment Scoring


                                 FINAL AUDIT REPORT




                                    ED-OIG/A04I0043
                                     September 2009


Our mission is to promote the                         U.S Department of Education
efficiency, effectiveness, and                        Office of Inspector General
integrity of the Department's                         Atlanta, Georgia
programs and operations.
                         NOTICE

Statements that managerial practices need improvements, as well as
other conclusions and recommendations in this report represent the
opinions of the Office of Inspector General. Determinations of
corrective action to be taken will be made by the appropriate
Department of Education officials.

In accordance with Freedom of Information Act (5 U.S.C. § 552),
reports issued by the Office of Inspector General are available to
members of the press and general public to the extent information
contained therein is not subject to exemptions in the Act.
                           UNITED STATES DEPARTMENT OF EDUCATION
                                           OFFICE OF INSPECTOR GENERAL

                                                                                                                         Audit Services
                                                                                                                            Region IV



                                                         September 30, 2009

Dr. Eric Smith
Commissioner of Education
Florida Department of Education
325 West Gaines Street
Tallahassee, Florida 32399-0400

Dear Dr. Smith:

Enclosed is our final audit report, Control Number ED-OIG/A04I0043, entitled Florida Department of
Education Controls Over State Assessment Scoring. This report incorporates the comments you provided
in response to the draft report. If you have any additional comments or information that you believe may
have a bearing on the resolution of this audit, you should send them directly to the following Education
Department official[s], who will consider them before taking final Departmental action on this audit:

                                           Thelma Melendez de Santa Ana
                                           Assistant Secretary
                                           Office of Elementary and Secondary Education
                                           400 Maryland Avenue, SW
                                           Room 3W315
                                           Washington, D.C. 20202


It is the policy of the U. S. Department of Education to expedite the resolution of audits by initiating
timely action on the findings and recommendations contained therein. Therefore, receipt of your
comments within 30 days would be appreciated.

In accordance with the Freedom of Information Act (5 U.S.C. § 552), reports issued by the Office of
Inspector General are available to members of the press and general public to the extent information
contained therein is not subject to exemptions in the Act.

                                                                 Sincerely,

                                                                  /s/
                                                                 Denise M. Wempe
                                                                 Regional Inspector General for Audit


Enclosures




 The Department of Education's mission is to promote student achievement and preparation for global competitiveness by fostering educational
                                                   excellence and ensuring equal access.
             List of Acronyms/Abbreviations Used in this Report


AYP           Adequate Yearly Progress

CALA-FSU      Center for Advancement of and Learning and Assessment – Florida
              State University

CTB           CTB/McGraw Hill

Department    U.S. Department of Education

ESEA          Elementary and Secondary Education Act, as amended by the No Child
              Left Behind Act of 2001

FCAT          Florida Comprehensive Assessment Test

FLDOE         Florida Department of Education

FERPA         Family Educational Rights and Privacy Act

IDEA          Individuals with Disabilities Act

LEA           Local Educational Agency

MI            Measurement Incorporated

OIG           Office of Inspector General

PII           Personally Identifiable Information

SEA           State Educational Agency

SFSF          State Fiscal Stabilization Fund

SSS           Sunshine State Standards

Standards     1999 Standards for Educational and Psychological Testing
                                              TABLE OF CONTENTS

                                                                                                                                Page

EXECUTIVE SUMMARY ...........................................................................................................1

BACKGROUND ............................................................................................................................3

AUDIT RESULTS .........................................................................................................................7

          FINDING NO. 1 – Florida Comprehensive Assessment Test (FCAT)
                          Gridded Response Discrepancies .................................................7

          FINDING NO. 2 – Insufficient Monitoring of Florida Department of
                           Education’s (FLDOE’s) Contractor...........................................10

          FINDING NO. 3 – FLDOE’S Contractor Delayed Federal Audit by
                          Limiting Access to Assessment Documentation ........................17

OTHER MATTERS ....................................................................................................................20

OBJECTIVE, SCOPE, AND METHODOLOGY ....................................................................21

Enclosure 1: Glossary .................................................................................................................23

Enclosure 2: FLDOE Response .................................................................................................24
Final Report
ED-OIG/A04I0043                                                                                      Page 1 of 36


                                       EXECUTIVE SUMMARY


The objective of our audit was to determine whether controls over scoring of assessments at the
Florida Department of Education (FLDOE) were adequate to provide reasonable assurance that
assessment results are reliable. Our review covered the Florida Comprehensive Assessment Test
(FCAT)® administered in school year 2007-2008. 1 The FCAT is used for evaluating individual
students and making adequate yearly progress (AYP) determinations under Section 1111(b)(2) of the
Elementary and Secondary Education Act (ESEA), as amended by the No Child Left Behind Act of
2001. Section 1111(b)(3) of the ESEA also requires accurate, reliable, high-quality assessment data.
Assessments are used to hold schools accountable for student achievement.

We found that FLDOE has internal controls over scoring the FCAT assessment to provide reasonable
assurance that assessment results are reliable. However, we found discrepancies in the FCAT gridded
responses and that FLDOE did not sufficiently monitor contractor activities to ensure compliance with
contract requirements. In addition, our audit was delayed because FLDOE’s contractor limited access
to documentation required for our audit.

Based on our findings, we recommend that the Assistant Secretary of the Office of Elementary and
Secondary Education require FLDOE to
        1.1    Ensure the contractor is correctly setting the Technology Intensity Calibration Algorithm
               to capture students’ gridded responses in the scanner. For responses that are manually
               entered, have a second verification of the entry to ensure the gridded response items are
               captured correctly.
        1.2    Implement procedures to test a sample of the gridded responses during live scoring to
               ensure students’ gridded responses are accurately scanned;
        2.1    Use unique identifiers instead of name, social security numbers, and dates of birth on
               assessment documents.
        2.2    Ensure that all contractors are aware of the proper handling of PII and include language
               in their contracts to properly address the correct handling procedures related to the
               disposal of student PII.
        2.3    Monitor the contractor to ensure compliance with contract provisions and include a table
               of penalties in the contract for non-compliance with contractual requirements.
        2.4    Monitor document control procedures at the contractor facilities at least annually.
        3.1    Include a Federal audit clause provision in contracts for Department funded programs.
        3.2    Include a table of penalties in the contract for non-compliance with a Federal audit.



1
 The Spring 2008 FCAT are the assessments administered during the 2007-2008 school year used to calculate adequate
yearly progress.
Final Report
ED-OIG/A04I0043                                                                                       Page 2 of 36

In its comments to the draft report, FLDOE did not agree with Finding No. 1 and Finding No. 2.
FLDOE agreed in part with Finding No. 3 but disagreed with the part of the finding related to limiting
access to documentation. FLDOE provided corrective actions to address Recommendations 1.1, 2.2,
2.3, 2.4, and 3.1. FLDOE provided corrective actions that partially address Recommendation 2.1 and
3.2 and stated that it has procedures already in place to address Recommendation 1.2. Based on
additional documentation provided to address the discrepancies identified in Finding No. 2, we
modified the finding reducing the number of discrepancies, accordingly. 2 The reduction in the
number of discrepancies did not significantly change the finding and, as such, required no change to
the recommendations. FLDOE’s comments on the draft report are summarized at the end of each
finding and included in their entirety as Enclosure 2 to this report.




2
  Measurement Incorporated (MI), the handscoring subcontractor to FLDOE’s assessment contractor (CTB), provided
additional documentation supporting the degree for 9 handscorers from the Nashville regional site; 1 handscorer for the
Tampa regional site and 9 handscorers for the Durham regional site. In addition, MI provided 8 interview sheets used to
document the applicant’s recommendation for employment – the interviewer is required to document that proof of a degree
is in the applicant folder.
Final Report
ED-OIG/A04I0043                                                                                      Page 3 of 36


                                              BACKGROUND


ESEA § 1111(b)(3) requires States to implement a set of yearly academic assessments. The
assessments are used as the primary means of determining adequate yearly progress (AYP) of the
State and each of its local educational agencies (LEAs) and schools in enabling all children to meet
the State’s student academic achievement standards. States must use the assessments to measure the
achievement of students against State academic content and student academic achievement standards
in Mathematics, Reading or Language Arts, and Science. ESEA § 1111(b)(3)(C)(iii) states that these
assessments shall be used for purposes for which such assessments are valid and reliable, and
consistent with relevant, nationally recognized professional and technical standards. In June 2007, the
Department found that Florida’s assessment system (not including alternate assessments) met all
ESEA requirements.

Section 1111(b)(3) of the ESEA also requires accurate, reliable, high-quality assessment data.
Assessments are used to hold schools accountable for student achievement. For the 2007 award year, 3
FLDOE received $15.9 million in ESEA Title VI funds for State assessments; and $18.48 million for
Individuals with Disabilities (IDEA) related activities, of which $306,000 was used for assessment
testing.

The Standards for Educational and Psychological Testing 4 (Standards) differentiates between high-
and low-stakes testing based upon the importance of the results for individuals, organizations, and
groups. According to the Standards
        At the individual level, when significant educational paths or choices of an individual
        are directly affected by test performance, such as whether a student is promoted or
        retained at a grade level, graduated, or admitted or placed into a desired program, the
        test use is said to have high stakes.… Testing programs for institutions can have high
        stakes when aggregate performance of a sample or of the entire population of test takers
        is used to infer the quality of services provided, and decisions are made about
        institutional status, rewards, or sanctions based on the test results…. The higher the
        stakes associated with a given test use, the more important it is that test-based
        inferences are supported with strong evidence of technical quality.

Accordingly, State assessments required by ESEA are considered high-stakes for States, LEAs, and
schools for the purposes of calculating and reporting AYP. However, depending on the use of the
results, these assessments may be considered high-stakes for individual students.


3
 The 2007 award year for Federal funds is July 1, 2007, to September 30, 2008.
4
 The Standards for Educational and Psychological Testing (1999) were developed jointly by the American Educational
Research Association, the American Psychological Association, and the National Council on Measurement in Education.
Final Report
ED-OIG/A04I0043                                                                                         Page 4 of 36
FLDOE State Assessments
FLDOE uses the FCAT to assess student achievement in grades 3 through 11. The FCAT consists of
criterion-referenced tests measuring benchmarks from the Sunshine State Standards 5 (SSS) in four
content areas – Mathematics (FCAT Mathematics), Reading (FCAT Reading), Science (FCAT
Science), and Writing (FCAT Writing +). FLDOE administers the FCAT Writing+ assessment in
February and the FCAT Reading, Mathematics, and Science assessments in March. Students’ mastery
of the content areas is evaluated by multiple choice, gridded-response, extended-response, and essay
test items.

FCAT results, which are typically released to school districts by early May, play an instrumental role
in 1) third grade promotions, 2) deciding whether high school seniors earn a diploma, 6 3) grading
Florida’s public schools, and 4) calculating AYP. As a result, the FCAT is considered high-stakes not
only for FLDOE, LEAs, and schools, but for individual students as well.

The FCAT is scored through the coordination of the following three entities
    •   CTB McGraw-Hill (CTB) – FLDOE entered into a $131.9 million contract with CTB for the
        period March 31, 2005, to November 30, 2010. Based on the contract, CTB is responsible for
        completing administrative work tasks and activities required for developing, printing, and
        distributing ancillary material; printing, distributing, and retrieving test books and answer
        documents; scanning and scoring answer documents; imaging and handscoring responses to
        performance tasks; and designing, printing, and distributing reports of results in Reading,
        Mathematics, Science, and Writing at selected grade levels of the FCAT.
    •   Measurement Incorporated (MI) – CTB entered into a contract 7 with MI for the period
        November 30, 2005, to November 29, 2008. MI is responsible for handscoring FCAT
        Writing+. MI is also responsible for securing test materials, hiring and training readers based
        on approved rubrics and anchor sets, and maintaining an acceptable level of inter-rater
        reliability with scoring personnel and the State.

    •   Center for Advancement of Learning and Assessment – Florida State University
        (CALA-FSU) – FLDOE entered into a $1.5 million contract with CALA-FSU effective
        February 1, 2007, to January 31, 2010. CALA-FSU conducts an independent, third party
        review of FCAT results from the scoring contractor (CTB) and subcontractor (MI).




5
  The SSS were approved by the State Board of Education in 1996 and provide guidelines for Florida’s curriculum
framework.
6
  Grade 10 students are given six opportunities to pass the FCAT. If students have not passed the FCAT by Grade 12, they
will not receive their high school diplomas, unless their American College Testing (ACT) score is sufficient to waive the
FCAT requirement or they receive a passing grade on the FCAT retake examination.
7
  The amount of the contract is proprietary information since the contract is between CTB and MI, not FLDOE and MI.
Final Report
ED-OIG/A04I0043                                                                                          Page 5 of 36
FLDOE’s contracts for assessment services total approximately $133.4 million. The following Table
provides a summary of FLDOE contracted services and the associated award amount for assessment
contractors.

                            Table – FLDOE Assessment Contract Expenditures
                                                                                                     Federal
                                                                     Total Amount of
                                                                                                  Expenditures To
          Contractor              Assessment/Service                    Contract 8
                                                                                                       Date
    CTB McGraw-Hill          FCAT Administration                             $131,916,000              $52,719,000
    CALA-FSU                 Independent Review                                 1,500,000                       09
                            Total                                            $133,416,000              $52,719,000

FLDOE Scoring Process
The FCAT is scanned at CTB’s regional scanning facilities by temporary employees. Apple One, a
human resource firm, hires seasonal employees to perform CTB’s warehouse and scanning operation
functions. The scanning process captures students’ responses for multiple choice and gridded
response items as well as images of handwritten responses to performance task items. Data pertaining
to the multiple choice and gridded responses are electronically scored in CTB’s mainframe. However,
written responses are scored by handscorers.

CTB is responsible for handscoring the Reading, Mathematics, and Science performance tasks and
subcontracts with MI to score Writing+. Although CTB and MI hire their own scorers, 10 both
assessment contractors must ensure that all scorers have a bachelor’s degree in the content area or
field related to the subject area being scored; participate in a training program wherein they score
papers under the supervision of an experienced scoring director and an FLDOE content area expert;
and pass qualifying tests before being hired. Candidates selected for hire receive other training and
undergo quality control checks to include supervisory review of their work; pseudo scoring; and,
when necessary, retraining. FLDOE monitors the inter-rater reliability of scorers through a live,
secure File Transfer Protocol site. Scorers that do not maintain an acceptable level of scoring
accuracy are dismissed.

Several controls are included in FLDOE’s scoring process to ensure accurate and reliable reports of
FCAT results. Specifically,
      •    During each FCAT administration, CTB is contractually obligated to develop a data
           verification plan. One component of the plan, mock data processing, tests that all scanning,
           editing, scoring, and reporting functions are working properly prior to live scoring. FLDOE
           performs a number of checks to ensure the accuracy of the answer keys.


8
 All dollar amounts rounded to the nearest $1,000.
9
  Federal funds are not used to fund the CALA-FSU contract.
10
   Kelly Services provides scoring candidates to CTB by a contractual arrangement. MI is directly responsible for hiring
its scorers.
Final Report
ED-OIG/A04I0043                                                                                            Page 6 of 36
     •   Before official scores are released, FLDOE’s Data Analysis Reports and Psychometric
         Services Team crosscheck students’ individual responses to the answer keys, compare the
         scale scores 11 submitted by CTB to the scale scores computed by FLDOE, and discuss the
         results with CTB. All verification procedures are conducted using a different program and
         software than that used by CTB.

In addition, FLDOE contracts with CALA-FSU for a third-party, independent test of student scores.
At the LEA level, school districts are provided an opportunity to verify the accuracy of demographic
data and review individual student performance. LEAs may also request that the State review a
student’s score before the scores are finalized and published.




11
  The score used to report test results on the entire test. Scale scores on the FCAT Sunshine State Standards tests are 100-
500 for each grade level and content area. A computer program is used to analyze student responses and to compute the
scale score. The scale score is used on the Student and Parent Reports for Science and Writing.
Final Report
ED-OIG/A04I0043                                                                                       Page 7 of 36


                                             AUDIT RESULTS


FLDOE has internal controls over scoring of the FCAT assessment to provide reasonable assurance
that assessment results are reliable. However, we found discrepancies in the FCAT gridded responses
and insufficient monitoring of FLDOE’s contractor to ensure compliance with contract requirements.
In addition, FLDOE’s contractor limited access to documentation required for our audit, delaying the
audit.

FINDING NO. 1 – Florida Comprehensive Assessment Test (FCAT) Gridded Response
                Discrepancies

We identified 9 gridded response discrepancies 12 related to the FCAT Reading and Mathematics test
and answer booklets for the sample of 50 students reviewed. 13 The gridded response question 14
requires students to select a numeric answer and shade in the corresponding bubble on a bubbled grid.
In 6 of the discrepancies, the students’ initial response appeared to be erased clearly enough that the
scanner should have recorded only the darkened revised response, but it did not. In 1 of the
discrepancies, the scanner picked up a response that was not selected by the student. In 2 of the
discrepancies, students did not completely erase the first response and 2 bubbles were dark enough for
the scanner to record both responses, but it did not pick up either of the responses. As a result, one
sixth grade student did not receive credit for a correct response.

ESEA § 1111(b)(3)(C)(iii) states that assessments shall “be used for purposes for which such
assessments are valid and reliable, and be consistent with relevant, nationally recognized professional
and technical standards.”

FLDOE acknowledged that incomplete erasures are problematic. According to FLDOE officials,
some erasers leave enough carbon residue for the scanner to pick up the erasure, others do not.
Specifically, the CTB Technology Intensity Resolution Algorithm setting (i.e., the scanner’s reading
setting) should be at a level so the scanner
        •    recognizes as a mark any response with an intensity level 15 of 5 or above on the
             15-point scale used by the scanner,
        •    is especially sensitive to whether or not the middle of the bubble is filled in, and
        •    will choose between 2 marks in a column if the 2 are 3 or more intensity-levels apart.
12
   These discrepancies related to test booklets for 8 of the 50 students sampled.
13
   Gridded responses appear on the FCAT SSS Mathematics test in Grades 5-10 and the FCAT SSS Science test in Grades
8 and 11. We tested Grades 5-10 Mathematics only.
14
   This test item appears in the FCAT Mathematics test in Grades 5-10 and the FCAT Science test in Grades 8 and 11.
15
   The intensity level is a measure of the darkness of the students shading of the bubble in response to the question.
Final Report
ED-OIG/A04I0043                                                                            Page 8 of 36

For example, in a 5-column, gridded-response item, the student must be careful to completely erase
when changing an answer, because each column is judged independently by the scanner. The student
should shade the changed answer as darkly as possible. In addition, when an erasure has no other
bubble in the column, it will likely be picked up unless it is thoroughly erased.

As a result of the discrepancies identified, students may not be receiving proper credit for their
responses based on the intensity, or lack thereof, of an erasure. Inaccurate scanning of gridded
responses could affect the individual student’s overall score and potentially otherwise impact the
student given the high-stakes implications of the FCAT.

Recommendations

We recommend that the Assistant Secretary of the Office of Elementary and Secondary Education
require FLDOE to

1.1     Ensure the contractor is correctly setting the Technology Intensity Calibration Algorithm to
        capture students’ gridded responses in the scanner. For responses that are manually entered,
        have a second verification of the entry to ensure the gridded response items are captured
        correctly.

1.2     Implement procedures to test a sample of the gridded responses during live scoring to ensure
        students’ gridded responses are accurately scanned.

FLDOE Comments

FLDOE did not concur with Finding 1. FLDOE did not specify whether or not it concurred with the
related recommendations but rather indicated that it already had procedures in place to address the
recommendations.

In its comments, FLDOE stated that it employs an answer document decoding process that uses
nationally recognized professional and technical standards and state-of-the-art scanning technology to
ensure that assessment results are reported accurately. FLDOE’s comments to the draft report
included the following 4 points related to the finding.

      1. FLDOE took exception to our methodology for validating the scanning accuracy of gridded
         responses using copies of answer documents rather than reviewing and/or
         re-scanning original documents. Specifically, FLDOE asserted that the pages of the test
         booklets do not photocopy well and it would be difficult for the human eye to discern the
         difference between an erasure and a stray particle.

      2. FLDOE indicated that current practice is not fully or accurately described in the audit report
         related to this finding. Specifically, the scanner has built in checks for miscalibration; the
         assessment contractor CTB follows standard operating procedures for scanner calibration
Final Report
ED-OIG/A04I0043                                                                           Page 9 of 36

       including recalibrations after every 5,000 scans; FLDOE staff are present at the scoring site to
       perform an early scan check to ensure scanning accuracy; and student demographic data and
       response arrays on original answer documents are compared to the electronic scan file to
       ensure that documents are accurately scanned.

    3. FLDOE has 2 processes for resolving scanning errors missed during front-end checks. First,
       an automatic manual rescore of tests for Grade 12 – Adult Retake students who have failed by
       25 or fewer points; and second, the districts are allowed to submit a request to have student
       scores reviewed for scoring anomalies or scanning problems.

    4. Prior to the test administrations, students are instructed on responding to multiple choice items,
       erasing completely, completing gridded response items, and checking their marks.

Although acknowledging potential quality control issues with key entry data, FLDOE stated that its
scoring specifications and the contractor’s procedures require subsequent verification of the data
manually entered into the system.

FLDOE stated that the new assessment contractor, Pearson, will be required to ensure the calibration
of the scanner is completed and documented to show the frequency of the scanner calibrations.

OIG Comments

FLDOE provided no additional information that would require a change to Finding No. 1 and the
related recommendations. The methodology of reviewing copies of test booklets was employed to
respond to limitations placed on the audit by the assessment contractor, CTB. Basically, the audit
team had 4 days to pull a sample of 84 FCAT test and answer booklets, which could not be removed
from the contractor’s warehouse. Copies of the sample documents were required to allow the auditors
to adequately review the sample. However, we provided copies of all sample documents with
discrepancies to FLDOE and adjusted our finding when FLDOE provided an original document that
resolved the discrepancy. Although FLDOE had the opportunity to resolve any discrepancy with
original documentation, it only resolved 1 discrepancy – the remaining discrepancies noted in the
finding were not resolved.

Our report acknowledges that FLDOE has an automatic manual rescore process for Grades 12-Adult
Retake students who have failed the FCAT Reading and Mathematics test by 25 or fewer scale score
points. However, this automatic manual rescore process does not include grades 3 through 10. Since
the FCAT is considered a high-stakes exam and affects promotion for grades 3 and 10, it is important
that FLDOE implement procedures to capture all student responses correctly. At a minimum, FLDOE
should expand its manual rescore process to include grades where the FCAT is considered a high-
stakes examination.
Final Report
ED-OIG/A04I0043                                                                          Page 10 of 36
Although standard operating procedures for scanner calibration include recalibrations after every
5,000 scans, FLDOE did not include in its response the scanning and calibration reports to validate
that the assessment contractor followed those procedures. In addition, as reported in the Other
Matters section of this report, FLDOE does not consider the “specifications,” which detail the
calibration requirements, as part of the assessment contractor’s contractual requirements.

FLDOE addressed Recommendation 1.1 by including requirements in its contract with the new
assessment contractor, Pearson. Specifically, FLDOE stated it will require the contractor to perform
scanning calibrations and document the procedures to show the frequency of the scanner calibrations.
However, FLDOE did not include in its response a copy of the new contract for us to validate that the
requirement is included.

FLDOE indicated that it has procedures in place to address Recommendation 1.2. Specifically,
FLDOE stated that it tests a sample of the gridded responses prior to each test administration to ensure
students’ gridded responses are accurately scanned. We observed the testing while on-site prior to the
administration of the test. However, despite the testing done by FLDOE, we identified a number of
gridded response discrepancies in this report. As such, FLDOE should review its methodology for
sampling gridded responses and perform the testing during live scoring.

FINDING NO. 2 – Insufficient Monitoring of Florida Department of Education’s
                 (FLDOE’s) Contractor

FLDOE did not sufficiently monitor its assessment contractor, CTB. Specifically, we found that
FLDOE was not aware that CTB (1) had inadequate safeguards for discarding FCAT Student PII;
(2) did not comply with contract terms; and (3) did not have adequate document control procedures.

Inadequate Safeguards for Discarding FCAT Student PII
During our visit to the CTB contractor facility in Indianapolis, Indiana, we found that student PII was
plainly visible after being discarded in the trash. Specifically, CTB had inadequate controls for
safeguarding PII, an issue that we identified in two of the three States selected for review. CTB
officials acknowledged the incident and immediately issued a Quality Alert Memorandum and took
steps to resolve the issue. However, the contractor needs to implement additional procedures,
including secure trash pickup or shredders at each quality control workstation to better safeguard
FCAT material.

According to the Forum Guide to Protecting the Privacy of Student Information: State and Local
Education Agencies, issued by the National Forum on Education Statistics, agency and school
personnel are legally and ethically obligated to safeguard the confidentiality of student data. In
addition, a policy letter issued by the Director of the Federal Policy Compliance Office in
January 2008 states that when contractors are used to provide services that require access to student
PII, they are bound by the same restrictions on disclosure and destruction of information that apply to
the SEA. The SEA is also responsible for ensuring that its contractors do not disclose PII or allow
any other party to have access to any PII from education records. Although the FLDOE contract
Final Report
ED-OIG/A04I0043                                                                                            Page 11 of 36
included a statement on the Family Educational Rights and Privacy Act (FERPA), CTB did not
comply with the contract and, as a result, we found PII documentation discarded and plainly visible in
the contractor’s trash. FLDOE’s current practice for printing or displaying student social security
numbers instead of other unique identifiers on assessment documents demands a higher degree of
diligence from everyone handling those documents.

Non-compliance with Contract Terms
Although required by contract, CTB did not 1) notify FLDOE of changes to subcontractors; 2) have a
disaster recovery plan available at the Indianapolis facility; and 3) sufficiently monitor its
subcontractors. As a result, CTB was not in compliance with contract terms.

The FLDOE contract requires CTB to 1) receive advance, written consent from FLDOE prior to
assigning or subcontracting all or any portion of the contract; 2) have a disaster recovery plan on hand
at the facility; and 3) monitor subcontracts. In addition, 34 C.F.R. § 80.40 states
         Grantees are responsible for managing the day-to-day operations of grant and
         subgrant supported activities. Grantees must monitor grant and subgrant supported
         activities to assure compliance with applicable Federal requirements and that
         performance goals are being achieved. Grantee monitoring must cover each program,
         function or activity.

Change of Subcontractor Without Notification. The CTB Temporary Employment Services
Agreement specified Kelly Services as the subcontractor responsible for hiring temporary personnel
for scanning/scoring operations with CTB. However, CTB changed from Kelly Services to
AppleOne 16 without notifying FLDOE. The FLDOE assessment contract states that “the Contractor
may not assign or subcontract all or any portion of this Contract without the advance written consent
of the Department.” However, FLDOE officials were not notified that CTB changed its subcontractor
for hiring operations personnel from Kelly Services to AppleOne.

No Disaster Recovery Plan. The Indianapolis, Indiana, scoring operations facility does not have a
disaster recovery plan. According to the FLDOE contract with CTB, a disaster recovery plan was
required to be on hand at the facility. In addition, 34 C.F.R. § 80.36 (b)(2) states
         Grantees and subgrantees will maintain a contract administration system which ensures
         that contractors perform in accordance with the terms, conditions, and specifications of
         their contracts or purchase orders.

Although the General Manager described procedures and stated that the disaster recovery plan
was maintained at the CTB office in Monterrey, California, we did not receive a copy of the
plan.

16
  Howroyd-Wright Employment Agency, Inc. dba AppleOne Employment Services (AppleOne). AppleOne services
include temporary, direct hire, and temp-to-hire staffing; employee screening and training; payroll and tax filing; and time
and attendance processing. AppleOne holds a contract with the U.S. General Services Administration (Contract Number:
GS-02F-0011U) effective October 2007 and due to expire in October 2012.
Final Report
ED-OIG/A04I0043                                                                                        Page 12 of 36

Insufficient Monitoring of Subcontractors. CTB did not sufficiently monitor its subcontractors,
AppleOne and MI, to ensure that they hired qualified employees. As a result, we found that 14
AppleOne employees were not qualified to be hired and 17 MI readers should not have been allowed
to score the Writing+ test.

CTB’s General Manager stated that the subcontract requires that operational personnel hired to
process the FCAT receive a qualifying score on AppleOne’s basic skills test (Prove It). However,
CTB did not sufficiently monitor AppleOne to ensure that its employees passed the test, and we found
that 14 of 72 employees did not pass the test.

Although CTB is responsible for the Writing+ test that is scored by the subcontractor, MI, CTB did
not sufficiently monitor MI to ensure that its employees had the required degrees to score the FCAT
Writing + test. The FLDOE RFP and Contract Specification require that “All scorers will have at
least a bachelor’s degree in the FCAT content area or related fields of English, Mathematics, Science
or Education to perform testing.” In addition, in 2007, FLDOE amended the contract, requiring CTB,
and in turn, MI to hire a third party independent contractor to verify the degree information of
handscoring applicants. We reviewed 647 MI training files and found that MI was unable to provide
adequate documentation showing that its handscorers obtained bachelor’s degrees or that the degree
information was verified in a timely manner. Of the 647 reviewed, we found that 16 MI readers 17
should not have been allowed to score the Writing+ test. Specifically,
     •   Of the 206 handscorers from the Nashville 18 regional site, 15 did not have degree verification
         by a third party independent contractor completed until 2009; 4 did not have a copy of a
         degree or transcript 19 in the file.
     •    Of the 241 handscorers from the Tampa 20regional site, 1 did not have degree verification by a
         third party independent contractor completed until 2009 and 3 did not have a copy of the
         degree or transcript in the file.
     •   All 200 handscorers from the Durham 21 regional site had their verifications by a third party
         independent contractor completed in 2007 or 2008 (prior to the FCAT scoring). However, 2
         did not have a copy of the degree or transcript in the file and 3 did not have a copy of the MI
         interview sheet 22 in the file.


17
   As indicated in the bullets, the 16 consists of the 15 MI employees from the Nashville field office and 1 MI employee
from the Tampa field office, who did not complete verifications until after scoring had been completed.
18
   The Nashville Regional office tests the fourth grade writing.
19
   We accepted the transcript and teaching certificate if it stated Bachelor’s of Science or Bachelor’s of Arts degree
conferred with the year indicated.
20
   The Tampa Regional office tests the eighth grade writing.
21
   The Durham Regional office tests the tenth grade writing.
22
   The MI Interview sheet is used to document the applicant’s recommendation for employment. On the interview sheet is
a documentation review/applicant briefing box. The interviewer is required to document that proof of a degree is in the
applicant folder.
Final Report
ED-OIG/A04I0043                                                                                          Page 13 of 36
Although FLDOE tests a 20 percent sample of FCAT Writing+ handscorers during the testing season,
it does not verify names to transcripts. MI through CTB provides FLDOE an employee number as the
identifier and copies of transcript information on an employee sheet. FLDOE does not see or verify
names or transcripts to ensure that personnel are qualified to score the FCAT Writing+ exam. FLDOE
officials stated that because of Florida’s open records laws, the MI/CTB employee handscorers degree
information is PII which FLDOE does not want to be responsible for maintaining. According to State
officials, FLDOE relies on its assessment contractors to verify the degree requirements and ensure that
personnel are qualified to perform the scoring.

Inadequate Document Control Procedures
CTB does not have adequate document control procedures at the Indianapolis, Indiana, facility. We
requested that FLDOE provide the FCAT Reading and Mathematics test and answer books for 50
students. 23 However, FLDOE was not able to convince its assessment contractor to provide the
requested information, and the assessment contractor, CTB, was not willing to retrieve the answer
books without payment. After CTB refused to provide required documentation necessary to address
the objective of our audit, we ultimately retrieved the data without contractor assistance. However,
the retrieval of the required documentation was hindered by unreliable information provided through
CTB’s document control system (ASAP 24).

FLDOE officials stated the FCAT booklets were archived in the CTB warehouse and retrieval of the
test booklets for a Federal audit was not part of the FLDOE contract with the CTB contractor. CTB
provided a worksheet request for payment of $3,750 to retrieve the FCAT sample. Since CTB was
unwilling to provide the requested data without payment, we issued an OIG subpoena to obtain the
information needed for our audit.

In response to our January 2009 subpoena for the required documentation, CTB again requested
payment of $3,750 to retrieve the sample. In subsequent discussions, CTB agreed to provide access to
the information if audit staff would retrieve the test booklets themselves from its warehouse.
However, CTB’s document control system was unreliable and contained incorrect information.

After receiving 4 separate reports from CTB related to the location of the FCAT booklets in the cages
and CTB’s document control personnel pulling the wrong cages, we were able to retrieve only 70 of
the selected 84 answer booklets 25 while on-site. In addition, we noted the following issues with
CTB’s document control system – (1) rows were not numbered in 7 cages; (2) the reports provided by
CTB included 5 Science books even though Science books were not included in the sample; (3) CTB



23
   Our sample of Reading and Mathematics tests for 50 students would have totaled 100 single test and answer booklets
(50 Reading plus 50 Mathematics) but instead totaled 84 single test and answer booklets because the sixth, seventh, and
ninth grade booklets were one combined Reading and Mathematics test.
24
   ASAP is the name of CTB’s document retrieval system and is not an acronym.
25
   Our sample of 50 Reading and Mathematics tests equated to 84 single test and answer booklets because the sixth,
seventh, and ninth grade booklets were one combined Reading and Mathematics test.
Final Report
ED-OIG/A04I0043                                                                                     Page 14 of 36
                  26
had to “re-cage” a cage to find 1 answer booklet; and (4) we located 1 book in a row different than
the row reported by the ASAP system.

We requested CTB provide the remaining sample of 14 FCAT answer booklets that we were unable to
locate. Of the 14 FCAT booklets CTB retrieved, we noted that there were problems with 2 of the test
booklets. One test booklet had 2 pages missing, and the wrong FCAT booklet was provided in the
other instance, requiring a third request to FLDOE and CTB to receive the complete test booklets.

FLDOE did not sufficiently monitor the assessment contract to ensure that CTB 1) safeguarded FCAT
student PII information; 2) complied with contract terms, including monitoring subcontractors; and
3) had adequate document control procedures to locate assessment documentation from the storage
facility. FLDOE officials stated that the main concern for the State is that assessment scores reported
are reliable and accurate. FLDOE acknowledged the issues the audit team found but stated that the
issues identified did not affect the accuracy of the reporting of FCAT scores.

FLDOE included penalties for non-compliance in the contract for critical activities and extremely
critical work tasks; however, there are no penalties in the contract for the non-compliance identified in
this finding. Because of the high-stakes nature of the FCAT assessment program, all aspects of test
administration, scoring, and reporting require proper diligence toward maintaining test security, as
well as the accuracy and reliability of test results. FLDOE should document and monitor contractor
performance in areas that are not currently reviewed yet are important to reliability of the assessment
results.

Recommendations

We recommend that the Assistant Secretary of the Office of Elementary and Secondary Education
require FLDOE to

2.1     Use unique identifiers instead of student social security numbers and dates of birth on
        assessment documents.

2.2     Ensure that all contractors are aware of the proper handling of PII and include language in their
        contracts to properly address the correct handling procedures related to the disposal of student
        PII.

2.3     Monitor the contractor to ensure compliance with contract provisions and include a table of
        penalties in the contract for non-compliance with contractual requirements.

2.4     Monitor document control procedures at the contractor facilities at least annually.



26
  Document control personnel had to remove every FCAT booklet from a cage, rescan, renumber, and place back in the
proper order to find the one FCAT answer booklet.
Final Report
ED-OIG/A04I0043                                                                           Page 15 of 36
FLDOE Comments

FLDOE generally concurred with Finding No. 2. Although FLDOE’s comments did not specifically
address whether it concurred with the recommendations, the comments indicated that FLDOE had
implemented actions to address Recommendations 2.2, 2.3, and 2.4, but did not implement any action
at this time to address Recommendation 2.1.

FLDOE stated that it enforces the protection of PII and requires its contractors to protect PII as well.
FLDOE disagreed that our finding PII discarded in a non-secure fashion merited the conclusion that,
“FLDOE did not sufficiently monitor its assessment contractor.” FLDOE stated that its Scoring and
Reporting personnel are present at scanning sites during the scoring process of each FCAT
administration to ensure that all aspects of the scanning operation proceed according to its contractual
agreement and that related requirements are met. FLDOE stated that it regretted that the cited incident
occurred but did not think the one instance warranted the audit team’s conclusion.

FLDOE disagreed that CTB’s change from Kelly Services to Apple One for hiring operations violated
contractual requirements. Although FLDOE’s comments included the contract section requiring any
change of subcontractors to be approved in advance, FLDOE stated that neither Kelly Services nor
Apple One were included in CTB’s chart of subcontractors and services. According to FLDOE, CTB
explained that it had long-standing relationships with both agencies for hiring temporary employees
and, as such, were not required to list them on the proposal. FLDOE agreed that it is not unreasonable
that CTB only request approval for changes to subcontractors identified in the proposal.

FLDOE concurred that the contractor should have been able to provide a copy of the Disaster
Recovery Plan and added that it has requested written assurance from the current assessment
contractor, Pearson, that the Plan is readily available for review at all scanning sites. FLDOE stated
that verification of the Disaster Recovery Plan will be added to its monitoring checklist.

FLDOE stated that it reviewed a random sample of 20 percent of scorer candidates’ qualifications and
either approved or dismissed candidates based on the results of the review. FLDOE acknowledged
that a review of the entire pool of candidates may have resulted in findings similar to the audit
findings, but the sample of 20 percent was agreed to by FLDOE and CTB in recognition of the time
critical factor and human resources required to conduct the reviews.

According to FLDOE, the assessment contractor complied with all document control procedures
called for in the contract. While it acknowledged that the audit team’s experience indicated some
corporate inefficiencies, FLDOE maintained that its needs for document retrieval have been met.

Regarding Recommendation 2.1, FLDOE stated that it would continue to use the process currently in
place to ensure timely and accurate reporting of student results and maintain stringent security policies
and procedures, but would explore the potential use of alternative identifiers for students.
Final Report
ED-OIG/A04I0043                                                                          Page 16 of 36
Regarding Recommendation 2.2, FLDOE included an excerpt from the contract with the new
assessment contractor, Pearson, which included requirements for properly handling PII.

Regarding Recommendation 2.3, FLDOE stated that it has several quality assurance measures
including performance metrics to hold the contractor accountable for contractual obligations. FLDOE
added that if requirements are not met, payment is withheld; and it provided an excerpt from its
current contract with Pearson documenting that provision in the contract.

Regarding Recommendation 2.4, FLDOE stated that is has added specific items and processes to its
monitoring checklist, including verification of procedures used by the contractor to manage and
dispose of PII and verification of the Disaster Recovery Plan.

OIG Comments

Based on additional documentation that the handscoring subcontractor provided separately from
FLDOE’s comments, we modified the finding. Specifically, MI, the handscoring subcontractor of
FLDOE’s assessment contractor (CTB), provided additional documentation supporting the degree for
9 handscorers from the Nashville regional site; 1 handscorer for the Tampa regional site and 9
handscorers for the Durham regional site. In addition, MI provided 8 interview sheets used to
document the applicant’s recommendation for employment – the interviewer is required to document
that proof of a degree is in the applicant folder. Based on the additional documentation, we reduced
the number of discrepancies, accordingly. The reduction in the number of discrepancies did not
significantly change the finding and, as such, required no change to the recommendations.

In response to FLDOE’s contract requirements for properly handling PII in its current contract with
Pearson, we noted the same language in its contract with CTB and still found student PII information
discarded in a trash receptacle. Because FLDOE stated that it will continue to use PII information
while exploring alternative identifiers for students, it is important that FLDOE provide oversight and
monitoring to ensure compliance with the contract’s requirements for handling PII.

Although FLDOE can waive contract requirements, we maintain that CTB’s change from Kelly
Services to Apple One for hiring operations violated explicit provisions in the contract. The contract
did not exclude long-standing contract relationships from the required approval and, therefore, CTB
should have requested FLDOE’s approval to change subcontractors.

FLDOE stated that it reviewed a random sample of 20 percent of scorer candidates’ qualifications and
either approved or dismissed candidates based on the results of the review. However, based on the
number of discrepancies identified in our audit, FLDOE should consider re-evaluating its scorer
qualification review methodology.

In response to FLDOE’s contract requirements for quality assurance measures, including performance
metrics to hold the contractor accountable for contractual obligations and withholding payment when
requirements are not met, we noted that the same language is in its contract with CTB. However, the
Final Report
ED-OIG/A04I0043                                                                             Page 17 of 36
liquidated damages section of the contractor did not address any of the discrepancies noted in our
audit report. Therefore, it is important that FLDOE monitor the contractor to ensure compliance with
the contract. FLDOE should consider adding language related to standard penalties for general non-
compliance so that in encompasses all contractual requirements.

Although FLDOE indicated that it has added specific items and processes to its monitoring checklist,
it did not include in its response a copy of the revised monitoring checklist for our review.

FINDING NO. 3 – FLDOE’S Contractor Delayed Federal Audit by Limiting Access to
                Assessment Documentation

FLDOE and its assessment contractor did not provide access to documentation required to address the
objectives of our Federal audit. In our effort to obtain documentation from the contractor, we found
that FLDOE’s contracts were not adequate to facilitate a Federal audit. FLDOE had a contract with
CTB for the FCAT Reading, Math, Science, and Writing+ assessments. However, the contract did not
include the required audit clause 27 allowing access to data for audits by Federal officials despite the
fact that FLDOE contracted with these vendors to assist in its effort to meet Federal requirements
under ESEA § 1111, and each contract was paid, at least in part, using more than $52.7 million in
Federal funds (as shown previously in the Table).

In addition to test booklets, we requested that the assessment contractor and subcontractors provide all
training files, degrees, and qualifying score sets for contractor personnel that scored the Spring 2008
FCAT. The assessment contractor and subcontractors stated that the information was proprietary and
could not be provided to the audit team and that the retrieval of the test booklets for our audit was not
part of the FLDOE contract and would require an additional payment. Even though the contracts for
assessment services with CTB and Harcourt contained clauses authorizing State audits, FLDOE was
unable to obtain this documentation from its contractors for use in our Federal audit.

As a result, we issued subpoenas to CTB and two of its subcontractors to provide the required source
documentation. In response to the subpoenas, the two subcontractors complied and provided the
required source documentation; and CTB, the assessment contractor, complied with a portion of

the subpoena. However, CTB provided a work statement and price quotation for $3,750 to retrieve
the audit sample requested to complete our audit. Although CTB was not willing to provide the
required information without payment, it allowed the audit team to retrieve the audit sample
themselves.

The Inspector General Act of 1978 states –
           In addition to the authority otherwise provided by this Act, each Inspector General, in
           carrying out the provisions of this Act, is authorized—


27
     Required by 34 C.F.R. § 80.36 (i)(10).
Final Report
ED-OIG/A04I0043                                                                            Page 18 of 36
       (1) to have access to all records, reports, audits, reviews, documents, papers,
       recommendations, or other material available to the applicable establishment which
       relate to programs and operations with respect to which that Inspector General has
       responsibilities under this Act.

34 C.F.R. § 80.36 (i)(10) provides that a grantee’s contracts must contain a provision requiring access
by the grantee, the Federal grantor agency, or any duly authorized representatives to any books,
documents, papers, and records of the contractor which are directly pertinent to that specific contract
for the purpose of making audit.”

Assessment contractors play a large role in the scoring of FLDOE’s State assessments. Because
FLDOE did not include the audit clause allowing access to documentation for Federal audits in its
contracts with assessment vendors and the vendors would not comply with our request for
information, we issued subpoenas to FLDOE’s primary contractors and two subcontractors to compel
them to provide the documentation needed to complete the audit. The need to use OIG subpoena
authority delayed the audit. If Federal auditors are unable to gain access to records and personnel to
obtain relevant information while conducting audits, Federal oversight is diminished and Federal
funds and programs may be at risk.

Recommendations

We recommend that the Assistant Secretary of the Office of Elementary and Secondary Education
require FLDOE to

3.1    Include a Federal audit clause provision in contracts for Department funded programs.

3.2    Include a table of penalties in the contract for non-compliance with a Federal audit.

FLDOE Comments

FLDOE agreed that it did not include a Federal audit clause in its contract but did not agree that the
assessment contractor delayed the Federal audit by limiting access to assessment documentation.
FLDOE agreed to Recommendation 3.1 and stated that it would review its liquidated damages
section of the contract and determine whether penalties for non-compliance should be included in
future contracts.

In its response, FLDOE stated that it and its assessment contractor did not limit access to
documentation required to address the objectives of the Federal audit. According to FLDOE, from the
initial site visit by the auditing team to the conclusion of the audit, all requested information and data
regarding Statewide assessments in Florida were provided. FLDOE stated that, although it suggested
multiple strategies to physically check a representative sample of original answer documents against
the electronic scan file during live scanning, the audit team requested the 2-year-old documents from a
secure warehouse where processed documents are stored and archived. FLDOE added that due to the
Final Report
ED-OIG/A04I0043                                                                          Page 19 of 36
nature of the request, the contractor may have taken longer to process this request than desired by the
audit team, but at no point was access denied or limited.

OIG Comments

We acknowledge that eventually we were provided access to the information needed to address our
audit objectives. However, we were forced to issue federal administrative subpoenas for records that
FLDOE’s contractors were required to provide to us by regulation. Further, the access eventually
granted was limited and under the direct supervision of assessment contractor staff.

During the course of the audit, we requested test booklets and answer sheets to assess the adequacy of
controls to ensure accurate scanning and that the scanned data maintained its integrity during export to
the contractor’s system. The assessment contractor and subcontractors initially stated that the
information requested was proprietary and could not be provided to the audit team. Subsequently, the
assessment contractor stated that the retrieval of the test booklets for our audit was not part of the
FLDOE contract and would require an additional payment. In response to subpoenas, the contractor
allowed us supervised access to the documentation needed to complete our objectives, and the
subcontractors provided limited information to access qualifications. The extended process used to
eventually gain access to documentation delayed the audit by several months.

FLDOE provided a copy of the contract amendment that includes language that addresses the Federal
audit clause. The contract amendment sufficiently addresses Recommendation 3.1. In addition,
FLDOE stated it would consider liquidated damages in future contracts in response to
Recommendation 3.2. FLDOE’s assessment contractor limited access to the documentation required
and delayed our audit while receiving more than $52.7 million in Federal Funds. As such, we
maintain that FLDOE needs to modify its contract with its new assessment contractor, Pearson, and
include a liquated damages clause for non-compliance with a Federal audit.
Final Report
ED-OIG/A04I0043                                                                                          Page 20 of 36


                                             OTHER MATTERS


Error Identified in FLDOE’s 2006 Third Grade Reading Test
In 2007, FLDOE discovered a problem with the 2006 third grade Reading test. FLDOE concluded
that the problem was a one-time mistake caused by the accidental misplacement of “anchor” questions
that ensure the FCAT difficulty level is consistent from year to year. In response to the problem,
FLDOE named a technical advisory committee of district officials and testing experts, who in turn
hired the Buros Center for Testing at the University of Nebraska to conduct an independent review of
the issue. In a report released in December 2007, the Center stated that the error was caused by an
“anchor” issue, which had been corrected. Although FLDOE initiated procedures to identify and
correct the issue from 2006, the incident was published in Florida newspapers, and user confidence in
the reliability of FCAT results may have been undermined. The accuracy and reliability of assessment
data are important because if users do not perceive FCAT results as reliable, they may not use the
assessments to identify the specific academic needs of individual students, as intended by Federal law.
In addition, we found discrepancies during our audit, which confirm that FLDOE needs to implement
additional controls to ensure contractors are adhering to contract requirements.

FCAT Specifications Not Part of Contractual Obligations
During the course of our audit, we learned that FLDOE does not consider the FCAT specifications 28
in the contract as a contractual requirement. Specifically, FLDOE officials stated
     •   The specifications are contained in a living document that changes frequently.
     •   FLDOE does not have the manpower or time to change the specifications every time a change
         is found to be necessary.
     •   There are critical tasks in the specifications that are contractual and liabilities will be assessed
         for non-compliance by the contractor.

FLDOE pays its assessment contractor $131.9 million dollars to administer the FCAT contract. It
specifically states in the contract and Request for Proposal that all specifications are part of the
contract terms. Accepting specifications as a contractual requirement is especially important since
FLDOE has just awarded a new contract for $254 million dollars to Pearson beginning in the
2009-2010 assessment year. All aspects of the contract should be adhered to according to the terms of
the contract.




28
  The FCAT Specifications are the documents that explain in detail the policies and procedures the contractor will use
during the assessment scoring period. FLDOE provided seven FCAT specifications for 2008 - Handscoring, Data
Verification, Packing and Distribution, Scoring, Technical Scoring, Reporting, and Pre-ID.
Final Report
ED-OIG/A04I0043                                                                        Page 21 of 36


                    OBJECTIVE, SCOPE, AND METHODOLOGY


The objective of our audit was to determine whether controls over scoring of assessments at the
FLDOE were adequate to provide reasonable assurance that assessment results were reliable.
Our review covered assessments administered in school year 2007-2008 that were used for evaluating
individual student achievement and making Adequate Yearly Progress (AYP) determinations under
Section 1111(b)(3) of the ESEA.

To accomplish the objective of our audit of FLDOE, we

   •   Compiled and reviewed media reports, assessment articles, prior OIG audit reports, prior audit
       reports issued by the Government Accountability Office and the Florida State Comptroller,
       and the monitoring reports and peer review determination letter issued by the Office of
       Elementary and Secondary Education (OESE).
   •   Identified Title VI and Title I, Part A funding and contacted the OESE for additional
       information on FLDOE.
   •   Reviewed the database of errors compiled by the National Board of Educational Testing and
       Public Policy to identify potential risk areas in FLDOE’s system for scoring State assessments.
   •   Reviewed laws, regulations, and guidance applicable to State assessment scoring and student
       privacy rights.
   •   Obtained an understanding of the assessments administered by FLDOE, as well as the control
       system in place to ensure reliable student scores through interviews of relevant personnel.
   •   Observed Retake scanning at FLDOE’s contractor scanning scoring facility in Indianapolis,
       Indiana.
   •   Conducted interviews with personnel from FLDOE and CTB and MI to better understand
       internal control processes and procedures over assessment scoring.
   •   Reviewed requests for proposals, vendor proposals, and contracts for the FCAT, as well as
       documentation supporting FLDOE’s procurement process in relation to the contract.
   •   Reviewed documentation over the background and administration of the FCAT.
   •   Obtained and reviewed FCAT documentation describing the specifications used in processing
       and reporting assessment results.
   •   Reviewed the FCAT Pre-ID process that allows FLDOE to provide pre-labeled booklets for
       students taking paper exams and individual user accounts for online testing.
   •   Reviewed a list of incident reports at FLDOE school districts regarding FCAT data.
   •   Interviewed FLDOE key staff related to ESEA assessments (Reading, Math, Science, and
       Writing+).
   •   Documented FLDOE’s policies, procedures, and controls over the FCAT scoring process.
   •   Interviewed contractor personnel at CTB’s Indianapolis, Indiana, scoring facility.
   •   Reviewed FCAT request for proposals, contractor proposals, evaluations, and contracts.
Final Report
ED-OIG/A04I0043                                                                         Page 22 of 36
   •   Observed and documented controls and security procedures at CALA-FSU and the FLDOE
       Test Control Facility.
   •   Observed scanning operations at the CTB central scanning facility, shipping warehouse, and
       cutting facility in Indianapolis, Indiana.
   •   Documented policies and procedures related to CTB’s scanning procedures.
   •   Obtained a copy of the raw data file from FLDOE with the universe of 1.8 million FCAT test
       results (Reading, Math, Writing, and Science) for use in selecting a sample.
   •   Obtained a copy of the electronic test define (answer key).
   •   Conducted a partial review of the CTB assessment contractors training records.
   •   Reviewed documentation provided to support monitoring of contractor services.
   •   Issued subpoenas to CTB, MI, and Kelly Services for written policies and procedures, reader
       qualifications documentation, and training records.
   •   Reviewed inter-rater reliability for compliance with quality assurance requirements established
       in the contract and Scoring Specifications between FLDOE and CTB.
   •   Reviewed qualification documentation provided by MI for the 647 readers who participated in
       training to score the Spring 2008 FCAT Writing assessment. From the documentation
       provided, we verified the degrees obtained from non-foreign educational institutions.

We selected a random sample of 50 Reading and Mathematics FCAT scored answer records from a
population of 1,545,696 scored Reading and Mathematics FCAT records. Our combined sample of 50
Reading and Mathematics tests should have totaled 100 single test and answer booklets (50 Reading
plus 50 Mathematics) but instead totaled 84 single test and answer booklets because the sixth, seventh,
and ninth grade booklets were one combined Reading and Mathematics test. We compared the
hardcopy student answer documents with the electronic data file records provided by FLDOE to
ensure accurate scanning and that the scanned data maintained its integrity during export to the
contractor’s system. We identified 9 discrepancies related to the 50 FCAT Reading and Mathematics
scored records as detailed in Finding No. 1.

We did not use information from a database in order to address our objectives. We used data supplied
by FLDOE, CTB, MI, and Kelly Services in Microsoft Excel spreadsheets to conduct our testing.

We reviewed the sufficiency of FLDOE’s contractual safeguards for protecting student information
during the assessment process in accordance with the Family Educational Rights and Privacy Act
(FERPA). We found the relevant FERPA student information statements were contained in the
FLDOE contracts.

We began our audit work on-site at FLDOE in Tallahassee, Florida, on June 23, 2008. We held an
exit conference with FLDOE officials on June 4, 2009. Our audit was performed in accordance with
generally accepted government auditing standards appropriate to the scope of the review described
above. These standards require that we plan and perform the audit to obtain sufficient, appropriate
evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives.
We believe that the evidence obtained provides a reasonable basis for our findings and conclusions
based on our audit objectives.
Final Report
ED-OIG/A04I0043                                                                          Page 23 of 36

                                   Enclosure 1: Glossary
Anchor Sets – According to FLDOE’s contract between CTB, subcontracted to MI, “sample essays
which clearly exhibit the criteria for each score point.”

Blind Scoring – According to FLDOE’s contract with MI, “a method of scoring that allows each
student response to be scored independently, without the reader having knowledge of any previous
scores.”

Equating – According to the Standards for Educational and Psychological Testing, it places two or
more essentially parallel tests on a common scale.

Gridded Response – Questions that require students to solve problems and “bubble” or shade their
numerical answers in answer grids. Answers may be gridded using several correct formats. Students
must accurately fill in the bubbles below the grids to receive credit for their answers.

Inter-rater Reliability – According to the Standards for Educational and Psychological Testing, the
consistency with which two or more judges rate the work or performance of test takers.

Psychometrician – According to Margin of Error: The Education Testing Industry in the No Child
Left Era, a report published by Education Sector, experts trained in measurement theory and statistics
who analyze how test items perform in field trials and conduct other statistical analyses in test-
making.

Scale Score – The score used to report student results for the entire test in FCAT Reading,
Mathematics, and Science. Scale scores on the FCAT range from 100 to 500 at each grade level.
The scale score is the result of Item Response Theory processing and equating.

Reliability – According to the Standards for Educational and Psychological Testing, the degree to
which test scores for a group of test takers are consistent over repeated applications of a measurement
procedure and hence are inferred to be dependable and repeatable.

Rubrics – FLDOE’s contract with CTB (MI), “scales (currently six-point) used to determine the
range of excellence in the papers written in response to a given prompt.”
Final Report
ED-OIG/A04I0043                                 Page 24 of 36
                  Enclosure 2: FLDOE Response
Final Report
ED-OIG/A04I0043                                                                           Page 25 of 36
FLDOE Response to USED FINDING NO. 1 – Florida Comprehensive Assessment Test (FCAT)
                          Gridded-Response Discrepancies
We disagree with this finding. FLDOE employs an answer document decoding process that uses
nationally recognized professional and technical standards and state-of-the-art scanning technology to
ensure that assessment results are reported accurately. Four points related to this finding need to be
clearly articulated.

       1. It is our opinion that the methodology used by the auditing team to validate the scanning
          accuracy of gridded responses on FCAT answer documents was not in-depth enough to
          arrive at the conclusions cited in Finding No. 1. Specifically, the auditors determined that
          scanning discrepancies existed by viewing copies of answer documents rather than
          reviewing and/or re-scanning original documents. Copies are susceptible to varying
          degrees of shading, and the human eye cannot consistently and accurately discriminate
          bubble intensity to the fine degree that today’s scanning technology can. In addition, to
          ensure that scanners decode student responses instead of printed information, scannable
          books are printed in light shades of ink, usually gray. Due to this, these pages do not
          photocopy well and it would be difficult for the human eye to discern an erasure from a
          stray particle.

       2. Current practice is not fully or accurately described in the audit report as it relates to this
          finding. Specifically, the Scoring Operations Specifications for Spring 2008 FCAT (see
          Exhibit 1) clearly detail that the scanners used by the scoring contractor (CTB) have built-
          in checks for miscalibration. Real-time hardware bias checking is used to verify that
          scanner calibration is maintained during the scanning process. Additional checks, both
          programmatic and procedural, are implemented by CTB to reinforce the built-in hardware
          checks and to ensure optimal scanner setup. Operations follow standard procedures for
          scanner calibration, including the recalibration of scanning machines every 5000 scans.
          Additionally, FLDOE personnel are present at the scoring contractor’s scanning site at the
          beginning of each administration’s scanning process to perform an early scan check during
          live scoring to ensure scoring accuracy. Student demographic data and response arrays on
          original answer documents are compared to the electronic scan file to ensure that
          documents are being accurately scanned.

       3. We have two processes in place at the end of the scoring and reporting process that resolve
          any scanning errors missed during the front-end checks. The first process involves an
          automatic manual rescore of tests for Grades 12-Adult Retake students who have failed the
          FCAT Reading or Mathematics test by 25 or fewer scale score points. This ensures that
          incomplete erasures did not affect the outcome for any student who needs a passing score
          for graduation. The second process allows any district to submit a request through a web-
          based system to have student scores reviewed for scoring anomalies or scanning problems.
          This process is well-advertised, available for every test
Final Report
ED-OIG/A04I0043                                                                           Page 26 of 36
           administration, and open to anyone who would like to request it through his or her district
           testing office.

       4. FCAT Test Administration Manuals and test materials contain detailed instructions
          that are read aloud to students prior to all test administrations. The directions (see
          Exhibit 2) clearly instruct students on responding to multiple-choice items, erasing
         completely, completing gridded-response items, and checking their marks to ensure
         they have responded in the correct manner. In addition, FLDOE distributes Sample
         Test Materials to all schools prior to test administration dates which reinforce correct
          answer-marking strategies and give students an opportunity to practice marking
          answers in the correct manner.

Exhibit 3, prepared by our administration contractor for the 2008 FCAT, shows the result of
researching two answer documents that were cited as problematic by the auditors. This
document clearly confirms that the scanner accurately decoded the bubbles on the documents.

The rules for decoding marks on answer documents are shown in Exhibit 1. The minimum intensity
level for a bubble to be picked up is 5 out of the possible 15 intensity levels. Many erasures meet the
threshold of intensity level 5 or higher because students do not erase completely. The threshold is set
at level 5 in the best interest of students. If the threshold were set lower than level 5, more erasures
would be decoded as intended marks. If it were set higher than level 5, fewer intended lighter marks
would be counted as valid marks.

Scanners are programmed to choose the darkest mark whenever two answer bubbles are darkened in
the same multiple-choice item or in the same column in a gridded-response item if the intensity levels
of the two marks are two or more intensity levels apart. In the case of gridded-response items, if an
erasure is the only bubble gridded in a specific column, it will be picked up as a valid mark if it meets
the intensity level thresholds described in Exhibit 1.

Recommendations from the Auditors

Regarding Recommendation 1.1:
        1.1 Ensure the contractor is correctly setting the Technology Intensity Calibration Algorithm
to capture students’ gridded responses in the scanner. For responses that are manually entered, have
a second verification of the entry to ensure the gridded-response items are captured correctly.

As indicated in Exhibit 1, we require that scanners be calibrated after every 5000 sheets. This entails
running a calibration sheet through the scanner to ensure that the same level of sensitivity is used
when all documents are scanned. Our current scoring contractor, Pearson, will be required to fulfill
this requirement. As an additional assurance, we will require that the contractor produce a report to
show the frequency of the scanner calibrations. This will provide documentation validating that
required calibration is completed and guarantee the consistency of the scanning process across all
documents.
Final Report
ED-OIG/A04I0043                                                                       Page 27 of 36

While issues regarding manually entered responses are not referenced in Finding No. 1, we understand
the auditors’ recommendation in that there are potential quality control issues with key entry data.
Standard procedures for responses entered manually require contractor staff to key enter student
responses when scanning errors occur. Our Scoring Specifications and contractor procedures then
require subsequent verification of the entry. When FLDOE staff are onsite, this practice is among
those monitored for compliance.

Regarding Recommendation 1.2
       1.2 Implement procedures to test a sample of the gridded responses during live scoring to
ensure students’ gridded responses are accurately scanned.

The Department has procedures in place to perform this task. Prior to each administration, we send a
team to the scanning/scoring site to verify that the early scanning of documents is meeting our
standards. We check the scanning accuracy of documents from at least four districts for all subjects
and grade levels. Gridded responses, multiple marks, blank answers, and student demographic
information are reviewed for accuracy. We print documents based on the electronic record provided
by the scan and/or post-edit files and compare these printed documents to the students’ original
answer documents submitted for scoring.

EXHIBIT 1: Excerpt from the Scoring Operations Specifications for Spring
           2008 FCAT

Scanning and document processing
CTB will scan the FCAT answer documents on 5000i scanners using well established scanning rules
chosen for Florida. The scanners will pick up carbon marks made by the student on each page. These
marks will be resolved into valid marks or invalid marks. Invalid marks are classified as blanks or
multiple-marks. The minimum threshold for a valid mark is set to CTB’s standard intensity level 5.

D.1 Intensity Resolution Algorithm
The Florida Department of Education (FLDOE) has requested a customized solution for resolving
multiple marks on selected response items in scannable documents.
CTB Technology defined logic:
   The following defines CTB’s understanding of the algorithm:
      • A carbon mark of intensity 4 or lower is considered No Mark.
       • A carbon mark with intensity 5-15 is considered a valid mark.
Excerpt form the Technology Appendix to the Scoring Specifications:


The software applies the following logic to resolve multiple marks for all scanned FCAT documents.
       • Rule 1: Marks in the No Mark range are ignored.
       • Rule 2: If there is no bubbled mark, the item is considered as a Blank.
Final Report
ED-OIG/A04I0043                                                                                 Page 28 of 36

        • Rule 3: If there is one bubbled mark, the item is considered as a Single Mark.

   If there are two or more bubbled marks then the darkest two marks are considered in the following
   rules:

        • Rule 4: If the difference in the two intensities is equal to or greater than 2 (e.g., [5, 7] or [13,
        15] or [5, 8]) then the item is considered as a Single Mark and resolved to the bubble with the
        higher intensity.
        • Rule 5: If the difference in the two intensities is less than 2 (e.g., [5, 6], [13, 14], [6, 7]) then
        the item is considered as a Double/Multiple Mark.
        • Refer to the table below for examples.


Note:
        • Documents that must be updated or inserted manually cannot be included in this analysis.
        This would include, but not be limited to, documents that are folded, crumpled, bent, stapled,
        or torn.


   Operations Process for key entry:
             o If there is a multiple mark, and the clerk can determine which mark the student
             intended to make, they will key enter just that mark.
                o If there are multiple marks for an item and the clerk can not determine which mark
                the student intended to make, they will key enter “M” (for multiple mark). The “M” is
                later converted to the normal multiple mark symbol by the system.

The following table shows examples of the application logic that resolves multiple marks for all
scanned FCAT documents:
Final Report
ED-OIG/A04I0043                                                                                Page 29 of 36

                                 Multiple Marks Resolution Algorithm (Examples)
Highest intensity response   Second highest        Valid response and         Scoring Comments
choice                       intensity response    Erased response
                             choices
15                           15, 14                  Not counted for erasure    Considered multiple mark
15                           13, 12,                 Valid response = 15        Considered as erasure
                             11,10,9,8,7,6,5,4,3     Erasure = Second highest
                                                     intensity
14                           14, 13                  Not counted for erasure    Considered multiple mark
14                           12,11,10, 9, 8, 7, 6,   Valid response = 14        Considered as erasure
                             5, 4, 3                 Erasure = Second highest
                                                     intensity
13                           13, 12                  Not counted for erasure    Considered multiple mark
13                           11,10, 9, 8, 7, 6, 5,   Valid response = 13        Considered for erasure
                             4, 3                    Erasure = Second highest
                                                     intensity
12                           12, 11                  Not counted for erasure    Considered multiple mark
12                           10, 9, 8, 7, 6, 5, 4,   Valid response = 12        Considered as erasure
                             3                       Erasure = Second highest
                                                     intensity
11                           11,10                   Not counted for erasure    Considered multiple mark
11                           9, 8, 7, 6, 5, 4, 3     Valid response = 11        Considered as erasure
                                                     Erasure = Second highest
                                                     intensity
10                           10, 9                   Not counted for erasure    Considered multiple mark
10                           8, 7, 6, 5, 4, 3        Valid response = 10        Considered as erasure
                                                     Erasure = Second highest
                                                     intensity
9                            9, 8                    Not counted for erasure    Considered multiple mark
9                            7, 6, 5, 4, 3           Valid response = 9         Considered as erasure
                                                     Erasure = Second highest
                                                     intensity
8                            8, 7                    Not counted for erasure    Considered multiple mark
8                            6, 5, 4, 3              Valid response = 8         Considered as erasure
                                                     Erasure = Second highest
                                                     intensity
7                            7, 6                    Not counted for erasure    Considered multiple mark
7                            5, 4, 3                 Valid response = 7         Considered as erasure
                                                     Erasure = Second highest
                                                     intensity
6                            6, 5                    Not counted for erasure    Considered multiple mark
6                            4, 3                    Valid response = 6         Considered as erasure
                                                     Erasure = Second highest
                                                     intensity
5                            5, 4                    Not counted for erasure    Considered multiple mark
5                            3                       Valid response = 5         Considered as erasure
                                                     Erasure = 3
Final Report
ED-OIG/A04I0043                                                                            Page 30 of 36
EXHIBIT 2: Directions for students
Directions for Completing the Response Grid
Note: These directions are located in Sample Test Materials, Test Administration Manual scripts read
to students, and in student test documents for students to reference during testing.

1. Work the problem and find an answer.

2. Write your answer in the answer boxes at the top of the grid.

       • Print your answer with the first digit in the left answer box, OR with the last digit
         in the right answer box.
       • Print only one digit or symbol in each answer box. Do NOT leave a blank answer
         box in the middle of an answer.
       • Be sure to write a decimal point or fraction bar in the answer box if it is part of
         the answer.

3. Fill in a bubble under each box in which you wrote your answer.
         • Fill in one and ONLY one bubble for each answer box. Do NOT fill in a bubble
           under an unused answer box.
         • Fill in each bubble by making a solid black mark that completely fills the circle.
         • You MUST fill in the bubbles accurately to receive credit for your answer.

You may refer to the Directions for Completing the Response Grid at any time while taking the
mathematics test.
Additional directions for students printed in test documents
The third symbol is a pencil and a grid. This symbol appears next to questions that require you to fill
in your answer on a grid in your Answer Book. Answers may be gridded using several correct
formats. You MUST fill in the bubbles accurately to receive credit for your answer.
Excerpt from the test administration script
Now, read silently as I read aloud the Directions for Completing the Response Grid at the bottom of
the page.
(Directions for Completing the Response Grid above are read aloud to students)
You’ll also answer multiple choice questions. For multiple choice questions, choose the best answer
and, in the space provided in your answer book, fill in the bubble for the answer you choose. If you
use your eraser, do so gently so you don’t tear or rip the paper.
        • Mark only the bubble for the answer you choose.
        • Do not make a mark in any of the other bubbles.
        • Do not circle answers. If you circle the answers instead of gridding the bubbles, answers
           will NOT be scored.
        • When you have completed this session, check through your answers to make sure you have
           filled in only one bubble for each question.
Final Report
ED-OIG/A04I0043                                                                           Page 31 of 36
FLDOE’s Response to USED FINDING NO. 2 – Insufficient Monitoring of Florida Department
                        of Education’s (FLDOE’s) Contractor

Inadequate Safeguards for Discarding FCAT Student PII

The FLDOE understands and enforces the protection of Personally Identifiable Information (PII).
Staff take extraordinary precautions to ensure that PII is treated confidentially and destroyed in a
secure manner when necessary. Further, we require all entities that contract with the Department,
including assessment contractors, to similarly protect PII.

Section IX. of Attachment C in the CTB contract states:

       It is understood and agreed that, in accordance with the Family Educational Rights and Privacy
       Act (FERPA) and the Children’s On-line Privacy Protection Act (COPPA) it is the intention of
       the parties that personal information, such as student names, grade level and/or other
       identification of students, including students who may be under the age of thirteen, and other
       data concerning students needed to determine or report test answers, test scores, and/or test
       result analyses (hereinafter, “Personal Data”), will be held in confidence by the parties and, if
       gathered or transmitted on-line, whether directly or indirectly, will be deemed gathered or
       transmitted by or under the authority of the Department, including transmission between the
       computers or servers of or under the control of the Department or the Contractor. With respect
       to such Personal Data, it is agreed that all such Personal Data shall be the property of the
       Department and shall be gathered and transmitted to the Contractor by or under the authority
       of the Department in order to enable the Department to assess the performance of the students
       to whom the test is administered and to carry out the Department’s educational or other
       responsibilities under law.

The fact that a specific CTB employee discarded PII in a non-secure fashion, albeit in a secure area,
does not merit the conclusion that, “FLDOE did not sufficiently monitor its assessment contractor”.
FLDOE Scoring and Reporting personnel are present at scanning sites during each FCAT
administration’s scoring process to ensure that all aspects of the scanning operation proceed according
to our contractual agreement and that related requirements are met. While we regret that this one
incident occurred, we do not think this one instance warrants the audit team’s conclusion.

Non-compliance with Contract Terms

Change of Subcontractor Without Notification

The FLDOE disagrees with the audit team’s conclusion that CTB’s change from Kelly Services to
Apple One for hiring operations violated contractual requirements.


Section 5.31 (Contractual Information) of the Request for Proposal 2005-001 states:
Final Report
ED-OIG/A04I0043                                                                          Page 32 of 36
       To the extent that the contractor’s proposal included named subcontractors, the contractor shall
       utilize the named subcontractors in the manner prescribed in its proposal. Any change of
       subcontractors must be approved in advance by the Department. In the event of poor
       performance by a subcontractor, the Department reserves the right to direct the contractor to
       replace that subcontractor.

End Excerpt from the Request for Proposal --

The following chart of subcontractors contained in CTB’s proposal indicates which subcontractors
were identified to provide the requirements described in the RFP. Please note that neither Kelly
Services nor Apple One is listed on this chart. CTB senior leadership has explained that they
considered their long-standing arrangements with these agencies for hiring temporary employees
(such as scanning staff and handscoring staff) as part of CTB’s corporate structure, and were not
required to list them per the RFP. The Department agrees. It is not unreasonable that CTB requested
FLDOE approval only for changes to subcontractors specifically identified in the proposal as
responsible for the actual work associated with the FCAT program.




In addition, RFP 2008-07 states:

       A separate chart in the proposal will identify all of the subcontractors proposed to be involved
       in the program, the assessment components they will be involved with, and the specific
       services they will provide. Should changes be proposed in subcontractors after the contract
       award, new subcontractors must be approved by the Department throughout the life of the
Final Report
ED-OIG/A04I0043                                                                          Page 33 of 36
       contract. Not less than sixty (60) days in advance of a proposed subcontractor’s, start of work
       on program processes or deliverables, the contractor will provide the Department with a
       qualifications and experience summary for review and consideration for acceptance. As part of
       the Annual Report, all subcontractors, their services, and Departmental approval status will be
       listed.

       A detailed description of the subcontracted services to be provided under the ensuing contract
       must be included. All subcontracted services will be documented. Each service (e.g., printing,
       computer operations, test development, psychometric services, legal counsel, quality control,
       reporting, etc.) should be fully described including resource, level of service, scheduling and
       quality commitments. The contractor will identify the proportion of materials, etc., to be
       generated by any and all involved subcontractors.

FLDOE believes that this more specific language regarding subcontractors will be sufficient to
monitor our current assessment contractor. In addition, the nature of subcontracted work (i.e., directly
working on assessment-related activities) is clearly stated.

No Disaster Recovery Plan

A Disaster Recovery Plan was not available at the assessment contractor scoring site for review by the
audit team. We concur that the contractor should have been able to provide the audit team a copy of
this plan, and have requested written assurance from our current assessment contractor that a Disaster
Recovery Plan is readily available for review at all scanning sites. Verification of a Disaster Recovery
Plan will be added to the checklist used by FLDOE when monitoring scoring sites for each test
administration.

Insufficient Monitoring of Subcontractors

FLDOE fulfilled its contractual obligation to review a random 20% sample of the scorer candidates’
qualifications, and the results are on file at the Test Development Center. A review of these records
indicates that follow-up was requested for candidates that appeared to not have the required
qualifications. The subsequent investigations resulted in verification and approval of candidate
qualifications, or dismissal of the candidates. As a result of these reviews, additional candidate IDs
were sampled to further ensure a qualified pool. It is possible that a review of the entire pool of
candidates would have resulted in findings similar to the audit findings, however, a 20% sample was
agreed upon by FLDOE and CTB in recognition of the critical time factor and human resources
required to conduct the reviews.
FLDOE and Test Development Center staff have documented that all contractually-required controls
were followed. In addition, monitoring occurred that went beyond the contractual requirements.
Evidence of this is in the documentation previously provided to the audit team detailing the extended
on-site visits to handscoring sites by FLDOE and TDC staff, and in the detailed notes on daily
handscoring conference calls among the handscoring sites and FLDOE/ TDC leadership.
Final Report
ED-OIG/A04I0043                                                                           Page 34 of 36
Inadequate Document Control Procedures

The assessment contractor complied with all document control procedures called for in the assessment
contract for the 2008 administrations. We require our assessment contractors to be able to retrieve
(within a 30 day period) assessment documents from the current year’s administration. Documents
from prior years are required to be archived in a secure storage facility, but there is no requirement
imposed on the assessment contractor to archive these documents in such a way that retrieval can be
accomplished within a period of time less than 30 days. While the audit team’s experience indicates
some corporate inefficiencies, FLDOE’s needs for document retrieval have been met.

Recommendations from the Audit Team

Regarding Recommendation 2.1:
       2.1 Use unique identifiers instead of student social security numbers and dates of birth on
assessment documents.

The FLDOE will explore the potential use of alternate identifiers for students. However, at this point,
we will continue to utilize the process currently in place to ensure timely and accurate reporting of
student results and maintain stringent security policies and procedures. The current process has
contributed to the accuracy and power of Florida’s nationally recognized longitudinal data system.

Regarding Recommendation 2.2
        2.2 Ensure that all contractors are aware of proper handling of PII and include language in
their contracts to properly address the correct handling procedures related to the disposal of PII.

We believe that our current contract with Pearson is clear on this issue in several places.

Contract 09-658 – Attachment C states:

       IX. The Contractor further warrants that the Contractor shall not disclose to any third party,
       without the express, prior, written approval of the Department, any personally identifiable
       information about any student. This applies to information which came from any record or
       report of a Florida public education institution or from any education record which is subject
       to the Family Educational Rights and Privacy Act, 20 U.S.C. Section 1232g. The terms
       “record a report” and “student” shall have the meanings prescribed in s. 1002.22(2)(c) and (d),
       F.S. The term “educational record” shall have the meaning prescribed in 20 U.S.C. Section
       1232g(a)(4).
Further, the RFP 2008-17, Section 4.4 addresses the disposition of materials:

       Destruction of secure documents and CBT student response files must be requested in writing
       and authorized by the Department. The contractor and all subcontractors must submit
       certificates of destruction that describe in writing the specific items/files destroyed.
Final Report
ED-OIG/A04I0043                                                                          Page 35 of 36
It is our opinion that the above excerpts provide clear and appropriate requirements for the handling of
PII.

Regarding Recommendation 2.3
       2.3 Monitor the contractor to ensure compliance with contract provisions and include a table
of penalties in the contract for non-compliance with contractual requirements.

FLDOE has several quality assurance measures including Performance Metrics to hold our contractor
accountable for contractual obligations. If requirements are not met, payment is withheld as stated in
the following excerpt from Contract #09-658 – Contract Attachment C (FLDOE’s current assessment
contract with Pearson).

       II. The Contractor shall prepare an invoice for the amount due and mail it to the Department of
       Education Comptroller after having delivered the products and services required under this
       Contract to the Contract Manager. The invoice shall set forth details sufficient for a proper
       pre-audit and post-audit including, where applicable, the products and services delivered and
       completion dates. Upon receipt of the invoice, the Department of Education Comptroller will
       request confirmation from the Contract Manager that the delivered products and services are
       satisfactory and payment is due. If for any reason they are not satisfactory, payment will be
       withheld until the unsatisfactory condition or conditions are corrected. Upon receipt of the
       Contract Manager’s approval, the Department of Education Comptroller shall process each
       invoice in accordance with the provisions of s. 215.422, F.S.

Regarding Recommendation 2.4
      2.4 Monitor document control procedures at the contractor facilities at least annually.

The FLDOE has generated additional specific items and processes that will be included in the
monitoring checklist FLDOE personnel will use when visiting contractor scanning sites. One of these
monitoring items will be to verify that the procedures used by the contractor to manage and dispose of
PII and ensure that no PII is disclosed or mishandled. FLDOE staff will be on site to monitor these
processes during each Fall Retake administration, and twice during the Spring administrations (once
for Spring Retake and grade 3 assessments, and once for grades 4-11 assessments).
Final Report
ED-OIG/A04I0043                                                                           Page 36 of 36
 FLDOE’s Response to USED FINDING NO. 3 – FLDOE’s Contractor Delayed Federal Audit
                    by Limiting Access to Assessment Documentation

Notwithstanding the omission of such a clause in 2008 contracts, the FLDOE and its assessment
contractor did not limit access to documentation required to address the objectives of the Federal
audit. From the initial site visit by the auditing team to the conclusion of the audit, all requested
information and data regarding statewide assessments in Florida were provided to the auditing team.
Multiple strategies were suggested to the audit team on how they might be able to physically check a
representative sample of original answer documents against the electronic scan file during live
scanning. However, the audit team requested the two-year-old documents from a secure warehouse
where processed documents are stored and archived. Due to the nature of this request, the contractor
may have taken longer to process this request than desired by the audit team, but at no point was
access denied or limited.

Recommendations of the Audit Team

Regarding Recommendation 3.1:
      3.1 Include a Federal audit clause provision in contracts for Department funded programs.

Our contracts have been modified to include the following federal audit clauses. This language is
included in the recently executed Pearson/FCAT 2-EOC contract in Attachment A, Section III, D and
E.
       D. Section XIV. of Attachment C: The Contractor shall grant access to all records pertaining
          to the Contract to the Department’s Inspector General, General Counsel and other agency
          representatives, the State Auditor General, the Office of Program Policy and Government
          Accountability, the Chief Financial Officer and Federal Funding Agency (or its authorized
          representative).
       E.    Section XV. of Attachment C: The Contractor agrees to permit onsite visits by designated
            Department employees or agents (including any Federal Funding Agency and its
            authorized representative) to conduct audits or to ensure compliance with Section 20.055,
            Florida Statutes. These audits may require Department access to records and data,
            computers and communications devices, and other materials whether owned or operated by
            the Contractor. Access may include, but is not limited to, user level and/or system level
            access to any computing or communications device; access to information (electronic,
            hardcopy, etc) that may be produced, transmitted or stored on the Contractor's equipment
            or premises; access to work areas; and access to interactively monitor and log traffic on the
            Contractor's networks.

Regarding Recommendation 3.2:
      3.2 Include a table of penalties in the contract for non-compliance with a Federal audit.
The Department will review the liquidated damages section of our contracts and determine whether or
not penalties for non-compliance should be included in future contracts.