oversight

Audit of the Department's Performance Based Data Management Initiative.

Published by the Department of Education, Office of Inspector General on 2005-09-29.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                     Audit of the Department’s 

            Performance Based Data Management Initiative



                                    FINAL AUDIT REPORT




                                         ED-OIG/A11-E0003

                                           September 2005 



Our mission is to promote the efficiency,                 U.S. Department of Education
effectiveness, and integrity of the Department’s          Office of Inspector General
programs and operations.                                  Information Technology Audits Division
                                                          Washington, DC
Statements that managerial practices need improvements, as well as other conclusions, and
recommendations in this report, represent the opinions of the Office of Inspector General.
Determination of corrective action to be taken will be made by appropriate Department of
Education officials.

In accordance with the Freedom of Information Act (U.S.C. Section 552), reports issued by the
Office of Inspector General are available if requested; to members of the press and general
public to the extent information contained therein is not subject to exemptions in the Act.
                      UNITED STATES DEPARTMENT OF EDUCATION

                                      OFFICE OF THE INSPECTOR GENERAL




MEMORANDUM                                                                                    SEP 2 9 2005


TO: 	           Tom Luce
                Assistant Secretary for Planning, Evaluation and Policy Development


FROM: 	         Thomas ·A. Carter, Depmy Inspector General
                Office of Inspector General

SUBJECT: 	 FINAL AUDIT REPORT
           Audit of the Department's Performance Based Data Management
           Initiative
           Control No. ED-OIG/AI1-E0003


Attached is the subject final audit report that covers the results of our review of the
Department's Performance Based Data Management Initiative. An electronic copy has
been provided to your Audit Liaison Officer. We included your fom1al response to our
draft report in Appendix A of the final report.

Corrective actions proposed (resolution phase) and implemented (closure phase) by your
office will be monitored and tracked through the Department's Audit Accountability and
Resolution Tracking System (AARTS). ED policy requires that you develop a final
corrective action plan (CAP) for our review in the automated system within 30 days of
the issuance of this report. The CAP should set forth the specific action items, and
targeted completion dates, necessary to implement final corrective actions on the findings
and recommendations contained in this final audit report.

In accordance with the Inspector General Act of 1978, as amended, the Office of
Inspector General is required to report to Congress twice a year on the audits that remain
unresolved after 180 days from the date of issuance.

In accordance with the Freedom oflnformation Act (5 U.S.C. §552), reports issued by
the Office of Inspector General are available to members of the press and general public
to the extent information contained therein is not subject to exemptions in the Act.

We appreciate the cooperation given us during this review. If you have any questions,
please call David Cole of my staff on 202-245-6917. Please refer to the control number
in all correspondence related to the report.

Attachment


                                  400 MARYLAND AVE., S.W., WASHINGTON, DC 20202-1510
                                                           www.ed.gov

          Our mission is to ensure equal access to education and to promote educational excellence throughout the nation.
                                       Executive Summary


The Office of the Inspector General (OIG) audited the Performance Based Data Management
Initiative (PBDMI) project and evalua ted the Project Management Team’s (PMT) Education
Data Exchange Network (EDEN) 1 systems implementation to determine whether significant
implementation and project management risks have been effectively mitigated to ensure that
PBDMI will meet its investment goals of consolidating current data collection activities in a
way that improves data quality and reduces the reporting burden for national education
partners. Other audit objectives include determining whether system development
methodologies are ensuring that system functionality requirements have been adequately
defined and production systems meet end user performance requirements; project management
and contract monitoring procedures have been implemented to ensure that project milestones
are met and contract deliverables are performed; and the Department’s EA has been
consistently updated to support the technical requirements and business processes of PBDMI.

The Office of Management and Budget (OMB) has recognized that State and Local Education
Agencies (SEA/LEA) engage in burdensome data reporting processes to comply with the
Department’s multiple education collection and reporting activities. Accordingly, OMB tasked
the Department of Education (Department or ED) to develop an initiative that will create a
web-based system for collecting timely student achievement data, and eliminate unnecessary
and burdensome reporting of education data. In response, PBDMI was established and is
considered critical to the success of several of the Department’s programs, including the
implementation of the Elementary and Secondary Education Act, as amended by the No Child
Left Behind Act of 2001 (P.L. 107-110). Also, PBDMI supports the Department’s goals of
creating a culture of achievement and establishing management excellence.

Although the EDEN systems will not be fully developed and implemented until fiscal year
2006, the PMT indicates that the project has achieved success in a number of areas:

•	 The PBDMI Decision Support System (DSS) Pilot successfully collected large amounts of
   education data (i.e. over 2,200 files containing 63 different file types) from 50 SEAs via the
   Internet;
•	 The PMT met major investment goals by developing and implementing a production
   EDEN Submission System within original milestone dates;
•	 The EDEN Submission System has received education data submissions from 50 states;
   and
•	 The PMT has defined over 140 common data elements and definitions that will be used by
   the Program Managers and SEAs for collecting and reporting education information.

However, our audit determined that the PMT needs to improve certain system implementation
and project management controls to ensure that PBDMI will meet intended project goals.
Specifically, we noted: sufficient data input controls have not been implemented to ensure that
data is complete, accurate, and valid; versions of the EDEN Submission System were

1
 The Department is using PBDMI to develop and implement EDEN; a database repository system that will allow
users to obtain organized and formatted information about the status and progress of education programs in the
States, districts and schools.
Audit of the Department’s Performance                   1                                Final Audit Report
Based Data Management Initiative
ACN: ED -OIG/A11-E0003
implemented that did not meet user system functionality and performance requirements;
standard data elements have not been fully defined to ensure that PBDMI data collection can
replace the Department’s numerous data collections efforts; inadequate user training and
documentation has resulted in ongoing data submission errors; insufficient project management
controls resulted in milestone dates that were not met and payments were made to certain
SEAs without evidence that contract deliverables were performed.

We also identified deficiencies in the Department’s Enterprise Architecture (EA) that will not
facilitate the success of PBDMI. For example, a PBDMI Meta Data dictionary has not been
adopted into the Department’s EA to adequately define standard data elements, data definitions
and the business rules associated with collecting those data elements. Additionally, we noted
that the EA Technical Reference Model has not been fully developed to establish technical
standards, products, and services that are required to build systems in support of the
Department’s required business functions and information security requirements. As a result,
the PBDMI project is at risk of not meeting its goals of consolidating the collection of
education information in a way that improves data quality and reduces the paperwork burden
for national education partners.

We recommend that the Deputy Secretary devote additional resources to ensure the PBDMI
PMT can adequately address the system development risks identified in this report.
Additionally, we recommend that the Director, Strategy Accountability Service and the PMT:

•	 Provide guidance to SEA/LEAs in developing effective data input controls (i.e. edit checks,
   business processes) to verify the quality of education data at the source level and to ensure
   that data quality is maintained during the collection and data transmission process.
•	 Implement comprehensive requirements and configuration management controls to develop
   EDEN integrated systems that will meet system functionality and performance
   requirement s, as well as encourage a proactive system development approach.
•	 Develop a comprehensive approach to enable the PMT to proactively define, collect, and
   standardize a complete listing of standard data elements and definitions.
•	 Develop comprehensive training programs and guidance to assist SEAs in reducing
   ongoing data submission errors with the EDEN Submission System.
•	 Develop comprehensive project management controls to effectively manage the timely
   completion of project milestones and oversee multiple contractors responsible for
   developing the EDEN integrated systems.
•	 Coordinate with the Department’s Enterprise Chief Architect (ECA) to adopt a PBDMI
   Meta Data dictionary as part of the Department’s EA to define standard data elements, data
   definitions and the business rules associated with collecting those data elements.

We recommend that the Department’s Office of the Chief Information Officer (OCIO) ensure
that the Department’s ECA fully develops the EA Technical Reference Model to establish
technical standards and information security requirements that are needed to build systems in
support of the Department’s critical operations. Additionally the Department’s ECA needs to
improve the Department’s current EA and clearly define its Target Architecture and Transition
Plan to ensure it is aligned with the OMB Federal EA.

During the audit and following the issuance of our Draft Audit Report, Department officials
have taken steps to strengthen system development controls (e.g., data quality, enterprise
Audit of the Department’s Performance           2                         Final Audit Report
Based Data Management Initiative
ACN: ED -OIG/A11-E0003
architecture, and configuration management controls ) for PBDMI and devoted additional
resources to the project. In their response to our draft report, the Assistant Secretary for
Planning, Evaluation and Policy Development, on behalf of the Deputy Secretary, generally
concurred with our findings and recommendations, and described specific actions taken and
additional planned actions to address our concerns. The Department did not respond to
recommendation 1.8 (Provide system development guidance that would address the risks
associated with using the RAD methodology) in the report. We request that the OCIO develop
a corrective action plan to address the outstanding recommendation. The Assistant Secretary
for Planning, Evaluation and Policy Development response is included in its entirety in
Appendix A of this report.




Audit of the Department’s Performance         3                         Final Audit Report
Based Data Management Initiative
ACN: ED -OIG/A11-E0003
                                        Table of Contents


Objectives, Scope, and Methodology                                                              5

Background                                                                                      7

Finding No. 1: PBDMI System Implementation
Controls for the Development of the EDEN System Need Improvement                            11

Finding No. 2: PBDMI Project Management Controls Need
Improvement                                                                                 22

Finding No. 3: Enterprise Architecture Shortfalls Need to be Addressed to
Ensure PBDMI Success                                                                        25

Other Matters and Statement on Internal Controls                                            34

Appendix A. Department’s Response to Draft Audit Report                                     35


                                 Acronyms Used in This Report

Department or ED            Department of Education
PBDMI                       Performance Based Data Management Initiative
PMT                         Project Management Team
OMB                         Office of Management and Budget
EDEN                        Education Data Exchange Network
EA                          Enterprise Architecture
DARS                        Decision Analysis and Reporting System
OCIO                        Office of the Chief Information Officer
DSS                         Decision Support System
RAD                         Rapid Application Development
C4DQ                        The Center for Data Quality, Inc.
XML                         Extensible Markup Language
SEI-CMMI                    Software Engineering Institute - Capability Maturity Model
                            Integration
C&A                         Certification and Accreditation
GAPS                        Grants Administration and Payments System
ECA                         Enterprise Chief Architect
SEA/LEA                     State and Local Education Agencies




Audit of the Department’s Performance             4                        Final Audit Report
Based Data Management Initiative
ACN: ED -OIG/A11-E0003
                         Objectives, Scope, and Methodology


Objectives

Our audit objectives were to evaluate whether significant system implementation and
investment risks have been mitigated to ensure that PBDMI will meet its investment goals of
consolidating current data collection activities in a way that improves data quality and reduces
the reporting burden for national education partners. In addition, our audit objectives included
determining whether:

•	 system development methodologies are ensuring that system functionality requirements
   have been adequately defined and production systems meet end user performance
   requirements;
•	 project management and contract monitoring procedures have been implemented to ensure
   that project milestones are met and contract deliverables are performed; and
•	 the Department’s EA has been consistently updated to support the technical requirements
   and business processes of PBDMI.

Scope

We conducted our work at applicable Department offices from January 2004 through March
2005. We discussed the results of our audit with PMT and representatives from the OCIO at
an exit conference held on May 16, 2005. We conducted our audit in accordance with
government auditing standards appropriate for the scope of this review.

Methodology

In evaluating the PMT’s system development methodologies, we reviewed data input quality
controls, functionality and configuration management controls, standard data elements, and the
adequacy of end user training and documentation.

•	 Data Quality Controls - To determine whether the PMT has effectively addressed the
   project goals regarding improving data quality, we reviewed data quality evaluation reports
   prepared by The Center for Data Quality, Inc. (C4DQ); interviewed representatives from
   the PMT, SEAs, and the C4DQ regarding data quality and data collection; reviewed OIG
   audit reports that evaluated the completeness, accuracy, validity, and reliability of data
   provided by SEAs and LEAs in connection with the Elementary and Secondary Education
   Act (Title I, Part A) education program and the Perkins Vocational and Technology
   Education Act of 1998 (Perkins III);2 and held discussions with Department Program
   Managers and SEAs to obtain their perspective of PBDMI data quality issues or concerns.

2
  OIG reports reviewed include: 1) A Joint Audit Report on The Status of State Student Assessment Systems and
the Quality of Title I School Accountability Data, August 2002, S14-C0001; 2) Audit of Perkins III Performance
Data at OVAE, May 2004, A03-D0013; 3) Kentucky Department of Technical Education’s Management Controls
Over Perkins III Performance Data Needs Strengthening, October 2003, A04-D0007; 4) Audit of Indiana’s
Management Controls Over Perkins III Performance Data, September 2003, A05-D0012; and 5) Audit of
Florida’s Management Controls Over Perkins III Performance Data, July 2003, A03-C0019.

Audit of the Department’s Performance                  5                              Final Audit Report
Based Data Management Initiative
ACN: ED -OIG/A11-E0003
•	   System Functionality and Configuration Management Controls - To determine whether the
     PMT has effectively addressed project goals of improving efficiency and effectiveness of
     data collections, we reviewed requirements management and configuration management
     weaknesses identified during testing of the DSS Pilot; reviewed Integrated Project Status
     Reports, customer support center help desk records, EDEN Submission System Release
     documents to identify instances where the EDEN Submission System was not meeting end
     user functionality and performance requirements; and int erviewed representatives from the
     PMT to gain an understanding of systemic weaknesses that may prevent PBDMI from
     meeting investment goals.

•	 Standard Data Elements - To determine whether the PMT has proactively and fully defined
   standard data elements so the Department can ultimately discontinue its multiple data
   collection efforts, we interviewed representatives from the PMT, Program Offices, and
   SEAs and reviewed the standardized data elements and data dictionary developed in
   connection with PBDMI and approved by OMB.

•	 End User Training and Documentation - To determine whether the PMT has provided
   sufficient training and guidance to eliminate ongoing data transmission errors, we reviewed
   customer support center help desk logs and interviewed select SEAs to identify instances of
   system difficulties or data transmission errors with the EDEN Submission System.

To determine whether project management and contract monitoring procedures have been
implemented to ensure that project milestones are met and contract expenditures are properly
authorized, we interviewed representatives from the PMT and reviewed the PBDMI Exhibit
300 Business Case, Integrated Project Status Reports, and task order contracts in connection
with PBDMI. In determining whether the Department’s EA is developed in a manner that will
facilitate the success of PBDMI, we reviewed the Department’s EA documents to determine
whether a mature architecture has been defined and interviewed representatives from the PMT
and the OCIO to identify challenges associated with the Department’s EA.




Audit of the Department’s Performance            6                         Final Audit Report
Based Data Management Initiative
ACN: ED -OIG/A11-E0003
                                              Background


OMB recognizes that SEAs and LEAs have engaged in burdensome and paper- intensive
reporting processes to comply with the Department’s multiple, and sometimes redundant,
education collection and reporting activities. In response, OMB allocated $10 million to the
Department in fiscal years 2003 and 2004 and tasked the Department to develop an initiative
that will create a web-based system for collecting timely student achievement data and
eliminate unnecessary and burdensome reporting that divert Federal, State, and local resources
from activities that will improve student learning.

Consequently, the PBDMI project was conceptualized to fundamentally transform the way the
Department conducts its current operations. Within the Department’s fiscal year 2005 Budget
Passback, OMB indicated that PBDMI remained a high priority and the Department’s progress
will be a factor in assessing ED’s performance on the Bud get and Performance Integration
section of the President’s Management Agenda Scorecard. In addition, PBDMI is deemed
critical to the success of several of the Department’s programs and operations, including:

•	 Implementation of the Elementary and Secondary Education Act, as amended by the No
   Child Left Behind Act of 2001 (P.L. 107-110): The Office of Elementary and Secondary
   Education is currently collecting information central to No Child Left Behind, including
   data on teacher qualification, state assessments, and schools in need of improvement.
•	 Program performance measurement: PBDMI is designed to support the management and
   analysis of education data, and thus is crucial to any departmental strategy for program
   performance measurement.
•	 Information collection: The PMT is working with States and program offices to establish
   common data elements and streamline information collections.
•	 Enterprise architecture: PBDMI represents an agency-wide Information Technology
   solution for managing program data and analys is.

Also, in the Department’s FY 2005-2008 Information Resource Management Strategic Plan,
the Department identified seven line-of-business areas. One of these areas, “Evaluation” will
use common high quality data and analytics to reduce customer burden and improve evaluation
quality. PBDMI is also tied to the Department goals of creating a culture of achievement and
establishing management excellence.

The PBDMI project is a multi- year system development and implementation effort intended to
consolidate the information collected from States, districts and schools in a way that improves
data quality and reduces the paperwork burden for national education partners.

Specifically, PBDMI should streamline and centralize the data collection process thereby
replacing multiple data collections.3 Other PBDMI goals include:


3
  This includes state formula , as well as the grant programs administered by Office of Special Education
Programs, the Office of English Language Acquisition, Office of Elementary and Secondary Education, and
Office of Vocational and Adult Education; Common Core of Data collections administered by the National Center
for Education Statistics; and the Elementary and Secondary Survey administered by the Office for Civil Rights.

Audit of the Department’s Performance                  7                             Final Audit Report
Based Data Management Initiative
ACN: ED -OIG/A11-E0003
•	 Improving internal efficiency and effectiveness by centralizing K-12 data collections and
   enabling this information to be shared more quickly and conveniently between Federal,
   State and local governments;
•	 Providing higher quality and more precise data to better align program funding to program
   results for increased accountability; and
•	 Improving the efficiency of the data collection and management process by leveraging
   Internet-based technologies.4

To streamline existing data collection efforts and information management processes, the
Department is using PBDMI to develop and implement EDEN; a database repository
system that will allow users to obtain
organized and formatted information
about the status and progress of
education programs in the States,
districts and schools. EDEN is being
developed through three integrated
system development efforts: EDEN
Submission System collects education
data submitted by SEAs; EDEN Survey
Tool collects Civil Rights Data from
school districts; and EDEN Decision
and Analysis Reporting System will
allow Program Managers and States to
query and analyze data submitted to
EDEN. EDEN will ultimately allow
SEAs and Federal agencies to transfer
and analyze information about
education programs.

PBDMI is intended to consolidate the collection of education information from States, districts
and schools in a way that improves data quality and reduces the paperwork burden for national
education partners. Through its data collections, PBDMI will collect and manage the
following types of educational data: a) Achievement and performance statistics; b) School
characteristics; c) Demographics; d) Program financial data; and e) Geospatial data (school
longitude/latitude). To assist in the accurate collection of education data, the Department will
rely on heavy participation from States and local education communities to obtain, use, share,
and report educational information about program successes and education outcomes. To
assist States with the additional reporting requirements associated with PBDMI, the
Department provided each SEA $50,000 in fiscal years 2003 and 2004 via task order contract,
under the conditions that education data be provided to the Department; an Annual Work Plan
be provided with detailed actions and schedules for providing educational data; and a statement
of assurance signed by the Chief State School Officer that the State will provide as many
education data elements as possible.

4
  For example, rather than requesting information (“collections”) from each State for each program, under PBDMI
the Department would negotiate protocols with each State to establish routine data transfers from its information
systems to a centralized data repository.

Audit of the Department’s Performance                    8                              Final Audit Report
Based Data Management Initiative
ACN: ED -OIG/A11-E0003
To meet project timelines, the PMT established aggressive project milestones dates and
adopted a rapid application development approach in designing, developing, and implementing
an integrated and centralized system (EDEN) that will collect and display education data.
According to the PBDMI PMT staff, the EDEN Submission System was developed to mitigate
risks under a three-step approach. Version 1.0 was a usable system for the submission of files
from the SEAs. Version 1.1 addressed any issues that developed in the initial use by real users
in the SEAs as opposed to test situations. Version 1.3 brought in the data framework
functionality for use by the PMT. Consistent with this approach, the PMT conducted a
PBDMI DSS Pilot, from November 2003 – March 2004, to determine the feasibility of
electronically collecting large amounts of K-12 data from SEAs and to solicit feedback from
the Program Offices about the use of the data collected.

The DSS Pilot was comprised of a voluntary collection of limited sets of data elements from
50 SEAs. Because States experienced different levels of success while submitting education
data to the Department, the scope of the Pilot was expanded to include additional data
submissions and a State-by-State analysis to identify common problems and develop corrective
actions. The Department contracted CTG, Inc. to evaluate the DSS Pilot and identify positive
repeatable results and corrective actions for problems before implementing the EDEN
Submission System. The resulting report, “PBDMI Decision Support System Pilot Evaluation
Report,” (July 2004) identified findings and provided recommendations to improve the success
of education data submissions from the States; the PMT’s customer support to the States; and
the PMT’s system development methodologies.

System Implementation Methodologies

Several methodologies exist to assist organizations in developing and implementing
information systems. All have benefits and risks that need to be considered when deciding
which system development methodology to follow. The PBDMI PMT adopted a quick
implementation methodology, known as the rapid application development (RAD)
methodology for prototypes. Alternatively, traditional structured system development
methodologies, such as the waterfall methodology, breaks projects into finite pieces and relies
on thorough upfront planning to ensure success of the project.

When using a traditional system implementation methodology, the implementation team
benefits from having a thorough planning and robust documentation process. According to the
Carnegie Mellon Software Engineering Institute, “Capability Maturity Model Integration
(CMMI) best practices associated with structured system development methodologies enable
organizations to:

•	 more explicitly link management and engineering activities to business objectives;
•	 expand the scope of engineering activities to ensure that the resulting products or services
   meet customer expectations;
•	 incorporate lessons learned from additional areas of best practice (e.g., measurement, risk
   management, and supplier management);
•	 implement more robust high- maturity practices; and
•	 address additional organizational functions critical to its products and services.”


Audit of the Department’s Performance           9                          Final Audit Report
Based Data Management Initiative
ACN: ED -OIG/A11-E0003
However, more traditional means of system implementation often requires more time, in
comparison to less traditional methodologies, such as RAD. Characteristics of the RAD
methodology include deployment of an early prototype which results in frequent end user
feedback; frequent delivery of products within a short time period; and informal system
development processes governing requirements management, configuration management,
quality assurance, and project oversight key process controls. When correctly applied, RAD
will result in active end user involvement, greater end user ownership of the system, software
that is likely to meet end users’ functionality requirements, and implementing production
systems in a relatively short time period. If not effectively managed, this approach can result
in developing multiple versions of systems that do not meet users’ functionality and
performance requirements, and ultimately missing critical project milestones dates. According
to the PBDMI PMT, these risks were understood, anticipated, and planned throughout the
development efforts.




Audit of the Department’s Performance          10                         Final Audit Report
Based Data Management Initiative
ACN: ED -OIG/A11-E0003
         Finding No. 1: PBDMI System Implementation Controls for the 

             Development of the EDEN System Need Improvement


We reviewed the PBDMI PMT system development implementation methodology to
determine whether key processes were implemented to ensure that system functionality
requirements have been adequately defined and production systems are meeting end user
performance requirements. We determined that system implementation controls need
improvement to ensure PBDMI will meet intended project goals. Specifically, we identified
that significant project goals are at risk because:

•	 sufficient data input controls have not been implemented to ensure that data is complete,
   accurate, and valid;
•	 versions of the EDEN Submission System were implemented that did not meet user system
   functionality and performance requirements;
•	 application level change control processes have not been implemented to govern changes to
   common data elements, the Meta Data dictionary, and supporting business processes;
•	 standard data elements have not been fully defined to ensure that PBDMI data collection
   can replace the Department’s numerous data collections efforts; and
•	 inadequate user training and insufficient documentation have resulted in ongoing data
   submission errors between the Department and SEAs.

Data Input Quality Controls Need Improvement

Based on our procedures performed, we determined that improved data input controls are
needed to ensure the education data provided by the SEAs is complete, accurate, and valid.
For example, in July 2004, the PMT contracted with C4DQ to perform a data quality analysis
of education information submitted in connection with the DSS Pilot.5 C4DQ evaluated data
tables residing in the EDEN Submission System and assessed six key data quality checks:

•	 Content Analysis – performed a frequency distribution analysis of data fields to identify
   whether duplicate attributes were present;
•	 Data Population – evaluated each column and row within the database to identify blank
   data fields;
•	 Primary Key Uniqueness – determined whether a unique primary key has been defined for
   each row within the EDEN database;
•	 Row Uniqueness – determined whether duplicate rows were present within the EDEN
   database;
•	 Referential Integrity – performed to identify “orphan” referential keys that do not link to
   other data tables with the database; and
•	 Data Dependency (business rules) – evaluated data fields for evidence that system edit
   checks are not functioning as intended.


5
 The DSS Pilot was implemented to determine the feasibility of electronic collection of large amounts of K-12
data from SEAs and to solicit feedback from the Program Offices about the use of the data collected.



Audit of the Department’s Performance                  11                              Final Audit Report
Based Data Management Initiative
ACN: ED -OIG/A11-E0003
C4DQ analysis identified numerous data quality issues with data collected during the DSS
Pilot, including duplicate data elements; empty data columns and data tables; duplicate data
row attributes; errors within data tables; and invalid data values that indicate business rule
violations.

C4DQ is evaluating the quality of the November 2004 PBDMI data submissions and will
report to the PMT in Spring/Summer 2005. Based on our discussions with C4DQ, the PMT,
and selected SEAs, the quality of the November 2004 data submission is not expected to
significantly improve because the business processes for collecting education information have
not changed much due to competing project priorities. Accordingly, we reviewed 69 help desk
logs from the Partner Support Center from November 2004 to January 2005 and identified
several calls from SEAs regarding the quality of education data submitted to the Department.
Specific technical assistance issues included:

•	 Inaccurate data calculations – a SEA noticed an inaccurate data calculation of 465,000
   when the accurate total sho uld have been around 1,700. Other SEAs identified instances
   where grand totals of data submitted were incorrectly smaller than the sum of the
   respective sub-totals;
•	 Incomplete data – help desk calls from SEAs indicated that data submissions included
   missing data (i.e. phone numbers) for several school districts; and
•	 Inaccurate data values – callers identified incorrect ethnic counts such as reporting 661,178
   American Indian or Alaskan Native students instead of 1,178.

The OIG has performed a number of audits of Title I Part A and Perkins III education
programs and concluded that management controls must be strengthened to ensure that data
submitted to the Department is complete, accurate, valid, and reliable. The reports state that
State and local education agencies have provided inaccurate and incomplete education
performance data to the Department.

Several Program Managers and SEAs raised a number of concerns about the data quality to us
and whether the Department is committed to improving data quality. For example:

•	 Program Managers question the capacity of EDEN and whether some States can collect
   and provide quality data to the Department;
•	 Program Managers felt there was insufficient time to collect and provide all of the
   education data;
•	 Some SEAs felt the quality of the data provided to the Department is questionable because
   they had not received any formal data quality training; and
•	 SEAs stated that the Department has not provided sufficient funding for States to develop
   their own databases with the proper internal controls to promote quality data and
   information.

During our data quality discussions, the PMT stated they planned to meet with SEAs during
summer 2005 to discuss methods for improving the quality of data submitted to the
Department. The PMT also plans to develop data quality performance measures so the
Department can track its progress towards meeting its investment goals. Our review indicates
that competing project priorities have precluded the PMT from devoting adequate resources to
address data quality concerns at the State and local levels.
Audit of the Department’s Performance            12                      Final Audit Report
Based Data Management Initiative
ACN: ED -OIG/A11-E0003
For PBDMI to achieve its investment goal of improving data quality, sufficient data input
controls (i.e. edit checks, business processes) must be implemented to verify the quality of data
from the source and to ensure data quality is maintained when consolidating data at the State
and local education level. Additionally, comprehensive training programs are needed for the
SEAs and LEAs so they can verify the completeness, accuracy, and validity of education data
before it is transmitted to the Department.

At the same time, the PMT will need to develop internal testing procedures and performance
measures to verify the completeness, accuracy and reliability of education data before the
EDEN Decision Analysis and Reporting System (DARS) becomes operational.6 By improving
data quality, the PMT will be able to demonstrate that PBDMI data collections can provide
quality education data that will meet the business requirements of the Program Offices.
Alternatively, Program Managers will continue to collect education data from multiple sources
until the quality of data within EDEN is proven sufficient to replace the current data collection
efforts.

Requirements and Configuration Management Processes Need Improvement
Based on our procedures performed, we determined that current requirements management and
configuration management processes need improvement in order for the PMT to develop future
versions of EDEN that will meet system functionality and performance requirements and will
ensure that PBDMI will meet its original investment goals. The PMT’s RAD approach has
been instrumental in developing and implementing the EDEN Submission System within
project milestone dates. However, the PMT’s informal requirements management and
configuration management processes have resulted in versions of EDEN that when
implemented do not meet system functionality and performance requirements.

For instance, the PMT initially implemented the EDEN Submission System (Version 1.0)
in November 2004 and subsequently the Department began receiving education data
submissions from SEAs. However, because of shortcomings associated with EDEN
Version 1.0, the PMT made significant software coding and system configuration
modifications to correct system defects and address end user complaints
to ensure that the EDEN Submission                         PBDMI System Modifications
System meets functionality and
performance requirements.7                      Changes to
                                               accommodate                  Other
Accordingly, the PMT released EDEN                  user                  Changes
                                                 requests.                    4%
Submission System Version 1.1 in                    18%
December 2004 (about 40 days later)                                                   Changes to
                                                                                        correct
and Version 1.2 in January 2005                                                         system
                                                                                       functionality
(about 70 days from the initial               Changes to
                                               correct
                                                                                      shortcomings
                                                                                           42%
Version 1.0 release). The                      system
                                               defects.
accompanying chart illustrates the              36%

necessary system modifications since
the EDEN Submission System was
implemented.

6
  EDEN DARS will be available by Summer 2005 and will permit Program Managers to query data collected
from the Fall 2004 education data submission.
7
  Significant system defects were classified from “High” (no work-around) to “Very High” (critical for proper
system functionality) and were corrected in subsequent version releases of the EDEN Submission System.
Audit of the Department’s Performance                    13                            Final Audit Report
Based Data Management Initiative
ACN: ED -OIG/A11-E0003
Our review of EDEN Submission System Release documents (1.1 and 1.2) revealed numerous
system functionality defects in earlier versions of EDEN and indicates that current requirement
management and configuration management controls need improvement. Some significant
system functionality defects that we noted include:

•	 System is rejecting a “Assessment for Promotion/Graduation Status” field when populated
   with a valid data value;
•	 System is rejecting a “Integrated Technology Status” field when populated with a valid
   data value;
•	 Duplicate school period records are present causing duplicate schools to appear within the
   ED database;
•	 LEA current status XML file incorrectly gives a format error response;8
•	 XML “School Status” (Current Year) schema needs to be changed to allow values less than
   15 characters;

According to the PMT, frequent changing system functionality requirements, staffing
constraints, and aggressive project milestones have contributed to delays in developing and
implementing certain functional components of EDEN on schedule. Consistent with our
findings, the DSS Pilot evaluation report identified that system functionality requirements for
EDEN were not fully defined and the PMT needs to implement quality control reviews and
configuration management techniques to reduce system defects and system down time

While the PMT’s RAD system development approach informally addresses key process
controls (e.g., requirements management and configuration management), the PMT has not
developed formal policies and procedures to ensure that positive system development efforts
will be repeated and common system development problems will be identified and corrected.
The Department has not designated an official system development methodology although a
draft methodology policy document currently exists. Additionally, the draft policy does not
preclude the use of RAD system development methodology nor address the system
development risks associated with the RAD.

The PMT could benefit by adopting a formal structured configuration and requirements
management processes such as the Carnegie Mellon’s CMMI. A structured development
process would document policies and procedures that support key process controls and provide
guidance to system developers to proactively address issues and correct potential problems
before they occur. If the configuration and requirements management problems that we
identified are not addressed, the PMT will not be successful in proactively developing
integrated systems that meet the system functionality and performance requirements.
Additionally, the PMT has not devoted sufficient resources to effectively mitigate the risks
introduced under a RAD system implementation; thereby, putting the PBDMI project at risk of
not meeting its project goals.




8
 XML (Extensible Markup Language) is a widely used system for defining data formats.
Audit of the Department’s Performance               14                              Final Audit Report
Based Data Management Initiative
ACN: ED -OIG/A11-E0003
Insufficient Application Level Change Controls

Based on our procedures performed, we determined that the PMT needs to develop formal
change control processes to govern changes to PBDMI application level common data
elements and values, the related Meta Data dictionary, and supporting business processes
necessary to ensure user requirements will be met.9 During the development of the EDEN
Submission System, the PMT followed the Department’s change control and configuration
management processes, while implementing hardware and software supporting EDEN.
However, the PMT has not developed and documented change control processes to govern
changes to application level data elements, the Meta Data dictionary, and supporting business
processes. According to the PMT, one person is currently assigned to manage changes to
EDEN common data elements and the Meta Data dictionary, because of staffing constraints.
Consequently, changes to common data elements and Meta Data dictionary are not formally
reviewed and approved by a change control review board.

A formal change control process is important because it instills a review process for all
application level system changes and will help ensure that changes to the system and business
processes do not adversely impact system performance and the Department’s business
operations. Informal change control processes will not ensure that a history of system changes
are maintained to assist in identifying system performance issues. The PMT has indicated that
they plan to develop a formal change control process that governs changes to the EDEN Meta
data dictionary and the PBDMI data collection and reporting process, although these plans
have not been formalized and prioritized.

Standard Data Elements for Centralized Data Collections are not Fully Defined

We determined that the PMT has not fully defined all standard data elements in order for the
Department to ultimately discontinue its multiple data collection efforts. While the PMT has
defined over 140 standard data elements to be collected in connection with PBDMI,10 many
additional standard data elements must be identified to ensure that the centralized data
collection process will meet the needs of all Program Managers. The PMT has used a
reiterative approach to identify and define standard data elements and definitions to be used by
the Program Managers and SEAs for collecting and reporting education information.
However, this approach will ultimately prevent the PMT from proactively collecting a
complete listing of common data elements because the process relies solely on the input from
Program Managers and SEAs. The PMT needs to gain a thorough understanding of the
Department’s programs and reporting requirements in order to fully develop a complete listing
of standard data elements. One Program Manager described the data element development
process as providing many iterations of common data elements that the States can provide, but
not identifying the data element collection requirements of the Program Office itself.

A representative from the PMT has stated that aggressive project milestones and staffing
constraints have prevented the team from taking a more proactive approach in defining

9
  Meta data dictionary identifies the data elements, definitions, collection activities, and reporting processes; it
also helps ensure that future system changes are controlled and does not inhibit users from gathering their
respective educational data.
10
  According to the PMT, this represents approximately 80% of the total data elements to be defined.
Audit of the Department’s Performance                  15                              Final Audit Report
Based Data Management Initiative
ACN: ED -OIG/A11-E0003
common data elements and definitions. During Spring/Summer 2005, the PMT plans to
conduct Strategic Information Reviews and to meet with the Program Managers and
SEAs/LEAs to define additional common data elements that must be collected to meet the
Program Offices’ reporting requirements. The Department will not be able to meet its project
goals of improving efficiency and effectiveness if it does not proactively identify and
standardize all education common data elements in support of PBDMI.

Inadequate User Training and Insufficient System Documentation

We determined that inadequate user training and insufficient system documentation have
contributed to on- going data submission problems between SEAs and the Department; a
shortcoming that could undermine the project goal of achieving efficiency and effectiveness.
For example, our review of a sample of PBDMI help desk logs revealed that numerous data
transmission errors occurred during the period November 2004 to January 2005. Based on
discussions with SEAs, many of the data transmission errors could have been avoided if the
SEAs had received sufficient technical training about the data submission process. Specific
data transmission problems brought to our attention include:

•	 SEAs were unable to determine why the system is rejecting XML data when certain data
   fields are populated with a valid “NOTCOLLECT” data value;
•	 SEAs received invalid format error messages when submitting files that contained no
   errors;
•	 SEAs received invalid error messages stating that a table type abbreviation does not exist
   after the data value had passed edit checks; and
•	 SEAs received invalid transmission errors while attempting to upload school immigrant
   files to the Department (i.e. 0010/2460 – National Center for Education Statistics Country
   Codes).

Most SEAs expressed frustration as data transmission problems caused untimely delays in
submitting educational data to the Department. Additionally, SEAs indicated that they devoted
inordinate amounts of time to understand error messages and then needed to dedicate
additional staff to correct the error messages. Furthermore, SEAs stated that additional
technical training is necessary so that the States can efficiently respond to error messages and
verify XML data against XML schema and file specifications. Although no formal training
program has been developed, the PMT has indicated that better technical training is needed for
SEAs to reduce the number of data submission errors.

Our review of the help desk logs revealed many help desk issues concerning data submission
policy questions. Ideally, policy questions should be clarified through comprehensive system
documentation including complete user requirements and standard data elements and
definitions. Specific help desk questions pertaining to unclear data submission policies
include:

•	 Many inquiries about whether to report students by age or by age group as specifications
   imply that the Department needs both;
•	 Many SEAs inquired whether they should report student achievement results for the 2003­
   2004 school year or report the results when making the Adequate Yearly Progress
   determinations ; and
Audit of the Department’s Performance          16                         Final Audit Report
Based Data Management Initiative
ACN: ED -OIG/A11-E0003
•	 Many inquiries related to how to handle the file specifications for students with disabilities
   beyond age 21 “Children with Disabilities-Individuals with Disabilities Education Act of
   2004.”

Several SEAs representatives have complained that there is no clear guidance or fully
developed data element specifications to assist the SEAs in the data submission process.
Additionally, certain SEAs stated that the lack of clear guidance from the Department has
increased the reporting burden for States and has resulted in duplication of work for those
submitting education data to the Department. Due to competing project priorities, the PMT has
not devoted the resources to develop clear guidance that will assist SEAs during the data
submission process. The PMT needs to develop comprehensive data submission policies and
guidance so that the SEAs know exactly the type of data to submit, submission procedures, and
submission timelines.

Recommendations

1.1	    We recommend the Deputy Secretary devote additional resources to ensure the PBDMI
        PMT can adequately address the system development risks identified in this report.

Department Response
The corrective action for this recommendation is ongoing. The PBDMI PMT was augmented
by two new FTEs in the last quarter of 2004. In addition, primary responsibility for program
staff liaison on PBDMI data elements was assigned to three elementary and secondary
education performance analysts in the Strategic Accountability Service, who assist the program
offices in maintaining their program performance data. A new data acquisition team leader
position has been competed and is pending a response to the offer. As part of the Secretary’s
reorganization that created the Office of Planning, Evaluation and Policy Development, two
new deputy assistant secretary positions have been created with primary responsibility for
PBDMI policy coordination. Also, a senior OMB examiner is assigned to work part-time with
the PBDMI policy team. The PMT is currently reviewing overall PBDMI staffing levels to
ensure that the PMT can adequately address the system development risks identified in this
report and other project requirements.

OIG Response
We reviewed the Department’s response and action taken. Based on the Department’s planned
action, an acquisition team leader position is expected to be filled and two new deputy assistant
secretary positions have been created with primary responsibility for PBDMI policy and
coordination. A corrective action plan is required to ensure that audit follow- up processes
have corrected the issues identified.

Additionally, we recommend that the Director, Strategy Accountability Service and the PMT:

1.2	    Provide guidance to SEA/LEAs in developing effective data input controls (i.e. edit
        checks, business processes) to verify the quality of education data and to ensure that
        data quality is maintained during the collection and data transmission process.




Audit of the Department’s Performance           17                          Final Audit Report
Based Data Management Initiative
ACN: ED -OIG/A11-E0003
Department Response
The corrective action for this recommendation is ongoing. Data quality is a continuous
improvement process inherent in the Education Data Exchange Network. The current quality
of state education data provides much room for improvement. PBDMI has multiple strategies
underway to encourage and assist states to improve their data systems and the quality of their
data.

PBDMI is a systematic attempt to address those problems that exist in current State data
reporting systems:
           1.	 State data are transferred directly from State administrative record systems into
               EDEN rather than being transcribed by State program officials onto a Federal
               reporting form. This eliminates one potential source of error.
           2.	 EDEN’s first step in receiving State data is automated error checks. These are
               designed to assure appropriate formatting, consistency with previous
               submissions, and conformance with pre-defined parameters.
           3.	 EDEN’s Partner Support Center reviews issues such as those identified in 2,
               above, with States and to request resubmissions of accurate data, as documented
               by GAO.
           4.	 PBDMI has contracted with the Center for Data Quality (C4DQ) to review the
               accuracy of State data submissions and to provide recommendations for
               improvement.
           5.	 PBDMI also contracted with C4DQ to provide data quality reviews for
               individual States and to provide them recommendations and technical
               assistance. C4DQ has conducted reviews in 10 States to date.
           6.	 PBDMI has provided and will continue to provide data quality training and
               technical assistance to State PBDMI Coordinators in collaboration with NCES
               and CCSSO.

As data are reviewed for quality, additional error checks will be identified. As errors are
corrected in an individual State, the potential for similar errors will be identified to other States
and added to the review list for the Partner Support Center.

OIG Response
We reviewed the Department’s response and action taken. The Department has recognized
that the current quality of state education data provides much room for improvement.
Although the response indicates that corrective action has been taken, a corrective action plan
is required to ensure that audit follow-up processes have corrected the issues identified.

1.3	    Develop internal testing procedures and performance measures to ascertain the
        completeness, accuracy and reliability of education data before the EDEN Decision
        Analysis and Reporting System becomes operational.

Department Response
The corrective action for this recommendation is ongoing. We agree with this
recommendation and are aware of the need to ensure that high quality data is used for policy
decisions and program management. Many actions have been taken or are planned to ensure
EDEN uses the best available data (see 1.2 for a summary). EDEN data edits are based on
permitted values and prior years’ CCD submissions. As EDEN accumulates multiple years of
Audit of the Department’s Performance             18                           Final Audit Report
Based Data Management Initiative
ACN: ED -OIG/A11-E0003
data, it will be able to expand the database foundation for automated edits. In addition,
program office staff and performance analysts are reviewing state data submissions against
previous program office submissions as a basis for follow-up inquiries with States. These
inquiries will help States identify specific areas on which to focus their data quality efforts.
Importantly, the reporting system itself will enable additional analyses and reviews that will
further support our data quality efforts. Because the reporting system will be operational
before all data quality issues are finally resolved, users will be trained on the appropriate uses
of the data.

As mentioned earlier, data quality is a continuous improvement process. In the interim, the
data available for EDEN far exceeds the quality of what has been previously available to
policy makers and program managers.

OIG Response
We reviewed the Department’s response and action taken. A corrective action plan is required
to ensure that audit follow-up processes have corrected the issues identified.

1.4	    Implement comprehensive requirements and configuration management controls to
        develop EDEN integrated systems that will meet system functionality and performance
        requirement, as well as encourage a proactive system development approach.

Department Response
The corrective action was implemented in July 2005. ED selected Perot Systems, Inc. as
contractor for PBDMI Operations, Maintenance and Enhancement. That contract includes
enhanced configuration management controls negotiated with OCIO’s new contractor, CSC.
Comprehensive requirements were laid down in the contract and are being further specified in
a series of action plans under development by Perot and review by ED.

OIG Response
We reviewed the Department’s response and action taken. The selection of a single contractor
to assist in configuration management should assist in strengthening the management of these
controls. A corrective action plan is required to ensure that audit follow-up processes have
corrected the issues identified.

1.5	    Develop comprehensive procedures so the PMT can proactively define, collect, and
        standardize a complete listing of standard data elements and definitions in connection
        with PBDMI.

Department Response
Implemented September 2004 and under annual review. The PMT has completed in
collaboration with OCIO/RIMS the second PBDMI collection submission to OMB. The
proposed 2006-2008 collection builds on the 2004-2005 data elements and definitions with
minor changes necessitated by the new IDEA reauthorization and similar programmatic
changes. The identification of data elements is an iterative process. PBDMI began by asking
program offices to define their information requirements (legislation, regulation, reporting,
monitoring, program management, and risk avoidance.) It then reviewed these requirements
against data currently gathered by ED programs and offices, identifying gaps, redundancies
and inconsistent definitions. The resulting data set was reviewed with the states for
Audit of the Department’s Performance            19                           Final Audit Report
Based Data Management Initiative
ACN: ED -OIG/A11-E0003
availability. Concurrently, a series of policy questions were developed with senior officers and
data elements reviewed and modified to assure that those questions could be answered. We
expect user requirements to evolve substantially as program officials and senior officers are
able to access these data and further clarify their requirements.

The OPEPD planning team and new Executive level PBDMI Steering Committee will be
undertaking a review of all elementary and secondary data elements to assure that programs
rely on standard data elements and definitions that are aligned with PBDMI.

OIG Response
While the Department’s response states that this recommendation was addressed in September
2004, it also states that the OPEPD planning team has not fully reviewed elementary and
secondary data elements and identification of data elements is an iterative process.
Consequently, the PMT has not fully developed comprehensive procedures for proactively
defining and collecting a complete listing of standard data elements. Although the Department
states that these actions were in place, we did not observe that these controls were effectively
implemented during our review. Accordingly, we recommend that the PMT develop a
corrective action plan to address the need to fully develop comprehensive procedures to
proactively define, collect and standardize a complete listing of standard data elements and
definitions in connection with PBDMI.

1.6	    Develop a formal change control process that will govern changes to the EDEN Meta
        data dictionary and business processes associated with PBDMI data collection and
        reporting activities.

Department Response
The corrective action was implemented in July 2005. Perot Systems has proposed and has
implemented a formal change control process under the oversight of the PMT and Steering
Committee.

OIG Response
We reviewed the Department’s response and action taken. Contractor assistance should assist
the PMT in strengthening the change control process. A corrective action plan is required to
ensure that audit follow-up processes have corrected the issues identified.

1.7	    Develop comprehensive training programs and guidance to assist SEAs in reducing on­
        going data submission errors with the EDEN Submission System.

Department Response
The corrective action for this recommendation is ongoing. The PMT and Perot Systems are
updating the EDEN guidance documents and training based on the proposed 2006-2008 data
elements. Two and a half day training sessions have again been scheduled for State staff. Two
regional trainings will be offered, one in Denver, CO, October 24-26, 2005, and one in
Washington, DC, November 2-4. Both sessions will include introductory orientation for state
Staff new to the EDEN system. We expect near 100% participation again this year.




Audit of the Department’s Performance           20                         Final Audit Report
Based Data Management Initiative
ACN: ED -OIG/A11-E0003
OIG Response
We reviewed the Department’s response and action taken. The Department’s plans to update
training and documentation should strengthen the program. A corrective action plan is
required to ensure that audit follow- up processes have corrected the issues identified.

Additionally, we recommend that the CIO:

1.8	    Provide system development guidance that would address the risks associated with
        using the RAD methodology.

Department Response
The OCIO did not provide a response to this recommendation.

OIG Response
Although no response was provided, the OCIO is required to develop a corrective action plan
to address this outstanding recommendation.




Audit of the Department’s Performance         21                        Final Audit Report
Based Data Management Initiative
ACN: ED -OIG/A11-E0003
  Finding No. 2: PBDMI Project Management Controls Need Improvement

We reviewed PBDMI project management controls to determine whether procedures have
been implemented to ensure that project milestones are met and contract deliverables are
performed. We have concluded that project management controls need to be strengthened.
Specifically, we identified significant project milestone slippages and the Department made
payments to certain States without evidence that contract deliverables were performed. This
condition exists because the PMT established aggressive project deadlines and had insufficient
staff to support the project. Unless these problems are addressed, project costs will increase
and EDEN will not meet user expectations and performance requirements in a timely manner.

Missed Project Milestones

Our review of the PBDMI Daily Status Reports from December 2004 to January 2005 has
revealed that the PMT missed significant project milestones. Specifically, we noted slippages
in the following milestones:
•	 Preparing and Delivering Guidance for SEAs slipped by approximately 150 days. This
    milestone is important for ensuring that SEAs have clear policy and guidance for collecting
    and submitting timely education data to EDEN. As discussed earlier in this report, some
    SEAs complained that they lacked clear guidance. This has resulted in duplication of work
    and reporting, and data transmission problems with the EDEN Submission System.

•	 Hardware and Software Requirements for DARS slipped by approximately 90 days. These
   milestones are important for ensuring that the PMT develops and implements versions of
   the DARS that will meet functionality and performance requirements of the Program
   Managers who will rely on those systems to gather and report education data. As stated
   earlier in this report, informal requirements and configuration management controls have
   contributed to the PMT implementing versions of the EDEN Submission System that was
   burdened with software coding defects and functionality deficiencies.

•	 Certification and Accreditation (C&A) of EDEN slipped by approximately 120 days. This
   milestone is important for ensuring that operational systems are adequately protected from
   internal and external security threats and vulnerabilities. Although the EDEN Submission
   System began receiving operational education data before becoming formally certified to
   process transactions, the PMT is working with the OCIO to comply with the Department’s
   C&A requirements. The EDEN Submission System was formally certified to process data
   in March 2005.

•	 Organizational Transformation slipped by approximately 150 days. This milestone is
   important for defining who will be the system owner and responsible for system
   modifications as the EDEN solution enters into the operations and maintenance phase. It is
   not clear who is the ultimate system owner for the EDEN Submission System, Survey
   Tool, and DARS. For organization transformation of EDEN to be successful, the PMT
   needs to clearly define: the group responsible for coordinating transformation activities
   from development to maintenance phase; the appropriate training for all personnel
   involved in the transformation activities; and a complete inventory of all hardware and
   software that will be transitioned to the new system owner.
Audit of the Department’s Performance           22                         Final Audit Report
Based Data Management Initiative
ACN: ED -OIG/A11-E0003
The PMT has stated that aggressive project deadlines and staffing constraints have
predominantly contributed to the team missing project implementation dates. Additionally,
frequent changes to system functionality requirements and inaccurate ve ndor completion dates
have contributed to delays in system development and implementation. The PMT’s informal
RAD system development methodology has also contributed to frequent changes in system
functionality requirements. The PMT also stated that the lack of qualified staff has adversely
impacted the team’s ability to manage multiple contractors in meeting project milestones and
integrating systems on diverse platforms. To improve project management controls, the PMT
plans to award a single contract that will be responsible for managing various sub-contractors
who will maintain the EDEN data collection and reporting system.

State Contract Support for PBDMI Project

To help reduce the reporting burden associated with providing data under PBDMI, OMB
appropriated to the Department $50,000 to give for each SEA. We noted that in some cases,
the Department disbursed funds to the States without adequate documentation that the States
fulfilled the terms of the contract. Specifically, we noted that certain States did not provide a
detailed Annual Work Plan in accordance with the contract but still received the $50,000 from
the Department. We also noted that two SEA’s received $50,000 under the task order contract
but did not submit any data in November 2004 as required. Without effective contract
monitoring, the PMT cannot ensure that SEAs are efficiently and effectively using task order
funding in support of PBDMI. A PMT representative has stated that a lack of adequate
staffing has caused the PMT to focus their limited resources on the monitoring system
development contractors as they constitute the most significant contracts supporting PBDMI.

Recommendations

2.1	    We recommend the Deputy Secretary devote additional resources to ensure the PBDMI
        Project Management Team can adequately address the project management risks
        identified in this report.

Department Response
The corrective action for this recommendation is ongoing. The Department referred to the
same response as in recommendation 1.1 in response to 2.1.

Additionally, we recommend that the Director, Strategy Accountability Service and the PMT:

2.2	    Develop comprehensive project management controls to effectively manage the timely
        completion of project milestones and oversee multiple contractors responsible for
        developing the EDEN integrated systems.

Department Response
A corrective action for this recommendation was implemented November 2004. PBDMI has a
master project schedule, which the PMT has reviewed weekly with active contractors and
OCIO for the past year and a half. The Operations, Maintenance and Enhancement contract
awarded to Perot Systems, Inc., was designed to transition from the multiple contractors


Audit of the Department’s Performance           23                          Final Audit Report
Based Data Management Initiative
ACN: ED -OIG/A11-E0003
required for development to a single contract to manage EDEN with the PMT. This transition
will be complete at the end of this fiscal year.

OIG Response
The Department’s response states that a corrective action to this recommendation was
implemented as of November 2004. The weaknesses identified in this report were noted after
November 2004. While we recognize that the PMT has taken steps to improve project
management controls by recently awarding a single contract to manage multiple contractors
supporting PBDMI, the Department has stated that this transition process will not be fully
implemented until the end of the fiscal year. Until this trasition process is completed, we
recommend that the PMT continue to evaluate their project management controls to ensure that
they adequately support the timely completion of project milestones and effectively oversee
multiple contractors responsible for developing the EDEN integrated systems.

2.3 	   Develop comprehensive contract management controls to ensure that contract
        deliverables are performed and funds are disbursed in accordance with contract
        requirements. Furthermore, the PMT needs to develop procedures to ensure that all
        States provide education data in support of PBDMI.

Department Response
The corrective action for Part 1 was implemented in July 2005; Part 2 corrective action plan is
ongoing. Comprehensive contract management controls have been put in place for the
comprehensive Operations, Maintenance and Enhancement contract with Perot Systems, Inc.,
begun in July 2005. The PMT has developed a project management plan with Perot that is
reviewed weekly. The COR has been designated the coordinator of the project management
plan. A transition plan is also in place with Perot and the legacy contractors which is
coordinated by the alternate COR for the Perot contract.

The new Office of Planning, Evaluation and Policy Development has developed a strategy to
provide the incentives to states to ensure that all States provide education data in support of
PBDMI. This strategy includes communication by senior officers, waiver of reporting
requirements to those States that have transmitted the required data, and draft regulations
requiring State submissions through EDEN.

OIG Response
We reviewed the Department’s response and action taken. The Department’s awarding of the
Perot contract should assist in strengthening contract management controls as well as Perot’s
assistance in project management plan reviews. A corrective action pla n is required to ensure
that audit follow-up processes have corrected the issues identified.




Audit of the Department’s Performance           24                          Final Audit Report
Based Data Management Initiative
ACN: ED -OIG/A11-E0003
 Finding No. 3: Enterprise Architecture Shortfalls Need to be Addressed to 

                          Ensure PBDMI Success


We reviewed the Department’s EA to determine whether the EA has been consistently updated
to support the technical requirements and business processes of PBDMI. We have concluded
that the Department’s EA is not fully developed to ensure PBDMI’s success. Specifically, the
PBDMI Meta Data dictionary has not been adopted into the Department’s EA to further define
standard data elements, data definitions and the business rules associated with collecting those
data elements. Additionally, we noted that the EA Technical Reference Model has not been
fully developed to establish technical standards, products, and services that are required to
build systems in support of the Department’s required business functions and information
security requirements.

The Clinger-Cohen Act of 1996 (40 U.S.C., Section 1401 et. seq.) recognizes the importance
of architectures by requiring agency Chief Information Officers to develop, maintain, and
facilitate integrated systems architecture. A fully developed EA should fully describe business
functions of the agency; define the information needs and flows of information among the
business functions; and establish technical standards, products, and services that will be used to
build systems that support the defined business functions.

Enterprise Architecture Lacks Meta Data Dictionary

We noted that the Department’s EA does not include a Meta Data Dictionary supporting
PBDMI data collections. The Meta Data Dictionary defines: the data elements and definitions
collected in connection with PBDMI; acceptable data submission formats; permitted data
values; and applicable business rules and warnings associated with the data elements. A PMT
representative stated that adopting an Enterprise Meta Data Dictionary is important to
PBDMI’s success because it will clearly define standard data elements, data definitions, and
the business processes for which the data will be collected and reported across the Department.
Disagreements exist between the Program Managers and the PMT because standard data
formats, permitted values, and business processes have not been defined. Unless these
disagreements are resolved in a timely manner, the PMT faces the risk that Program Managers
will not fully support the goals of PBDMI, which could jeopardize the success of the project.
The Department’s ECA has indicated that the EA will be updated to incorporate a standard
Meta Data Dictionary in connection with the project.

EA Technical Reference Model Needs Development

We found that the EA Technical Reference Model has not been fully developed to establish
technical standards, products, and services to build systems in support of the Department’s
required business functions and information security requirements. According to the
Department’s ECA, the current Technical Reference Model is a broad document that
references to the Federal Enterprise Architecture but does not provide specific technical
standards for implementing information systems. A PMT representative stated that the lack of
a fully developed Technical Reference Model caused delays in implementing the EDEN


Audit of the Department’s Performance           25                          Final Audit Report
Based Data Management Initiative
ACN: ED -OIG/A11-E0003
Submission System. This is because the Department’s protocol standards, configuration,
firewall, and anti- virus requirements for implementing production systems were not defined
within the EA.

The ECA indicated that a Technology Roadmap, covering 16 technical subject areas, is being
developed that will assist contractors in developing systems that will comply with the
Department’s protocol, configuration, and information security requirements. The OCIO is
also performing a “gaps analysis”11 of the various Department functions to identify essential
business processes that need be identified in the Department’s Target EA. The Target EA will
ultimately contain descriptions of business processes, information flows, and a sequencing plan
for achieving a vision for the Department’s seven functional lines of business. The
Department has devoted considerable resources towards developing its EA in the last year.
Target EA activities should be completed during fiscal year 2006.

During fiscal years 2003 and 2004, OMB evaluated the Department’s EA and the resulting
scores (i.e., 2003 = 2, 2004 = 2.81 out of possible 5 points) indicate improvements are needed
in the clarity and vision of overall EA.12 OMB also noted a number of areas where the current
EA must be improved: target architecture is defined only as a series of visions with no
architectural modeling evident; transition plan is evident but tied only at a conceptual level to
the Target Architecture; performance measures are not integrated across all levels of the EA;
and the architecture does not provide an inventory of services with a specific view towards
identifying redundant service components.

Recommendations

3.1	    To help ensure the success of PBDMI, we recommend that the Department’s ECA
        coordinate with the PMT to adopt a PBDMI Meta Data dictionary into Department’s
        EA to further define standard data elements, data definitions and the business rules
        associated with collecting those data elements.

Department Response
Corrective action was implemented in March 2005. The ED EA team has developed and
implemented an ED EA Data Dictionary, which includes extensive metadata fields. It is
maintained in the ED EA Repository and is currently available to Department users via
ConnectED (http://connected1.ed.gov/po/ea/). The Department’s ED EA Data Team
established the Enterprise Data Dictionary August 2004. A corresponding Enterprise Data
Dictionary User’s Guide was developed at the same time. An overview of the Enterprise Data
Dictionary was completed December 1, 2004 and is posted on ConnectED
(http://connected1.ed.gov/po/ea/docs/ecdm-edd_overview.doc).




11
   A process described by the Department’s EA, in which the differences between believed and actual business
processes are identified.
12
   Scores ranging from 0-3 within the EA Assessment Framework resulted from evaluating the content of the
Department’s EA program and making recommendations for improvement. Scores ranging from 4-5 result from
evaluating the extent of EA programs and whether they have been successfully integrated into the Department’s
IT investment decision-making process.

Audit of the Department’s Performance                  26                             Final Audit Report
Based Data Management Initiative
ACN: ED -OIG/A11-E0003
Between August 2004 and March 2005, the Enterprise Data Dictionary added 1,494 data
elements including those developed by PBDMI (EDEN). Incorporation of additional data
elements and associated metadata has continued. As of July 2005 the ED EA Enterprise Data
Dictionary contained data from 14 data models totaling 2,072 data elements. Incorporation of
data elements into the ED EA Data Dictionary is continuing

The intent of establishing this Department-Wide ED EA Data Dictionary was to clearly define
standard data formats and permitted values across the Department. These standard formats and
values are directly related to the ED EA Enterprise Conceptual Data Model, and to Logical and
Physical data models developed within various programs such as PBDMI (EDEN).

Ensuring full support of PBDMI (EDEN) from the program offices takes two paths. The first is
the Information Collection approval process. Within the Department, that process includes a
review of proposed collections to establish how much of the collection can be obtained through
PBDMI (EDEN). Second, all new IT projects within the Department must conform to internal
Life Cycle Management review and approval “gates.” Included are gates for review of the
technical architecture (including data). At that time, the relationship between a proposed
collection and the existence of relevant data in extant collections is established. These
procedures had not governed program office systems or data collections prior to the launch of
PBDMI. However, these mechanisms are now in place (Information Collection approval) or
will be in place shortly (Life Cycle Management review), and will be used to curtail the use of
redundant information collection and data storage activities.

OIG Response
The Department’s response states that a corrective action to this recommendation was
implemented as of March 2005. While we recognize that the Department has taken actions in
response to our preliminary audit findings and Draft Audit Report, the Department states that
the incorporation of data elements into the EA Data Dictionary is a continuing process.
Furthermore, during our May 2005 exit conference, PMT members concurred with our finding
by stating that the Department lacked agency wide guidance for adopting Meta Data dictionary
elements into the EA. Accordingly, we recommend that the Department's ECA continue to
coordinate with the PMT to adopt all PBDMI Meta Data dictionary elements into Department's
EA to fully define standard data elements, data definitions and the business rules associated
with collecting those data elements.

Additionally, we recommend the Department’s ECA:

3.2	    Fully develop the EA Technical Reference Model to establish technical standards and
        information security requirements that are needed to build systems in support of the
        Department’s critical operations.

Department Response
ED’s EA documentation library FY2005-Q3 release includes ED’s Technical Reference Model
and Technology Standards Profile documents. These documents were developed together to
establish technical standards, products, and services to build systems in support of ED’s
business functions and information security requirements. These documents are also available
on ConnectED (http://connected1/test/po/ea/cdversion/2005-05-31_ED_EA_Start_Page.html).


Audit of the Department’s Performance          27                         Final Audit Report
Based Data Management Initiative
ACN: ED -OIG/A11-E0003
ED’s Technical reference Model (TRM) is consistent with oversight documents as follows:

    •	 In accordance with OMB’s EA Assessment Framework Version 1.5, ED’s TRM
       documents present the taxonomy used to categorize ED’s standard IT products and
       (non-proprietary) standards, and uses a custom TRM taxonomy that matches ED’s EA
       vision and mission needs and also aligns with the FEA TRM.
    •	 In accordance with OMB Circular A-130, ED’s TRM identifies and describes the
       information services used throughout the agency.
    •	 In accordance with the Practical Guide to Federal Enterprise Architecture, ED’s TRM
       is a taxonomy that provides:
    •	 a consistent set of service areas, interface categories, and relationships used to address
       interoperability and open-system issues
    •	 conceptual entities that establish a common vocabulary to better describe, compare, and
       contrast systems and components
    •	 a basis for the identification, comparison, and selection of existing and emerging
       standards and their relationships

ED’s Standards Profile is consistent with oversight documents as follows:

•	 In accordance with OMB’s EA Assessment Framework Version 1.5, ED’s Enterprise
   Standards Profile contains a list of non-proprietary interoperability standards currently used
   at ED, and maps these standards to BRM subfunctions, SRM service components, and
   TRM service categories.
•	 In accordance with OMB Circular A-130, ED’s Standards Profile defines the set of IT
   standards that support the services articulated in ED’s TRM, and includes a Security
   Standards Profile that is specific to the security services specified in the EA and, in
   conjunction with ED’s Baseline Security Requirements document, covers such services as
   identification, authentication, and non-repudiation; audit trail creation and analysis; access
   controls; cryptography management; virus protection; fraud prevention; detection and
   mitigation; and intrusion prevention and detection.
•	 In accordance with the Practical Guide to Federal Enterprise Architecture, ED’s Standards
   Profile is the set of rules that govern system implementation and operation, references the
   technical standards that apply to the architecture and how they need to be, or have been,
   implemented, and is time-phased to facilitate a structured, disciplined process of system
   development and evolution and to promote the consideration of emerging technologies and
   the likelihood of current technologies and standards becoming obsolete. ED’s TRM
   organizes ED’s Standards Profile. ED has presented the TRM and Standards Profile as
   separate documents instead of as a combined document. However, ED’s Standards Profile
   includes all of the information contained in ED’s TRM and is therefore a superset of ED’s
   TRM.

OIG stated that “A PMT representative stated that the lack of a fully developed Technical
Reference Model caused delays in implementing the EDEN Submission System. This is
because the Department’s protocol standards, configuration, firewall, and anti- virus
requirements for implementing production systems were not defined within the EA.”

ED’s TRM and Standards Profile are new documents that organize the standards and
guidelines, including protocol, configuration, firewall, and anti- virus standards, that are
Audit of the Department’s Performance            28                           Final Audit Report
Based Data Management Initiative
ACN: ED -OIG/A11-E0003
published in source documents that were used by the EDNet Technical Review Board process
at the time of PBDMI’s TRB reviews. These source documents include the EDNet Product
Support Plan, which is a list of products and product usages that are supported by EDNet
Operations, and the EDNet Technology Standards Guide Version 4, which is a more
comprehensive list of products and technical specifications supported on EDNet. EDNet’s
Technology Standards Guide Version 4 identifies EDNet’s protocol standards, configuration,
firewall, and anti- virus requirements.

OIG Response
We reviewed the Department’s response and action taken. A corrective action plan is required
to ensure that audit follow-up processes have corrected the issues identified.

3.3	    Improve the Department’s current EA and clearly define the EA Target Architecture
        and Transition Plan to ensure it’s aligned with the OMB Federal EA.

Department Response
ED’s EA Program is executing the ED EA Completion and Use Plan and has made progress in
addressing each of the areas of improvement identified by OMB as evidenced by ED’s EA Self
Assessment Summary and EA Self- Assessment Details. Feedback provided in OMB’s
Response to ED’s EA Completion and Use Plan indicated that ED’s planned actions have
achieved the next level of maturity. OMB currently rates ED’s EA as “effective,” as indicated
on the June 30, 2005 President’s Management Scorecard for E-Government:




Audit of the Department’s Performance         29                        Final Audit Report
Based Data Management Initiative
ACN: ED -OIG/A11-E0003
                                                         ., ".'.....      _-
   _"'__
                                                          " " . _... "" Of ED""ATK>M




   _
   --
  ~_>CO«''''''


              --._--           _",....,"'"
  ......... .............. ... ........ ..
       -~
                     ~
                   "..---                             """""""',~



  _. . . . . ___,-_.._"'_......
  "'-"'RENT

                        . ._'._·_
              ST.rus <XlLUUN

                                . . _--,-., . _----111 . . -
                                            .......                             ""

                                                                                                      :.=.
  fho....--...
  _       .. _____    .___          ... ".-.
                        . . . ,...___    .___    _,."'     ....
                                                     ......-.   _ " ' _ "'."",m
                                                                ___ ... ___  ___
                                                                                    ,..."' ...W""'_.
  _
          .._W                     ___     ._"X,           ____
                          _--
                                                                                              __

                        _
                                                                                                        ",on




  _.___---,_
  ~.




       _
  "' .........
                  _
               _._--,,_
  IY)."""""'_"'_" _ _ ';'._,..-.. "_" •..-_
           ......!!1111....
         _____- - _ __ _-
      _._--
                                  : .... _ .........
                                              .. _ ... _ ... ,_ .......... _.... .......
                   .. ... __ ... _-.....-, ........... ........­
                  _, """,,_...-         ~




         ... ... _..... . ... ... ._ .... ... ......
         .....
                                                                                   ......... "" ... '"



     .-. _
                                                                                      ......
    - ---..-          _--------....,-...--
         .. ... .........
         . .-.----.  ~"  ...
                             . . --.. - . . ---,<,-.. --.. . . . . .
         -""" .... ,,""""...........             • .. 1.... - - - - - - - , . . . - - , ... _ _ .. """' ....... -

                                                                                                          -~




    -- ....-----    .. - ..... .... -­         --... --
    _... _---, .. ...... _- ...... _--
              .... - ....
    """""' . . <:<>0..-


                             _---
                                          -~-
                                        -- ~

                                                  _,
                                                                      .
                                                                          -- ~        ....


           _..              __ _---
                                                              .--..

               . -,-".  . -_---.
          _~ :;.;'~;~~.;,.~~~4.-   . . _ ...
          ----
          -.,"'"
   ·-.. ._·-._---_
         --~.--
         ..."       _  -----
                       .. -.-. . _.,
               ----""' ---.. _-.-_ -
    ._..»-.. ----.
   .... _.-. ........        . -       . . . . --,-_00- . .
                                                         ..
                                                      --~--
                                                        ...
                                                            __
                                                                          ._"   ...              .,.-~-




                                                                                                            ..
                                                                                                                 ..


   -----_ _,--_.
       ----~                                                                    -.-


   -----._-_._ __...-
   "" _ _        ~_
                            ... - . ..
                                 ....           .....
                                                              .....
                          . . . . . . . . . . _ _ ft . . . - . . _




                                                                                                            ---
Audit of the Department’s Performance                                     30                     Final Audit Report
Based Data Management Initiative
ACN: ED -OIG/A11-E0003
Technical Corrections

(1) The audit report incorrectly refers to the records of technical assistance provided to SEAs
    by the Partner Support Center:

    •	 Page 5 “customer support center help desk compliant logs” should be “customer
       support center help desk logs” or “customer support center help desk records.”

    OIG Response: The language in our final audit report was changed to the PBDMI 

    terminology: “customer support center help desk records.”



    •	 Page 11 “Specific SEA comp laints included:” should be “Specific technical assistance
       issues included:”

    OIG Response: The language in our final audit report was changed to the PBDMI 

    terminology: “Specific technical assistance issues included.”



(2) The technical assistance issues are not accurately described on page 11.

        •	 The first bullet implies that the submission system calculates totals and records
           totals. The files provided by the SEAs often include subtotals and grand totals.
           When a file includes subtotals and grand totals, the submission system checks to
           determine if the subtotals are less the grand total. If the subtotals are more than the
           grand total, an error is generated. The log that was reviewed was simply an SEA
           working through resolving an error in their file. It was not an “inaccurate data
           calculation” by the Submission System.

        OIG Response: We do not believe the report makes an inaccurate implication as
        claimed by the Department. However, the report was rephrased in an attempt to
        address management’s concern and to clarify that the submission system did not
        incorrectly calculates totals and records totals. Accordingly, the wording in the final
        audit report was changed to read: “Inaccurate data calculations – a SEA noticed an
        inaccurate data calculation of 465,000 when the accurate total should have been around
        1700. Other SEAs identified instances where grand totals of data submitted were
        incorrectly smaller than the sum of the respective sub-totals”


        •	 The second bullet implies the EDEN is responsible for missing data. The technical
           assistance that is the basis for this bullet was from an SEA informing ED that the
           SEA does not require that schools provide the SEA with a phone number and
           therefore the SEA does not have all the phone numbers of the schools in the state.
           The data is incomplete because it doesn’t exist at the SEA not because of anything
           to do with EDEN.

        OIG Response: The Department’s response stating the report is making an implication
        to EDEN is inaccurate. In our report, we provide several examples regarding the
Audit of the Department’s Performance            31                          Final Audit Report
Based Data Management Initiative
ACN: ED -OIG/A11-E0003
        quality of data submitted to the Department and did not specifically state that EDEN
        was responsible for the missing data. Therefore we stand by our report findings and
        have not changed the wording in the Final Audit Report to address management’s
        concerns.

        •	 The third bullet implies either that the Submission System is changing data values
           provided by SEAs or creating its own data values. The Submission System wasn’t
           incorrectly reporting a data value. The Submission System was showing an
           incorrect value because an incorrect value had been inputted into the System.

        OIG Response: The Department’s claim that the report implies that the Submission
        Sysem is changing the data values or is creating its own values is incorrect. In our
        report, we provide several examples regarding the quality of data submitted to the
        Department and did not specifically state that EDEN was responsible for the missing
        data. Therefore, we stand by our report findings and have not cha nged the wording in
        the Final Audit Report to address management’s concerns.

(3) The concern listed on page 11 “program managers question the capacity of EDEN” may
    not meet the test of evidence because program managers have only partial knowledge about
    the system on which to base an opinion.

    OIG Response: We do not agree with the Department’s comment that program managers’
    statements about the capacity of EDEN does not meet the “test of evidence” standards. To
    gain an understanding of challenges facing PBDMI, we interviewed program managers,
    who interact directly with the system, to determine whether significant implementation
    risks have been effectively mitigated to ensure that PBDMI will meet its investment goals
    of consolidating current data collection activities in a way that improves data quality and
    reduces the reporting burden for national education partners. From these discussions,
    program managers expressed concerns because the PMT has not demonstrated to them that
    the quality of data collected will be sufficient to meet their operational needs. In order for
    PBDMI to consolidate the Department ’s multiple data collection efforts and meet its
    investment goals, the PMT must demonstrate to program managers (to obtain their buy- in)
    that the adequacy and quality of data collected will be sufficient to replace the program
    offices current data collection efforts. As management has previously indicated, data
    quality continues to be a significant risk that impacts the success of PBDMI, and the PMT
    has not effectively mitigated significant implementation risks related to this investment.
    Therefore, we stand by our report findings and have not changed the wording in the Final
    Audit Report to address management’s concerns.

(4) The description of the implementation of the EDEN Submission System is misleading. 	It
    implies that the PMT was not prepared for these issues. To mitigate risks, the EDEN
    Submission System was brought up in a planned, three-step approach. Version 1.0 was a
    usable system for the submission of files from SEAs. Version 1.1 addressed any issues that
    developed in the initial use by real users in the SEAs as apposed to test situations. Version
    1.3 brought in the data framework functionality for use by the PMT.

    OIG Response: Within the “Background” section of the Final Audit Report, we provided
    additional explanation about the PMT’s system development program. Specifically, we
Audit of the Department’s Performance            32                         Final Audit Report
Based Data Management Initiative
ACN: ED -OIG/A11-E0003
    added: “According to the PBDMI PMT staff, the EDEN Submission System was
    developed to mitigate risks under a three-step approach. Version 1.0 was a usable system
    for the submission of files from SEAs. Version 1.1 addressed any issues that developed in
    the initial use by real users in the SEAs as apposed to test situations. Version 1.3 brought
    in the data framework functionality for use by the PMT.”

The issues listed on page 13 as “significant system functionality defects” are mostly minor
coding issues of specific data fields. For example, the first bullet regarding the field for
“assessment for promotion/graduation status.” This data field is for an optional data group.

OIG Response: Based on our review of EDEN Submission System Release documents (1.1 and
1.2), we noted numerous system functionality defects in early versions of EDEN that required
coding modifications to ensure proper functionality of the system. We considered the
following classifications to be greater than “minor coding issues,” and significant because of
their nature and because a coding correction was required for subsequent releases of the EDEN
Submission System: 1)“High” – correction is required because “no work-around ” can be
identified; and 2) “Very High” – correction is required and “critical for proper system
functionality.” As the system functionality defects that we identified were classified as “High”
and “Very High” within the EDEN Submission System Release documents, we do not consider
these defects to be minor coding issues. Therefore, we stand by our report findings and have
not changed the wording in the Final Audit Report to address the Department ’s concerns.




Audit of the Department’s Performance           33                         Final Audit Report
Based Data Management Initiative
ACN: ED -OIG/A11-E0003
             Other Matters and Statement on Internal Controls

Other Matters

During the testing of the DSS Pilot, SEAs expressed concerns about whether education data
was adequately protected. Although the Department does not collect data at the individual
level, certain SEAs stated that education information about individuals could be derived
because sufficient access controls had not been implemented. In response, the PMT
implemented role-based security controls to eliminate privacy violations associated with “small
cell” data inquiries.

Statement on Internal Controls

As part of our audit, we reviewed internal controls related to the PMT’s ongoing system
development efforts for the EDEN Submission System, Survey Tools, and DARS. We
performed our review, in part, to determine the nature, timing, and extent of our substantive
tests to accomplish our audit objectives. Because of inherent limitations, a study and
evaluation made for the limited purpose described above would not necessarily disclose all
material weaknesses in the internal controls. Our assessment did disclose internal control
weaknesses that have impacted the system development efforts of EDEN.

In brief, we have determined that the PMT needs to improve certain project mana gement and
system implementation controls to ensure that the EDEN solution will ultimately consolidate
the collection of education information in a way that improves data quality and reduces the
paperwork burden for national education partners. In addition, the Department needs to fully
develop its EA so that Meta Data Dictionary is defined to identify the data elements collected,
submission formats, permitted values, applicable business rules and warnings, and the need for
information. Furthermore, the EA Technical Reference Model needs to be fully developed to
define technical standards, products, and services that must be adopted when implementing
information systems. These weaknesses and their effects are fully discussed in the “Findings”
section throughout this report.




Audit of the Department’s Performance           34                         Final Audit Report
Based Data Management Initiative
ACN: ED -OIG/A11-E0003
     Appendix A: Department’s Response to Draft Audit Report




                                    UNITED STAn:3 DEPARTMKNT OF EDUCATION

                                    " " , ,' U"'I'lAN"l>OG. "'>.W.<TIOl< ","p ""'"''' OC\'EWM<f:NT

                                                           SEP12:!X6

                MEMORAKDUM


                TO;             John lIisgilU
                                W]>t<tOt Grneral

                                Do" id Colo
                                         Inform.,ion T«hnology Audi"
                                0;"":101".

                FRo.\t


                SUBJECT,       ~"""""" '" IJrafi Ao<lIt R,1>OI\' Alldll ofth. Dcpartmcn". Paformmco
                               Bucd Data Managemen( Inui."". ('0111",1 "0. ED-OIGJA I I -EOOOJ


                Thank l""U lOr 1ho: """"nunity (0 "" i"", tl>e rlr.ofi aud" rcrort on the P",form""". BII.I<d
                D.to M>n>.ge"",nl InHiM,,·c (PBD~tl), Tn m)' fir" n." mooths .. ",,,,,!.ant S«reI.ry r",
                Plonnini, E'aT"I''''' and Polit), o.'·clopmcnt. I bove embraced PBDMI ., my top
                jJliori(y 111< s«rcury, "'00 . Iso Ii", P8D~!I 1l/OOTl<! ~ 10]1 prioritia. aud J 'S"'" tbot
                ach,«,>'mK 1ho: promi.., of No Ch,1d Left IM"nd dope",;. on "', ... to lh< kIM' of do"
                on4 .... Iytit 1001. bemi ~n<ralod toy PBD.\U . We , .. Iu. l""ur """,m""".IoIl"", "" 00"
                we con "=i'h"" PBI)\II and h,... < already impkmrnk<l mo" of them_

                We _        ""ilh ttl. fiodillj! that 'y>tcm impkmcn"'u"" OOI1t.ol. fur , .. oo"okopn"",( of
                tt10 EDEN . )'St'"'" could be imp",,"od and hO". ~rn "<1" to do!hoI w"kr, "Tl<!le
                operal,""'.   mlUntrnan<o and rnlwKcm"u contf"Ot:1 rOO" all PlID:'>11 ' )~.<m •. 0. ... quali'~
                io an ""!!<""i cllal l<n~ "..,th ;tot. aOO local ruocatioo data 'hat P I!I)M [ i, Jeoill"«110
                add",.. imm<di ... ly >Old O"Of the: IoIlg tam. The doto qWlh.y 'lISUes ldet1ufoed Ln )IOIIf
                repon "'<rC tt1_ "",,,",1.01 t.y PIJOMI proce .... or>.! h.,·< krn acted 00 ""ttl oor >!ate
                ponnetS, Ro~3nlin¥ the: impro"'-'H'"'' of ""1,.; .... ,."1. and toofll:W"tion mOll.lgcmcnt
                proc ....., the ~.p,d de'elopn"",1 melhodolollY rhooon by !'Em~tI by,,. ""ure resul(, i-.
                initi.1 ,-.m"", of the o)'Otero t~at .... ,efioed ;n nol>soqurnt ....,mons. As !'BD~II
                m"'"teJ. "e ,,'ill .rami<ion from 110< rapid &""Iop"'ent moth<>doloJ)" to • moTe trad it iorW
                ')"81crnJi ole"dropm"", ... 4 m"intrnlmCe mothodol"iY_ rOD\U'" ",'0"'00 tk>oel)" with
                OCIO to dC'·.I"" an Entcr;>ri'" Ar<h""",uro ,;s.ion. PMDM I dot. definot,,,,,, on<!
                >truetures bee ..... Lhe: b ..i. for Ihe Otponnoen", Oat. ArcM«tu.-e.

                Ll \Y. ""ommen d Iho D<p uIJ' & "ot.,}, d. ml. additio . . I ....... '""".o <n • ..-•• he
                   P8D ~ 1I I'MT r a n odrquo .. ly odd ,,,,,, Iht ') . (~ m d ••·. lopm<a . rI.", Id.nUIIeJ I_




                                              -.. _- -_ -
                                        _---- --
                   'hi< "p'''I.
                                             "'" """"""""' ..... _ "'... _     . oc ..."

                      "'- -    "   ..                                                                ....
Audit of the Department’s Performance                           35                                   Final Audit Report
Based Data Management Initiative
ACN: ED -OIG/A11-E0003
               Rn_ .. :          1hc PIIOMI PMT _ I • .,.-..l byt>o.o nco. FTEo III the Iu, q..... orof
               :/OCIol In ,0.);,,,,,,. pnm.y ~,"I"y "" prognm oulf!, ..."", on PliO.... ! dolo
               dcmcnlS,.".. "'JII«i to ,ht , ...... clorn.." .. y and oocoDdary edll<.lI"'" pctlQrmaocc
               anal}"" in the Somo';' ....<»Iln,."h,y Son'ice. ,,"" _ ... ~ omo .. in
               m.'"t .. n,n~ I~C'lf propam perf""",""". dolo, A nc",' ~ ""'Iui. i,;.;:,. ,.,..." Ie:oolC'l
               ~i'ion .... l>een <011,pcted or.l " pond'"S • "''POnS< to the ofT... A. pMt of,1I<:
               S""reIIry', fCQrpt1i . .""" lIoo, crt"""  'ht Oflk. ofPlom,ng, h.I ..." "" or.l Po>licy
               De>eloproon~ lwo new d'l""'r a&I'OW\IICC~"'Y _,""'" Ito"c t>cao crntod .. 'th
               primary """"""boh'y .... PIIOM ' policy «IC)rdi"""iexL Abo ...... ,,,. 0.\111 ...... ,"'" "
               usipod IO ..Qrt ~H,me ,,·,th "'" PIIO\U ""I.. y' ...... The PMT • <wn:ntly
               m ....." "11 ""<nIl1 PIIOM I ...m,. ic>'cI.1O........, tNI "'" PMT em odcqIo.,dy o.JcIreoo
               II>< .ystom de ......... '""~ ,ub Jdcmilkd '" !hoI f<pM and .",.., pno;ca      roqu"""""''''
                                                                                    d.,.
               1,l P ............. id.o.~ '" St:.V I.I:A. 100 ....-• ...,101 rlf.ttj,-. I.... , , •• , ..... (~.
                   «Ij, .~ b.>i .... p.-.o) 10 .-.rill 01", q..liI~ . ' .... <10'1000           tI.,••• _'"
                   ...... 'h' _'10 .... M')· h ............ d..-iooa .... <101 ....1000 .... d ...
                   'n••...w.... _
               s,.,."
               R.. _",:         Ooto ~"a1i'y " """U",""" ,mpro'en'oO<Il ~ ,n/oormo '" oloe
               !lclLICoIion Daoa h>c"",,lI" N.. ,,'OI'k_ 1'h< CUO'T'Clll q..li,y of " "'c edUCalion dol. ptU"~
               m",h room for improv'meni. PIIO\' I         n..  mul"plc "w<~ .. ."dmo.'ly (0 _"""SO OM<!
               ""st "....    to impro .. !boor dOl. ,,,,,c •• .00 tho qu"hly oftbcir dau.

               PHO\l1 i, • 'ys(cm.a". ",,..,,,,, '0 oddrtoo ."".. problems 1lw ex," in C..mnl S,." ....
               ,....lDB ~'stema;
                          1 S,.,. dolo ........ rtmd dtT<C'ly frocn Siale odnu ....u.JYe ~ 1)'II<mI
                               "*'
                                 ~OE'. _           N "",,.lraDI<ribed by S..... "'...... , oflkioIJ onlO I
                               Federal oqoOr'<m& bn> Tho. d"nillllCO """ potenIlal """"'" of .....
                           2_ EDDi·'5nI""Pi",""..... S_datl ................ """',h«b.                       n....
                               ..., <bopotd 10 .... '" _          _n&. ~ .. "" pmioo.tl
                                                             .... iIk
                               -.u.-, ....      ~                  ....
                                                                 b pre-defi ..... ,..,...........
                           J EDEN', pJrtMI S<oppon C...... ............ ..-. such .. oI>o»e """",fled on
                             2. 110m.....,Ih s...c. ....... ""!"C"' oaubm,_ of........ obta..
                             <Ioc:unooted by 0 ... 0
                           4 PBOMI .... """',.."... • 'th .... C.", .. b 001a Qual,!), (nOQ) to .....,_
                             tho ~.r1OCy ofSl.1tc data ""'m;"'or. and to "",vide: """"""",••hoi"""
                               roo- ;mI""Hn_
                           S   ~I}OM I .10" '''''' IaC!..! .. ,," C~DQ 10 pro"ide d.", qu. lttr mo ,.....   foo-
                               ,rodj"iduol Sl.1"" and 10 f'!1'1',d< ,hom reromonendouono and      toe""...1
                               ....1Ion< .. C~OQ hu <o<oduoclC<l rc ......·• in 10 S...... ,,, dole




                                                                                                                   ,


Audit of the Department’s Performance                         36                                      Final Audit Report
Based Data Management Initiative
ACN: ED -OIG/A11-E0003
                           6. P80MI has provided and will continue to provide data qual ity trainin~
                              and technical assistance to State PBDMI Coordinators in collaboration
                              with NCES and CCSSO.
               As data arc reviewed for quality, additional error checks will be identified. As eITors arc
               corrected in an individual State, the potential for similar errors will be identified to other
               States and added to the review list for the Partner Support Center.

               1.3 Develop inlernallesting procedures and performance measures 10 ascertain tbe
                   completeness, accu racy and reliability of education data before the EDE.N
                   Decision a nd Analys is Reporting System becomes operational.

               Slatus:         Ongoing

               Respo nse:      We agree with this recommcndation and are aware ohhe need to ensure
               that high quality data is used for policy decisions and program management. Many
               actions have been taken or are planned to ensure EDEN uses thc best available data (see
               1.2 for a summary). EDEN data edits are based on pennilled values and prior ye3rs'
               CCD submissions. As EDEN accumulates multiple years of data, it will be able to
               expand thc database foundation for automated edits. In addition, program omce staff and
               perfonnance analysts arc reviewing state data submissions against previous program
               omce submissions as a basis for follow-up inquiries with states. These inquiries will
               help States identify speci fi c areas on which to focus their data quality efforts.
               Importantly, the reporting system itself will enable additional analyses and reviews that
               will further support our data quality etTorts. Because the reporting system wi tl be
               operational before all data quality issues arc finally resolved, users will be trained on the
               appropriate uses of the data.

               As mentioned earlier, data quality is a continuous improvcment process. In thc interim,
               the data available for EDEN far exceeds the quality of what has been previously available
               to policy makers and program managers.

               1.4 Implement eomprehensi\'c requirements and configuration manageme nt
                   controls to develop EDEN integrated systems that will meet system fUlletionalit y
                   and performance requiremcnt, as well as cneounge a proactive system
                   development approach.

               Status:         Implcmented July 2005

               Respo nse:     ED selected Perot Systems, Inc. as contractor for PBDMI Operations,
               Maintenance and Enhancement. null contract ineludes enhanced configuration
               management controls negotiated with OClO's new contractor, CSC, Comprehensivc
               requiremcnts were laid down in the contract and arc being further specified in a series of
               action plans under development by Perot and review by ED,




                                                                                                                3




Audit of the Department’s Performance                       37                                 Final Audit Report
Based Data Management Initiative
ACN: ED-OIG/A11-E0003
              1.5 Develop comprehensive procedures so th e PMT ca n proacli\'ely defin e, collect
                  and slandardize a co mplete listing of standard data elements a nd definitions in
                  con nection with PBOMI.

              Status:          Implemented September 2004 and under annual review

              The PMT has complctcd in collaboration with OCIO/RIMS the second PBDMI collection
              "lIhmi~~inn   In OMR Thp- pmpn"p-n ?OOt) _?OOR ml1p.C'.Iinn hllilcls nn   lh~.   ?M4 _?OO'j   cl~l:1

              elements and definitions with minor changes necessitated by the new mEA
              reauthorization and similar programmatic changes. The identification of data elements is
              an iterative process. PBDM I began by asking program offices to define their infonllation
              requirements (legislation, regulation, reponing, monitoring, program management, and
              risk avoidance)_ It then reviewed these requirements against data currently gathered by
              ED programs and offices, identifying gaps, redundancies and inconsistent definitions.
              Thc resulting data set was reviewed with the states for availability. Concurrently, a series
              of policy questions were developed with senior officers and data clements reviewed and
              modified to assure that those questions could be answered. We expect user requirements
              to evolve substantially as program omcials and senior omeers are able to access these
              data and further clari fy their requirements.

              The OPEPD planning team and new Executive level PBDM! Steering Committee will be
              undenaking a review of all elementary and secondary data elements to assurc that
              programs rely on standard data elements and definitions that are aligned with PBDMJ.

               1.6 DC\'clop a formal cbange control p rocess that will govern changes to the EDEN
                   Meta data dictionary and bu siness processes associated with PBOMI data
                   collection and reporting activities.

               Status:         Implemented July 2005

               Response:      Pcrot Systems has proposed and has implemented a formal change control
               process under the oversight of the PMT and Steering Committee.

               1.7 Develop co mprehensive training programs and guidance to assist SEAs in
                   reducing on-going data submiss ion errors with the EDEN Submission System.

               Status:         Ongoing

               The PMT and Perot Systcms arc updating the EDEN guidance doeulllents and training
               based on the proposed 2006-2008 data clcments. Two and a hair day training sessions
               have again been scheduled for state stafr. Two regional trainings will be offered, one in
               Denver, CO, October 24-26, 2005, and one in Washington, DC, November 2-4. Both
               sessions will include introductory orientation for state staff new to the EDEN system.
               Wc expcet ncar 100% participation again this year.




                                                                                                                     4




Audit of the Department’s Performance                      38                                      Final Audit Report
Based Data Management Initiative
ACN: ED-OIG/A11-E0003
               l.I         Wo ,",,,mmond .ho O. pu.y Seemof}' d • ...:, •• oddi.io.ol ...oulU. ' o o., ~ ••
                           •ho PElD.\1I PMT con .d'q. ."ly .dd ..... he , )"t. m d.,·.lopmo •• ,;'k.
                           Klrn.ifi.d ia .~ i. report.

               S••   ,u"          Ongoing

                                  See rapon><: 10 1.1

               1.1         n.,,·<lop ,ompreb.n.i,·o project m••• ~ m.. " <ontrol, ' o .rr.. th,.ly m.u ~ •
                           •h •• imob ro,nplt1"" of P,(ljod mil ... o . .. ..d 0....... m""lplt '(In'''' '(lr>
                           '«jlOn.ibl< for d,v,loplnllh . [l)[:\' iD1<g ..1<d .)~ .. m •.

               Sto' ""            Imp lemented Nov<mber 2004

               M..   po.'"        Pl.IllM11tas • trW!" project ""hedulc which tho I'MT h.. ,,"',,",'00
               ..."CCI;ly" nh "".iYHontor1<>I'C an.) OCIO ro< .... put )"<:.,- om! 1 half. Th. Oper:aioo-n ••
               Maint<rWr. ond Enhanc"",,,,,t <oot .... l."'anletlio Porot S}'''<RIlI, Inc __ " .• , designed to
               ''''''''''lOn from til< "'"hipf< oonlnCtors ''''Iuired (0< d.velop",rnl '" • • ingk contrac. to
               nwugc EDEN with the PMT. Thi. tr1nS;.ion w;1I be oomplele .. 1h<."d ofthi.             f,"""
               )'C.r.

               1.3         [)e,.. lop .on' proh.n.i," ro • ••••• m... g.m.... <O'''rol•• o , n,u ••• _ ..
                           ""olro<' d . lh·... hl ... rt pt,fot notd .nd lund. d .. b.<Wd in .«o.don« ,,·i.b
                                                                     'h.
                           roa ......<qu i.. m..... ~'U o1~ <r .. 'o",    P," T nff<!,'0  d..·. lop p.oc.d....
                           '0 .n,u, . ,h ••• 11 Sto,,, pro,·KI. «Iu, .."'" do,a 10 ,uPPO" 0' PIIl).\II.

               slO,u"             Part I   iml'I""~l«J July 2005;   Port 2 ""lI"ing

               " .. pon ..,        Comp",hen. ive ron"'" rn""' gem"'" .ontrol, Mv. been pu, in ~I"". '0 •
               •he """'vrcl>aui,·. O~cr."io"", M. in.en:ance and Enhan.:em .... contrao, " i th PCf<>I
               Sysltrns, Inc .. be""n in July 2005. The PMT h.. dcwtopOO 1 projecl rn'"'''lIe"",,'' plan
               with Perot tha, i, ",v,,,,,-ed ",'e ckly, The COR .... been de,i"",,,,, .ho ooord,tloto< ofthc
               projoe. management pion. A. ",."... ",., pi .. " .Iso on pi"" ...ith   P."" om! .be itg""y
               OO",,,,,,tors whi.h i, coord,n .. ed by th< . Item ... COR fo. ,he Peru, oor:ltrxL

               The ne'" Offi,e of Plonruna. E,·.loution :and Poli.y Developm"'l t.... ,,"veloped I
               ..... ,<gy.o pro"ide 'he ;"" ...,i,'" 10 ".,.. to <n>ure that 01 1Sta••• pro,id.e «I"'.toon data
               in .upport of PODM I. Th,' stra'ei)' ir-.:ludoo oommuni,ation by "", ior offie ..... w.i,·"
               or rq>OJ1ing "'GuIf.m"",, to tho .. S...... lhat ""." transmitted th< ~ircd d . .... om! dratt
               re¥uI01i<>n' mfuiring S.... wbmi .. ion, .hrough EOEN,

               3.1       T n hrlp <n,ur. Ih. 'U'<n> of FElD." I ..... ,«omm.tld 'ho. 'h . flep."",.n ' "
               U ; ... ,,,,,rdlft ••• ,.I.b ,~ . rMT ,<0 .dop •• ~DUMI M". D••• d H;.ion0f}' 10'0
               Dtporlm.n.', ~:,\ 10 fort h" dell". ".ndord do, ••1.OMnB. d, •• d.ft.i'ion, •• d ,h .
               bu,;n ..... I........ ia.«I wl.h roll«tlnK ,h...., ,I" •• k """'t>.




                                                                                                              ,

Audit of the Department’s Performance                         39                                 Final Audit Report
Based Data Management Initiative
ACN: ED -OIG/A11-E0003
               Statuo: Implement«l M"",b 200S

              lI... p" .... : The ED EA team h.ts ok".\oped and implcm<:llt«l.., ED I'-A Do ..
              [J'ctt"nOlY. "'hieh ",dudes ate",,,,. m~ [,eld •. 1\ i. maint';ned in the ED EA
               R""",,,OtY      i.
                              ond currently ,,'';Iobl. to !>eportmen\ " .... _i.
                                                                               Connc<:tED
              (http'/lcon!!CC!<ll l oN irQ' mea/). The Dqmtm.nt", ED EA D"" Team .... blished the
              Em""""" Dota Dicti<>nary AuSU', 2004. A com:spooding Ent"""'. D.la Dictionary
              Uoer·, Guido w.. d,veloped a1thc >am< timo, An Oycrviow of the En' .."..... DJ!>
              D,cI,onary ..... oomplcu:d [),;comber I. 2004 ,nd i, pooted on Con"<.'CtED
              ( http://ron oo;!W l.ed, i<O\'1?9i g,OOq, ccdm::<dd o)'ea lOW !Ips).

               Bc1wccn August 2004 >nd M"",h 2O(jl. llIc: EnlcIJlriK D0'" [)jctionary odded 1.494 dOl.
              ''''>en" indud"'g those de\'Oloped by PBD~U (EDEJ'\."), lnoorpo..tion of odd"io",,1 (\ala
              e\en\ettts aM assooi",od motadato .... oontmuod. A, of luly 2005 thc ED EA EnterpnK
              Data Dictionary <o[uamod data from fourteen dota model. ",",lin, 2.072 do .. dem","
               Incor;><>Ta""" of dota .Ie.,en" in", the ED EA Dm D,ct;on.ry i. ,onlinuin8

              Tho intent of c>Ltbli!liing Ih"lkparlmrn,-Wid, ED EA Oat. Dicu",,"IY           w"
                                                                                             lO clearly
              Mr."" "..-.l.rd do .. form ... and pcm1i(1od ..I... , ""ross thc Dq>mmrnt. The .. OUI1<I.rd
              formot. and ".Iuco ~ d,""tly ",1"00 to tile ED EA Ent"'l'ri .. ConcO!>luoJ Dota Modol.
              and to LogicoJ ..... Ph},.;.;.1 d." model< dcve1opc<l ,,"hin nn"", progr>Im • ...,h ..
               PBDJ<.11 (EDEh1

              E"""';ng full ..,ppoM of PBD~I (ED£!<.1 from the program offices take. 1"0 pallu. Thc
              fin! is ,he Informat;on Col lection "1'1"0,..1 proc .... W"h ,n the ~I. !hal proce ..
              include •• "vi.", of propo;t;<l ""lloctio", to _          Ii<ll how much of Ille collection em be
              ""taiDed through I'BD~U (EVEN), St=od. . 11 ocw IT projcc .. ""ithm tile Dq>;utmcnl
              mu.ot eonfonD ,,> im<mal Lif. Cycle M'n~t review and ' 1'1'1<",.1 ..gat"":. 1",lwed
              .... gatco for rtvi .... of tile toehnic. 1archi'oe!"rt (incluoJmM dat.). At !hot 'imo. the
              ",!:it ion,h,p bel"" ..... propo>ed oollOClion aOO til< eXl".",e ofrel.vant do" ,n e_d,,"t
              eollocti"", i. c>Ltbli<h<d . TIt ... proc«l",es hlOI! oot BO"ocmod pr~gram omce     ')'$l"'"   or
              do!. <olle"io". prio:r to the i>.",""h ofI'BO.\lI, HowC"CT. these rnocil.>n ism, .... """ 'n
              plo« (Information eon"'tion "1'1"01·.1) .,. "ill be in p]"'e .oonl y (life Cycle
              M . n _ n l ",view) • .00 ""iii he LJsa1 to ourt.i] the .... of ,edund ... t m fonn", ion
              oollechon aM data ""' ",0 IICliv;ti=

              3.1     ~u(() d<l'tlop ,ho 1:,,\ T..,hk.1 R.r.rena Moo.I to .... bibb 'orb.lu l
              "a.dud. and I.form.""" .....11}' .... qu l.. ""n" [h . 1 are •• NINI to b.tld »".m,
              In ".PlIOn uf the !J'p." ... . t·, <T!tl<o' npo .. tl .....

              S I.. ",: Implemented M .y 2005

               R.. pon .. , ED·, EA documentation library FY2005 -Q3 role_ "",hides ED'. T.dUl"aJ
              Rof""""", Modol and T<:ehi1olo~y Stmd.>ni. l'rofilo """um.n". 1l>ese documcn" were
              """eloped [og",he, to ..'abl;," l<:ehn,e.1 0land..-d1. prod""" • .00 "",,·k.. lO build
              s)·"em. in  "'won of ED·, blUl""", (u"",ion.lind inronnotion ..con[y requ ,,,,,, ... ,,.




Audit of the Department’s Performance                       40                                  Final Audit Report
Based Data Management Initiative
ACN: ED -OIG/A11-E0003
              n.... documenu.~ 0100 ovo;lobi< on COM<CIED
              (bilpB9!JDSt"!sd 1i!ntioo:'W<dysni<>n100!=95.J I ED EA St"" l'm,htmll

              ED', TOOMk. 1~rcm.;< MO<k1 (TR.\'\) " "oo,..t.." "',th Qv<tS'i!It documenlJ ..
              follo ..'o;

              •      '" ~"';IhOM[l', EA ~ F.... <OIlH V<r$ion 1--'.1::0', TR.\,\
                  ............... poac:ut !be _""""'Y_       1O.,....anu ED', _ _ rr p<O<IuctJ IIIC!
                (_ po",",cury~-'one! _ . o.-l'R-\I w.ooomylhol ....1tcs EP',
                E!. "'lOll and m,"1011 II<Cdo one! .Ito oli.gno with lite FEA TIt.\I,
              •      In K«Irdao<e ,,"',th 01.18 ClfC~Iar" Ill), ED', TR.\I idont,r... IIIC! <ko<riltcs tit<
                ,nfomtat"", IerYlI><s used tltrougllO... tho 'g<!C'y,
              •      In a<""nlanoe "ith the 1'rat:',c.1 Guido 10 t'oJ .... ] Entc.pn'" "",h" .... un:, ED',
                TI<1>1 " ow"""",y IlIat provt<l ..
                " • ""n".,,,,,t .... of ....i<.         art""
                                                         ,,11,,"_ ''''<gorin, and .. lallOflSk,pI used 10
                    .oddn:.. in'cropcrabil;.y and .,.,......)'>Iom i ......
                " .~ .. I mli,'" Iho1 ntobIa/I • """"""" vocabularf 10 btl! .. detcrilw:,
                    """'fIOR. IIIC! C<IIlIrU! syuan.1IIC! OOUtpOtLOtl
                o • boooo "" Ibt ........,f.........
                      ........ ontI tIt<-ot_PI
                                                          "."'P"""'\' """odO<boo of <'XiItJ"I     MId
                                                                                                   <merpos




              •   In ~ ,,"11 O\\B', EIt "-.","1 ~''"''''' .. '''''' Vernor> 1.5, ED'I En.apriK
                  SWIll.,.... "",me <altai .... Ii" uf """,~ory ,mcfO!>C1"bilily ,!>M. rdI
                  clll'T .... ly"OO"' ED, and "_IIINO 1UI.n<lml.IO ~R.\I >UbfWIC'io .... SRM""";ce
                  C<lmpon ...... 000 Tll.M IlCn'ICO c"'(IOne..
              •   In ..."","""" .. illI O!l.lB 0=1 ... "·I:KI. [ I)', S,andud, i"rofilc ~.fi.... the oeI of IT
                  ........ Iho1 0\IPII<01 the ocrvicd .,"'u~ '" ED', TUI, and inol..,... Sontri'Y

                  """J.....uon    " 'jtll ED',
                  ..... 't<d. ,damrlCOIXtD, ...     _I<_..t
                  ScandonIo ""'file thai .. >pecif", LD Ihc .........'1.......:cs opcciFJoJ in !be EA -.I, in
                                                &od,,,.
                                                     S<atnly k"'l""_ doctmlaOl. ."'..... _
                                                                      _r<p><IuI"",; audil !nil
                  lII>Iy,..; occ.. «tIdn>!s; a)'I"""aplty ...... ccm<"'~ ,'iru$ pn>l<CllOtI, fnotd
                                                                                                    aut,,,,,I0Il

                  provatlton:  cIeIOCIion \OI'Id m.bplOll, """ ,......- ~_ >lid <Iot«11Otl
              •   In O<C<Ini:trlC<:   '0"'"
                                        !he Praclical Guido LD Ffdcnl Ezt'«prioc ltJclti,OCt....e, EO',
                  S\aDlJaro. Profil. i. the oc, o f ",leo lhai ..,,'....')'1\'"  'mpkmrnlltlOll ond O!KT'ti<>n,
                  ,<11:,"""" 11>0 t.dlllie.1 st.t>d.tnI, ""'I '[>ply 10 1110 or<b"ec,u", ond 110'0' .hoy need l~
                  Iw:. 00' 1'11... been. itnplcm<l'l'ed, ond i ,   \lmo-p~                  ,,,,,,,ured,
                                                                             '" r"",h'o'e • •
                  d'K 'plt""" pro<:'" ~f ')'>Irnl dcoidopment ond evoiLJIioo oro! to promo\e the
                  ~IOII ~f ttnetg>9/I: l«hnoIopeto one! !be I,k.liltood of <WT01tIl«bnoloPeo and
                  0IAnI,I0nI0 bccom"'l        """"lou
                                                    EO', TkM."......ueo ED', S~ Ptor.le. EI) "­
                  po ....... Ill< l'R-1.! """ Standardl Prof,le .. ~ docuoncnu inIIeaI of • •
                  """,boned _               , Haoo~...., EI)', SI>IdonIo ""'fil< ....... oil ollhc
                  'n .......... ioo .....,,"" ,n ED', TkM one! " "'""'~ • .."...... "rEO', TR.I.!



                                                                                                               ,


Audit of the Department’s Performance                              41                               Final Audit Report
Based Data Management Initiative
ACN: ED -OIG/A11-E0003
             010 Itmd !hal ~A I'",T Iq;o ......... e _ _ the '-'k of. fully d ...·eIo!><oJ To<l.I>O>.I
             Rcfero:nce Modd c......:! <IeI. yo in iniJil<mm\lnlllll< EDE.'" Subml ....... Syslem Thi,;,
             """au", lbe: !><pliltm",l '. pt<>l<>C<>1 .,. nd",,<Io. WI1fi¥unLioo, firewall. and anri-"iNl
             .... Ulrancn .. fOl implo:menloil;\lprod..,,,.....                                             M
                                                                 )'>1 .... , ••, "'" 004 olefin«! "';lhin Ibc: I!A.

             ED', TR-\t and SIaIIdardJ Profile ........ ' ............... _ mpDWI IbtNiJdardo and
             """"Iones. 00<10>01"., pral<>i:OL <Of\fipor.oon. f"",,-..lI. and .."i,vI"""'-" tNt . .
             pubhsbcd in ooun:c o*umenlJ lhot were u>«l by 1be: EDNct Technoc.l R ...,;ow B~rd
             prO<cu .. Ihe limo of PBOMr. T IlB ' ...·Iew., ThO$< ~ (\o( uIIIMlJ IO<lLl<k III<
             EOSeI Prod""1 Suppon Pi ..... "'hoc:h ia • Ii.. ofPR>d..," >Od pnoo.Iu<1 ...... """ ...,
             ~ by ED".. ()petaIiom. ord the EDN .. Technolo!Y S~ G.."k V_ ~ .
             ... hoc:h ... """" ~'''' 1111 ofprod\oc:1J _ lechnical opec,flaiioDl ouppOrICd ....
             1001'... ED"'.. •• TecllnoJoIY Sl>n<Wd. Guide Ven ..... 4 iden.. r"", EO:-:",', proIOCOI
             "andardo. con fi II""" 100. fi "",-.11. • ",I Onto-" I"'. r«t"1 ,<", ....1•.

             J..I       Imp'.'" ,~.I)q>.rI ..<.,'. <M,... I r.A •• d . ... rt, d.n •• ,M [A Tor'l"
             "rr~I'«1."      .ad T ....,;';. "".,~ ....... 11' , olit_ .. ~. llor OMII r ....""1 t'.A.

             s,.,." Implc""",lOolIWl<:wH
             11.",.,..... : EO',I!A Pro~ il (>:_1'1111: Ibc: EO eA CODipktion ond u ... Plato &lid tIM
             on&ik ""'VOW on tddra>i"ll ... h or ib< .. CIllO ofi,...o.'CnaII. idtrou~od by O\IB ..
             ... ookt>ccd by ED', F.A Self , , _ Summaryond I!A Self" ..               I..... Ikuolt.
             r.,.,.n,.:l p""'id«J ,n O.'>1B·, R<tpOrIf< to ED', I!A Compl<tion ord U'" Plan lodlnlcd
             1"- ED', planned ..";"",, han ochlc"od the "'~I I "".I of mal.oty. OMB <WT<olly nota
             EO',I!A .. "effCCli,c." os indloalN on lhe cnoloJe<l J...... 30. 200S J> ... id""f,
             Monopnml So<offi:ord fo< FA)", ........... (bon! ~y ... ..,hod~




             (I) Tho .o>oIlt repo<l IncorrecIly re ..... to \be: f«OnII ofl«hn ic.al _""""" p""yidod to
                 SI'.A. by tbc: ~onn<r Suppcn Cent...,

                    •   "OF S '"cusIomet iUppOIturrterho:tp dcol: .,.,..,p.... IopMIboooId br~
                        ~      _1Idp daoI: IoIlfM 01' """"""'- "'JiPOr1 _ .... bdp dcslo: 1'<><Ofds.M

                    •   p~ II ''SpOCifoo SEA """,pla;",. ;no1ud«l,Moboold be "Spee,fie lecioruoal
                        ...;<Unte I..."..' inclOOcd ,"




                                                                                                                  •

Audit of the Department’s Performance                          42                                   Final Audit Report
Based Data Management Initiative
ACN: ED -OIG/A11-E0003
                      •   The first bullet implies that the submission system calculates totals and
                          records totals. The files provided by the SEAs often include subtotals and
                          !,'fund totals. When a file includes subtotals and grand totals, the submission
                          system checks to detenuine if the subtotals are less the grand total. If the
                          subtotals are more than the grand total, an error is generated. The log that was
                          reviewed was simply an SEA working through resolving an crror in their filc.
                          It was not an "inaccurate data calculation" by the Submission System.

                      •   The second bullet implies the EDEN is responsible for missing data. The
                          technical assistance that is the basis for this bullet was from an SEA infonuing
                          ED that the SEA docs not require that schools provide the SEA with a phone
                          number and therefore the SEA docs not have all the phone numbers of the
                          schools in the state. The data is incomplete because it doesn 't exist at the
                          SEA not because of anything to do with EDEN.

                      •   The third bullet implies either that the Submission System is changing data
                          values provided by SEAs or creating its own data values. The Submission
                          System wasn't incorrectly reponing a data value. The Submission System
                          was showing an incorrect value because an incorrect value had been inputted
                          into the Systcm.

               (3) The concem listed on page 11 "program managers question the capacity of EDEN"
                   may 110t meet thc test of evidence because program managers have only panial
                   knowledge ahoUi the system on which to base an opinion.

               (4) The description of the implementation of the EDEN Submission System is
                   misleading. It impl ies that the PMT was not prepared for these issues. To mitigate
                   risks, the EDEN Submission System was brought up in a planned, three-step
                   approach. Version 1.0 was a usable system for the submission of files from SEAs.
                   Version 1.1 addressed any issues that developed in the initial use by real users in thc
                   SEAs as apposed to test situations. Version 1.3 brought in the data framework
                   functionality for use by the PMT.

               The issues listed on page 13 as "signi fican! system functionality defects" are mostly
               minor coding issues of specific data fields. For example, the first bullet regarding thc
               field for "assessment for promotion/graduation status." This data field is for an optional
               data group.


               Attachment




                                                                                                             9




Audit of the Department’s Performance                     43                                Final Audit Report
Based Data Management Initiative
ACN: ED-OIG/A11-E0003
                                        _-.'...       ..,-_.
                                        OIl ...... ' .. e . T Of EtMJCATION




Audit of the Department’s Performance                     44                  Final Audit Report
Based Data Management Initiative
ACN: ED -OIG/A11-E0003
                                             --'.".         __.
                                              DE ......... E " T~ EOOCA'ION




          _
      . ............ _-'._-
       ............
  """'-"""'-
   . _ ......               -""'.-
              .,.. ___ """'(1\"",._


                                .......... ._       ,                                              ........... . -
   .....-. . _"_"""
   _
  ... _-..--""
     ......".
                ......
                ....  ..   _-..-.
                             ..-.,            . ...  _
                                                 ,-----       ..         -
                                                     ................. . . -----I<"
                                                             _.,. """'-"""'
                                                                 _""   .    ..           ... _""-
                                                                                               . .
                                                                                                 "'~-...         -...
  "-.,._.
   w_' '......
                   '-
          '----'''''-_   "~
                             -
                         '---..--
                _ U ...... "" ....... _ _
                                                           ... __....._
                                            ..... _._'''''''
  'Y),fG''''a·I_ ... _ ' ' _ _ ·,,·_,__ ,,_ ... _ _
                                                                           _....--...__. _....._.,.... .
                                                                       ... _· ThO ........._
                                              ._"Xj .................. _
                                                                         ·~
                                                                                                   ....... _ _
                                                                                                  , """"-'


            _
           .... _----,_
  ..,.,,~-,,-


           _"'    ...
           --.-~---    _    ..
                          """'
                                -
                          ............
                   .. . ..·----.   _
                                            -
                                    .......__
                                              .·
                                                      ....-
                                                      ..
                                                ...1"__' .
                                                           --"'.-.,
                                                            .  ._""-
                                                              --_-_--1 ""_
                                                                                  . -­
                                                                                     .... . . --....,....-­
                                                                                                     "  " ",    . . . '.
          ~"
          .~--
          ~~--
                              "'"---
                          .......    "-- . . --"'-.       .......---. --_I·I-.
                                                                   ..........   -"- . --.. . . . . .
                                                                                      ......"._



                                                                                                            -~




Audit of the Department’s Performance                        45                                  Final Audit Report
Based Data Management Initiative
ACN: ED -OIG/A11-E0003
                                                                     oF EDUCAT"~
           .on _ _ .. _ ....                       OOP ....... ENl
                                                                            ... __ ..... _
                                     . . __ . . . . _
                                          .,_-..._T..,~




    _--_---,..... ..... ....,.-------
    u>o_"'~_""_




         .....
                     .."'....
                          ~""~-~   -
                                             ~




                                           , ..-_
                                   ....--~~~ ..
                                                                 ......
                                                                          ...
                              .. +     _ , _ _ ,,<____... _ _ _ _                                                               ..
          --""'--
         ''IOU''"'




                                  - ... _-,                        . _"''''"'''''--, .. _........... _-
                     _
                         ~
                                                                                                "
                                                                                                    '
                                                                                                        _
                                                                                                            ~
                                                                                                                "
                                                                                                                    "
                                                                                                                        "
                                                                                                                            '
   ...._..........--.. .......... _-,.-
   _. __ .......---.......---"'...-- ---_.. _,
        I<ID _


            ..-"',_
    ""..,.--~
                                         ...... ----
                         ........ _ _ "' _ _ _ _ . .... _


                                                              -
                                                  ................ -               ..........




   "'-_.'" = ---.. -                             .. ~--,

   """"'-"'''''''''''''''.....     ___""'-
   ""'-.......,... ...... ""'-- ....
                                           .....   -..-.,."




Audit of the Department’s Performance                                46                         Final Audit Report
Based Data Management Initiative
ACN: ED -OIG/A11-E0003