oversight

ED's Implementation of FMSS Oracle Federal Financials Phase II and III.

Published by the Department of Education, Office of Inspector General on 2002-01-14.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

MEMORANDUM

DATE:          January 14, 2002

TO:            Mark Carney
               Deputy Chief Financial Officer
               Office of the Chief Financial Officer

FROM:          Lorraine Lewis /s/

SUBJECT:       FINAL AUDIT REPORT: ED’s Implementation of FMSS Oracle Federal
               Financials Phase II and III (Control Number A11-C0007)

This audit report presents the results of our limited scope work related to the implementation of
the Financial Management System Software (FMSS) Oracle Federal Financials (Oracle
Financials) Phases II and III. The purpose of our audit was to identify potential risk areas in the
development and implementation of Oracle Financials. Our audit included a review of: (1)
testing, including interfaces and data conversion; (2) the status of the development of interfaces;
(3) independent verification and validation (IV&V) of Oracle Financials development; (4) Oracle
Financials training; (5) Oracle Financials security; and (6) the status of maintenance/support
plans for Oracle Financials.

Due to the planned January 22, 2002, Oracle Financials implementation date, we performed
limited scope audit work from October 31 to November 30, 2001, in order to provide you
information on potential risk areas before the new system is implemented. Specifically, we
focused our analysis on identifying risks in the development of Oracle Financials compared to
best practices, standards, and regulations. We generally did not attempt to quantify the effect of
these weaknesses or determine the underlying causes.


                                       AUDIT RESULTS

We identified risks in several areas of Phase II and III implementation: (1) test planning
documentation is incomplete; (2) testing of interfaces did not include all controls; (3) complete
IV&V will not be performed before implementation; (4) training may not adequately prepare
end-users; (5) user access controls do not follow security requirements; and (6) post-
implementation operations and maintenance plans have not been fully developed and
implemented. The Office of the Chief Financial Officer (OCFO) needs to address these risks to
ensure that required functions and controls will operate as intended upon implementation.
ED's Implementation of FMSS Oracle Federal Financials Phase II and III     Final Report                    ED/OIG A11-C0007



We also reviewed data conversion and identified that 4,250 problem items were reported. OCFO
officials stated that they are addressing these items. As of the end of our audit work on
November 30, 2001, we had not received complete documentation on how conversion problems
were being addressed; thus, we cannot evaluate how problems were resolved. This issue is
summarized in more detail in the OTHER MATTERS section of this report.

Management Comments and OIG Response

In the Deputy Chief Financial Officer’s (CFO) written comments to our draft report, OCFO
officials did not generally concur with Findings 1, 2, 4, 5, and 6. The comments did not include
a specific reference to Finding 3; however, in a subsequent electronic mail message, the OCFO
concurred with Finding 3 on the need to complete IV&V activities prior to system
implementation. The Deputy CFO also suggested wording revisions for clarification, which we
incorporated as appropriate. We have summarized OCFO's comments and provided the OIG
response, as appropriate, after each finding. A complete copy of OCFO's comments is provided
in ATTACHMENT B.


Finding 1: Test Planning Documentation Is Incomplete

During our fieldwork, we identified several concerns with the OCFO’s application and
integration testing of Oracle Financials Phase II and III functions required by the Joint Financial
Management Improvement Program (JFMIP).1 According to documentation we reviewed, we
identified that (1) some required functions are not included in test plans; (2) some functions were
only partially tested; and (3) actual testing results and supervisory reviews for many of the
functions were lacking at the time of our review. Risks in these areas could affect proper
functioning of required functions and controls when the system is implemented.

System development practices require documented test plans, test scripts, and test scenarios
detailing how each requirement is to be tested in order to provide assurance that required
functions will perform as intended. Testing staff needs to be provided detailed test scripts and
scenarios in order to fully test each function. However, we identified several weaknesses in the
testing. In a review of 52 sampled JFMIP required functions (provided in ATTACHMENT A),
we could only determine that 7 were fully tested.




1
    OMB Circular A-127 Financial Management Systems (Revised July 23, 1993), section 7g states: “Agency financial
     management systems shall conform to existing applicable functional requirements for the design, development, operation, and
     maintenance of financial management systems. Functional requirements are defined in a series of publications entitled Federal
     Financial Management Systems Requirements issued by the Joint Financial Management Improvement Program.” OCFO’s
     document, New FMS Accounting Model, February 23, 2000, documents in Section 2 Accounting Model Best Practices, the
     need to use JFMIP financial systems requirements when testing compliance of commercial-off-the-shelf products. This
     statement indicates OCFO’s acknowledgement of the need to use JFMIP requirements for testing of its system.




                                                                                                                                2
ED's Implementation of FMSS Oracle Federal Financials Phase II and III       Final Report                     ED/OIG A11-C0007



Of the remaining functions:

            •    9 functions2 were not referenced in test plans, test scripts, or test scenarios; thus, we
                 could not determine if they were included in testing;
            •    21 functions were only partially addressed in test planning documents; some did not
                 have detailed test scripts for use by testers; for some functions, only certain types of
                 transactions were tested; and some tests were performed using only valid data and did
                 not test using invalid data; and
            •    15 requirements lacked documented test results and supervisory review.

Without completed and thorough test plans, scripts, and scenarios for required functions and
documented test results and supervisory review, OCFO may not have assurance that required
tests have been satisfactorily completed and incurs a risk that required functions may not operate
correctly.

Recommendation:

1. To ensure that functions will operate correctly and comply with JFMIP requirements, we
   recommend that the CFO ensure that the Oracle Financials Implementation Team completes
   and fully documents test plans, scripts, and scenarios for JFMIP requirements and ensures
   tests have been successfully completed.

Management Comments and OIG Response

The Deputy CFO generally did not concur with this finding, explaining that applicable guidance
does not require an agency to test core financial management functions not applicable to its
operations and that some of the test plan discrepancies cited for the sampled requirements were
caused by functions (1) not deployed during Phases II and III, (2) not applicable to OCFO
operations, (3) not properly mapped to test plans, or (4) being documented in test plans related to
other Education Central Automated Processing Systems (EDCAPS) components.

The comments do not specify which of the 52 sampled requirements meet which of these
conditions; therefore, we could not thoroughly analyze the Deputy CFO’s statements. With
regard to the statement that some of the JFMIP requirements relate to functions not deployed

2
    The nine functions are:
     1. Accruals of contracts or other items that cross fiscal years.
     2. Separately identifies amounts that would be eliminated when preparing intra-agency and interagency consolidations.
     3. Multiple pre-final closings to accommodate incremental adjustments and closings.
     4. Year-end rollover of appropriate system tables into the new fiscal year under the control of an authorized system
        administrator.
     5. Reconciliation of all open accounting period (prior month, current month, prior fiscal year, and current fiscal year) balances
        to their respective subsidiaries through on-line queries and reports.
     6. Designated authorities to establish and modify the level of fund control using elements of the classification structure,
        including object class, program, organization, project, and fund.
     7. Designated authorities to establish and modify the system's response (either reject transaction or provide warning) to the
        failure of a funds availability edit for each transaction type.
     8. Identify payees who receive 1099s, including 1099 Cs.
     9. Comparison of the agency's payment schedule and disbursing office's accomplished payment schedule.




                                                                                                                                    3
ED's Implementation of FMSS Oracle Federal Financials Phase II and III   Final Report   ED/OIG A11-C0007



during Phases II and III, we note that the requirements we reviewed relate to Funds
Management, Purchase Orders, and Accounts Payable, which are the identified purposes of
Phases II and III. We are, therefore, concerned about when OCFO plans to implement these
requirements if they were not deployed during Phases II and III.

With regard to the statement that 3 of the 52 sampled requirements are not applicable to OCFO
operations, we reanalyzed information provided to us after the end of our fieldwork to determine
any discrepancies between this information and the information provided in Finding 1. We
could not identify any mapping or explanation of which JFMIP requirements are not applicable
to OCFO operations. The 52 sampled JFMIP requirements are identified by JFMIP as
“mandatory.”

We are not clear as to the statement that some requirements are not properly mapped to Oracle
Financials test plans. OCFO’s integration contractor, PricewaterhouseCoopers (PwC), explained
that there are no specific requirements documents and that contract deliverables were based on
discussions and electronic mail messages regarding requirements. OCFO needs assurance that
requirements have been adequately tested and determined to function correctly.

In regard to the comment that some of the functions’ test plans are related to other EDCAPS
systems, we did not identify references to other system test plans and did not receive
documentation for these plans even though we requested all documentation related to testing of
Phase II and III functions. Therefore, we cannot independently verify any functions that might
have been tested outside of the Oracle Financials specific testing environment.

The comments dated January 2, 2002, explain that some of the documentation requested was not
available during our fieldwork, which could account for some of the differences between our
findings and the comments. We recognize that testing was ongoing during our review and is still
continuing. We are concerned that the comments specify that, as of January 2, 2002, Application
and Integration test results packages have not been delivered for 6 of 30 Phase II and III test
areas. Application and Integration testing was scheduled for completion on November 2, 2001.
With Oracle Financials scheduled for full implementation on January 22, 2002, and all
Application and Integration test results packages not yet received by January 2, we affirm our
finding and recommendation that the OCFO complete and fully document test plans, scripts, and
scenarios for JFMIP requirements and ensure that tests have been successfully completed to
verify that functions operate correctly.


Finding 2: Testing of Interfaces Did Not Include All Controls

To ensure the integrity of system data, testing of interfaces includes testing of controls that data
is complete and accurate and that data interface submissions are complete. OMB Circular A-127
Financial Management Systems (Revised July 23, 1993) section 7j states, “Appropriate internal
controls shall be applied to all system inputs, processing, and outputs.” Our analysis identified
that testing did not include controls to ensure that duplicate information is not processed and that
data is being provided by an authorized source.




                                                                                                           4
ED's Implementation of FMSS Oracle Federal Financials Phase II and III   Final Report    ED/OIG A11-C0007



While interface controls for data integrity may be inherent within Oracle Financials software,
testing procedures included reviews of those software controls related to data completion, data
accuracy, interface completion, data from or to an authorized entity, and duplicate data
processing in ED’s interface test plans.

We reviewed the test plans that were available for 20 of the 23 interfaces. All 20 of the reviewed
plans included controls for data accuracy and complete interface submission. However, none of
the 20 interfaces were tested to ensure information is provided by an authorized source, and 19
of the 20 interfaces were not tested for adequate controls to prevent duplicate information
processing. Testing for three of the interfaces, Nortridge Promissory Notes, Checkfree, and
Lockbox was still ongoing during our review.

Recommendation:

2. To ensure the accuracy of data within Oracle Financials and connected systems, we
   recommend that the CFO ensure that the Oracle Financials Implementation Team fully test
   appropriate controls for each interface.

Management Comments and OIG Response

The Deputy CFO did not concur with this finding, explaining that for interfaces, OCFO uses a
combination of operational controls and automated controls to ensure that the source of the data
is authentic and that duplicate data is not introduced. The assertion that controls exist does not
provide assurance that these controls have been adequately tested. The OCFO comments
provided a sample set of references for three interfaces where the comments stated that adequate
testing was completed; however, the testing information that we reviewed during our audit did
not provide a complete description of the test purposes, making it difficult to determine that the
testing performed was sufficient to conclude that adequate measures are in place to avoid
duplicate transactions. Adequate testing of interfaces must be performed; therefore, we affirm
our finding and recommendation that OCFO fully test appropriate controls for each interface.


Finding No. 3: Complete Independent Verification and Validation (IV&V) Will Not Be
               Performed Before Implementation

In our report on Phase I implementation of Oracle Financials,3 we identified that the OCFO and
its contractor had not completed minimum IV&V tasks as specified by Institute of Electrical and
Electronic Engineers (IEEE) Standard 1012-1998, Software Verfication and Validation. We
recommended that the CFO direct the IV&V contractor to perform the minimum IEEE-specified
IV&V activities for the implementation of Oracle Financials and analyze what other IV&V tasks
need to be performed. OCFO officials concurred with this finding and have taken some actions.
However, the CFO will not have IV&V tasks completed to identify potential system risks prior
to the implementation of Oracle Financials.



3
    ED's Implementation of FMSS Oracle Federal Financials Phase I, ED/OIG A11-B0003 (December 17, 2001).



                                                                                                            5
ED's Implementation of FMSS Oracle Federal Financials Phase II and III   Final Report        ED/OIG A11-C0007



OCFO had utilized an IV&V contractor for Phase I of Oracle Financials but did not exercise the
option to continue the contract. OCFO has developed a Statement of Work for Phases II and III
IV&V activities but, as of the end of our fieldwork, had not yet awarded the contract.4 The
IV&V Statement of Work specifies that because OCFO is in the latter stages of the
implementation project, the contractor shall perform pre-implementation and post-
implementation tasks in accordance with industry defined best practices such as standards from
the IEEE and Software Engineering Economics by Boehm. The pre-implementation tasks
include:
           •    Tracing requirements to system design or testing scenarios;
           •    Acceptance and component test execution and verification;
           •    Retesting code including code reviews on interfaces, enhancement scripts, and
                customizations; and
           •    Risk mitigation assessments.
Post-implementation tasks include:
           •    Working unobtrusively alongside the integrator while they conduct validation testing.
           •    Independently verifying the inputs, processing, and outcomes of the testing relative to
                the expected results.
           •    Re-executing selected test scripts and/or scenarios, as deemed necessary by the
                contractor or as directed by the Department, in order to validate and verify the initial
                testing outcomes.

In addition, the OCFO awarded a contract in October 2001 for an Agreed-Upon-Procedures
review of the accounting logic within Oracle Financials. The objective of the procedures is to
assist the CFO in obtaining assurance that, upon implementation, the accounting logic (the
chosen debit/credit pairing assigned to each accounting event) in ED’s financial management
system will result in financial statements that accurately depict ED’s financial condition. OCFO
planned for the work to be completed by December 21, 2001. OCFO officials stated these
procedures do not serve as independent verification and validation of the Oracle Financials
system development effort and do not, nor are they intended to, include the IV&V tasks specified
in the Statement of Work.

Without an IV&V assessment of the Oracle Financials system development, the CFO is missing
a key tool to provide assurance that the system will provide required functions.

Recommendation:

3. We recommend that the CFO identify the most critical IV&V tasks to be performed and see
   that they are completed to ensure that required functions operate as needed.

Management Comments and OIG Response

The Deputy CFO did not specifically respond to this finding in his comments. However, in a
subsequent electronic mail message, the OCFO concurred with Finding 3.
4
    OCFO officials informed us that they had awarded a contract for IV&V effective December 13, 2001.



                                                                                                                6
ED's Implementation of FMSS Oracle Federal Financials Phase II and III   Final Report               ED/OIG A11-C0007



Finding No. 4: Training May Not Adequately Prepare End-Users

In our prior review of Phase I "Just-in-Time Training," we identified potential Oracle Financials
end-users in our sample who did not receive the necessary training to efficiently and effectively
use the system. We recognize that the Oracle Financials Training Team has taken action to
improve training based upon the recommendations in our Phase I report and through their own
end-user survey work. Our review of a sample of user surveys found positive responses about
the training. In structured interviews with a random sample of 12 users,5 we identified issues
similar to our Phase I review. These end-users explained that for the training to be more
effective, it needs to be tailored to users’ specific needs and levels of system access.

The Training Team is now providing training more frequently rather than on a “Just-in-Time”
basis, and the team is making greater effort to encourage attendance and notify users of training
schedules. Of the 285 Phase II and III end-users that signed-in for the training, at least 193
completed training evaluations. We reviewed these 193 training evaluations, provided by the
training team, and noted that they reflected improvements in the training.

According to OMB A-127, “Adequate training and appropriate user support shall be provided to
the users of the financial management systems, based on the level, responsibility and roles of
individual users, to enable the users of the systems at all levels to understand, operate and
maintain the system.” In interviews with a random sample of end-users and Training Team
Liaisons, we identified that a number of concerns remain. For example, we noted that:

1. End-users are still receiving “Super User” access in their training classes, which in many
   cases may not be similar to the access they will use in their everyday job performance.

2. End-users are finding their training to be either too general or area-specific for their job
   needs.

3. System changes due to customization are affecting end-user attendance and training team
   instruction.

Even if it is explained during training that the Super User level of access may be different than in
everyday performance, the training would be more effective if access during training was more
similar to the access to be used in the end-user’s everyday job performance. End-users also
expressed that they felt training was provided at too high a level, without specifying individual
job function, and some expressed that it was too much for their needs. Additionally, we
interviewed 4 out of 25 Training Team Liaisons who reported difficulties in providing the
training because of system changes. They described attendance issues as end-users wanting to
attend training at a later date, after system customization is complete. Training Team Liaisons
also reported difficulty in keeping the training adjusted with system changes/customizations due
to a lack of communication with the Oracle Financials Implementation Team.



5
 For Phases II and III, there were a total of 395 end users identified as requiring training. We stratified the list of
end users and randomly selected six end users from each phase.



                                                                                                                          7
ED's Implementation of FMSS Oracle Federal Financials Phase II and III   Final Report              ED/OIG A11-C0007



Addressing these training concerns should help reduce the risk that potential end-users will
experience difficulties in using the system and performing their job functions.

Recommendation:

4. For the remaining "pre" and "post" implementation training, we recommend that the CFO
   direct the Oracle Financials Implementation Team to consult more thoroughly with end-users
   prior to the training to identify their specific training needs.

Management Comments and OIG Response

The Deputy CFO did not concur with this finding, explaining that it would be impractical for
management to create user access levels specific to a student’s job tasks for each classroom
participant. Our concern is that the Super User access in the training might not be similar to the
access the students will use in their everyday jobs. We recognize that, due to the number of end-
users needing training, it would be difficult to train at the exact level of everyday access. We
recommend that the OCFO conduct more thorough consultation with end-users to identify their
specific training needs, which would help identify groups that should be trained together due to
their similar access levels bringing the training access closer to actual end-user access.

The comments indicate that OCFO is working closely with training liaisons to encourage
trainees to register and attend training classes. We recognized the increased effort to encourage
attendance and notify users of training schedules in our draft report; however through interviews
with several Training Team Liaisons, we identified that system changes due to customization are
affecting end-user attendance and training team instruction. More consultation with end-users
prior to the training will help to alleviate their concerns regarding system changes and encourage
their attendance. We affirm our finding and recommendation.


Finding No. 5: User Access Controls Do Not Follow Security Requirements

Some Oracle Financials users do not have restricted access to only those functions needed to
perform individual job duties. We identified that there are an excessive number of individuals
with “GL Super User” account access and one user with multiple User IDs. We also identified
that most of the user responsibilities are different from those specified in the Accounts
Payable/Purchase Order (AP/PO) Security Strategy Document. Without restricted access, users
could inadvertently or maliciously access Oracle Financials information possibly making
unauthorized modifications to the data.

We identified 16 user IDs with “GL Super User” access. OCFO officials stated that two of the
users are system administrators who need Super User access to implement updates and changes.
Super Users have a wide range of functions and privileges, which allow them almost unrestricted
access to Oracle Financials information. The Federal Information System Controls Audit
Manual (FISCAM),6 section AC-2.1 states, “Broad or special access privileges . . . are only


6
    General Accounting Office, “Federal Information System Controls Audit Manual,” January 1999.



                                                                                                                      8
ED's Implementation of FMSS Oracle Federal Financials Phase II and III       Final Report                     ED/OIG A11-C0007



appropriate for a small number of users who perform system maintenance or handle emergency
situations.”

To ensure that users would not inadvertently be assigned multiple responsibilities that could
provide them with an inappropriate level of system access, we reviewed the Oracle Financials
AP/PO Security Strategy Document provided by OCFO. Our review indicated that while the
security model provided in that document was sufficient to keep users from being assigned
incompatible responsibilities, the model was not followed when actual user roles and
responsibilities were assigned to staff. OMB Circular A-130 states that a set of rules should be
established concerning use of and behavior within the application; such rules shall clearly
delineate responsibilities and expected behavior of all individuals with access to the application.
FISCAM AC-2.1 further states, “The computer resource owner should identify the specific user
or class of users that are authorized to obtain direct access to each resource for which he or she is
responsible.”7

Recommendations

5.1. We recommend that the CFO ensure that the Oracle Financials Implementation Team
     determines the minimum number of users with Super User access and restricts access to
     only those who need Super User capabilities to complete their job functions.

5.2. We recommend that the CFO ensure that the Oracle Financials Implementation Team
     makes certain that user responsibilities adequately reflect the AP/PO Security Strategy
     Document to enforce security and access controls.

Management Comments and OIG Response

The Deputy CFO did not concur with this finding, explaining that the security requirements we
reviewed during our audit were specific to the test environment and not the production
environment. The OCFO comments specify that if the security requirements reviewed were for
the production environment, OCFO would agree with our finding that the controls do not follow
the requirements. During our audit, OCFO provided the lists, Active Users and Their Active
Responsibilities, showing the application names and responsibilities for users of Oracle
Financials. These lists were represented to us as the most current information of user profiles
already tested.

If the requirements reviewed are solely for the test environment, we are concerned whether the
access control structure has been tested for the production environment. Application and
Integration testing was scheduled for completion by November 2, 2001, with Customer
7
  FISCAM AC-2.1: “Resource owners have identified authorized users and their access authorized: The computer resource owner
should identify the specific user or class of users that are authorized to obtain direct access to each resource for which he or she is
responsible. This process can be simplified by developing standard profiles, which describe access needs for groups of users with
similar duties, such as accounts payable clerks. Access authorizations should be documented on standard forms, maintained on
file, approved by senior managers, and securely transferred to security managers . . . Broad or special access privileges, such as
those associated with operating system software that allow normal controls to be overridden, are only appropriate for a small
number of users who perform system maintenance or handle emergency situations. However, any such access should also be
approved by a senior security manager, written justifications should be kept on file, and the use of highly sensitive files or access
privileges should be closely monitored by management.”



                                                                                                                                     9
ED's Implementation of FMSS Oracle Federal Financials Phase II and III   Final Report   ED/OIG A11-C0007



Acceptance Testing occurring during the time of our fieldwork. We expected that the access
controls testing would have occurred at this time. The security requirements for the production
environment should be applied and tested on the test system before the system is placed into
production to ensure controls are in place and working as needed. We affirm our finding and
recommendation.

The comments also state that there are not an excessive number of users with GL Super User
access. As stated in the report, we identified 16 users with such access, and FISCAM states that
broad or special access privileges are only appropriate for a small number of users. Thus, we
affirm our finding and recommendation that the CFO ensure that the Oracle Financials
Implementation Team limits the number of users with such access.


Finding No. 6: Post-Implementation Operations and Maintenance Plans Have Not Been
               Fully Developed and Implemented

At the time of our fieldwork, documentation was not available identifying procedures to be
followed for the daily operations and maintenance of Oracle Financials. Basically, an
Operations and Maintenance plan provides computer operations personnel with a description of
the software and necessary instructions on how to operate the software including how to
complete non-routine, error, and recovery procedures. Though OCFO has indicated that time
constraints have not allowed focus on this area, OMB A-127 section 7i states that requirements
documents shall be adequate to allow technical personnel to operate the system in an effective
and efficient manner. In addition, National Institute of Standards and Technology Special
Publication 800-18, Guide for Developing Security Plans for Information Technology Systems
(December 1998), section 5.GSS.5 states that these procedures should be in place to ensure that
maintenance and repair activities are accomplished without adversely affecting system security.

Documentation of all aspects of computer support and operations is important to ensure
continuity and consistency. Creating and documenting post-implementation procedures for
operations and maintenance will reduce the risk for oversights in identifying recurring issues or
assessing system performance.

Recommendation:

6. We recommend that the CFO ensure that the Oracle Financials Implementation Team
   develops and implements an Operations and Maintenance plan for the Oracle Financials
   development effort.

Management Comments and OIG Response

The Deputy CFO did not concur with this finding, explaining that the OCFO document, Oracle
Application 11.03 System Operating Procedures dated September 27, 2001, and other documents
identified procedures to be followed for daily operations and maintenance of FMSS Oracle
Financials. During our audit, neither OCFO officials nor PwC contractors identified the
document, Oracle Application 11.03 System Operating Procedures, or the other specified
documents, but they did refer to the need to develop and implement an Operations and


                                                                                                           10
ED's Implementation of FMSS Oracle Federal Financials Phase II and III   Final Report   ED/OIG A11-C0007



Maintenance Plan. Subsequent to our review, we requested and reviewed the Oracle Application
11.03 System Operating Procedures and identified that this is a draft document which does refer
to a number of operations and maintenance procedures, but the document does not include other
procedures such as design of internal control and security procedures so that they can be
individually maintained, how to activate security procedures, or how to interconnect the software
with other related software or interfaces. These procedures are a step in the right direction, but
with Oracle Financials implementation scheduled for January 22, 2002, the CFO must ensure
that operations and maintenance documentation is finalized and complete. We affirm our finding
and recommendation.

                                                      OTHER MATTERS

With regard to data conversion, the Mock I Test Results Report identified that about 4,250
problem items were reported while converting nearly 20 million data items from the current
i.e.FARS system to Oracle Financials. During our fieldwork, OCFO officials stated that they
have addressed the Mock I data conversion problems. As of the end of our fieldwork on
November 30, 2001, we did not receive complete documentation on how conversion problems
were addressed; thus, we cannot evaluate how conversion problems were resolved. The OCFO
officials stated that they would run the Mock II conversion test to ensure that all problems have
been adequately addressed and to identify any issues in previously unconverted data.

                                                        BACKGROUND

The OCFO is in the process of implementing a new core financial management system to replace
the FMSS component of EDCAPS. OCFO is replacing the FMSS due to significant problems
experienced with the operation and maintenance of the legacy FMSS since its deployment in
October 1997 and due to deficiencies identified by financial statement auditors. ED has selected
Oracle Financials as the replacement FMSS. The implementation of Oracle Financials and
related interfaces is a large-scale system implementation effort. To minimize risks and manage
the complexity of such an effort, it is important that the work proceed in manageable increments.

There are four Phases in the Oracle Financials implementation effort:

     •    Phase I: AR and GL Phase - delivers AR and collections functionality and configures
          the Oracle Federal Financial General Ledger for subsequent phases (completed October
          2000). Refer to OIG audit report, ED's Implementation of FMSS Oracle Federal
          Financials Phase I, ED/OIG A11-B0003 issued in December 2001, for audit findings on
          the Phase I implementation.
     •    Phase II: Program System Integration Phase - delivers funds management, purchase
          order (i.e., obligation processing), and accounts payable functionality for program area
          funds and integrates the program systems with the new core Financial Management
          System (Scheduled for completion January 22, 2002).
     •    Phase III: Administrative System Integration Phase - delivers funds management,
          purchase order, and accounts payable functionality for administrative funds and




                                                                                                           11
ED's Implementation of FMSS Oracle Federal Financials Phase II and III   Final Report   ED/OIG A11-C0007



          integrates the administrative systems with the new core Financial Management System
          (Scheduled for completion January 22, 2002).
     •    Phase IV: Legacy FMSS Shutdown Phase - completes the transfer of all data and
          reporting functions from the legacy FMSS (Scheduled for completion post-
          implementation; a date has not been determined.)

As of December 2001, the estimated cost for developing and implementing Oracle Financials,
including IV&V and the Agreed-Upon-Procedures, is $27.5 million.

                               OBJECTIVES, SCOPE, AND METHODOLOGY

The overall objective of our audit was to identify potential risk areas in the development and
implementation of Oracle Financials. Our audit included a review of (1) testing, including
interfaces and data conversion; (2) the status of the development of interfaces; (3) IV&V tasks;
(4) Oracle Financials training; (5) Oracle Financials security; and (6) the status of
maintenance/support plans for Oracle Financials.

The purpose of this letter report is to bring to your attention concerns that we identified during
our audit of the implementation of FMSS Oracle Federal Financials (Oracle Financials) Phases II
and III. We generally based our work on the information provided to us as of November 30,
2001. Our analysis focused on determining conditions requiring corrective action and did not
always identify the effect or root causes for the conditions.

To accomplish the audit objective, we reviewed planning and implementation documents
relevant to our reviews of the testing of Oracle Financials functions, user access security, data
conversion, interfaces, enhancements and modifications, operations and maintenance, training,
and independent verification and validation.

Additionally, we interviewed program managers, Oracle Financials Implementation Team
personnel, contractors, selected Oracle Financials end-users, and selected Oracle Financials
Training liaisons. We also conducted interviews with appropriate officials. For the sample of
end-users interviewed, we stratified the list of end-users identified as needing to attend training
and selected a random sample of 12 end-users.

Our audit covered the Phase II & III implementation period. Our fieldwork was performed in
Washington, D.C. between October 31 and November 30, 2001. Our audit was performed in
accordance with government auditing standards appropriate to the scope of the audit described.

                              STATEMENT ON MANAGEMENT CONTROLS

As part of our audit, we reviewed management controls over the implementation of Phases II and
III. We specifically reviewed controls over testing, the development of interfaces, IV&V tasks,
training, security, and development of maintenance/support plans for Oracle Financials. We
performed our review, in part, to determine the nature, extent, and timing of our substantive tests
to accomplish the audit objectives.



                                                                                                           12
ED's Implementation of FMSS Oracle Federal Financials Phase II and III   Final Report   ED/OIG A11-C0007



Due to inherent limitations, a study and evaluation made for the limited purpose described above
would not necessarily disclose all material weaknesses in the management controls. Our
assessment did disclose management control weaknesses that adversely affected the
implementation efforts. These weaknesses and their effects are fully discussed in the AUDIT
RESULTS section of this report.

                                            ADMINISTRATIVE MATTERS

Please provide us with your final response to each open recommendation within 60 days of the
date of this report indicating what corrective actions you have taken or plan to take and the
related milestones.

In accordance with OMB Circular A-50, we will keep this audit report on the Office of Inspector
General (OIG) list of unresolved audits until all open issues have been resolved. Any reports
unresolved after 180 days from the date of issuance will be shown as overdue in the OIG’s
Semiannual Report to Congress.

Accordingly, please provide the Supervisor, Post Audit Group, Financial Improvement and Post
Audit Operations, OCIO and OIG’s Assistant Inspector General for Audit Services with
semiannual status reports. These reports should address promised corrective actions until all
such actions have been completed or continued follow-up is unnecessary.

In accordance with the Freedom of Information Act (Public Law 90-23), reports issued by OIG
are available, if requested, to members of the press and general public to the extent information
contained therein is not subject to exemptions in the Act.

We appreciate the cooperation given during the audit. If you have any questions or wish to
discuss the contents of this report, please call Andrew Patchan, Jr., Senior Director, Systems
Internal Audit Team on 202-863-9497. Please refer to the control number in all correspondence
relating to this report.

Attachments

cc:     William D. Hansen, Deputy Secretary
        Eugene W. Hickok, Under Secretary
        John Danielson, Chief of Staff, OS
        John P. Higgins, Management Improvement Team
        William Haubert, Assistant General Counsel, OGC
        Laurie Rich, Assistant Secretary, OIIA
        Greg Woods, Chief Operating Officer, SFA
        James Lynch, Chief Financial Officer, SFA
        Steve Hawald, Chief Information Officer, SFA
        Linda Paulsen, Deputy Chief Financial Officer, SFA
        Faye Harris, Acting Director of Internal Review Division, SFA




                                                                                                           13
ED's Implementation of FMSS Oracle Federal Financials Phase II and III   Final Report           ED/OIG A11-C0007


ATTACHMENT A

       Results of 52 JFMIP Requirements Reviewed

                 JFMIP Requirements Fully Tested                                            Section Total
                 Warehouses and schedules payments in accordance with applicable
             1   regulations. For example, OMB Circular A-125.
                 Provides the capability to capture, store, and process appropriate invoice
                 information, including: invoice number, invoice amount, obligating
                 document references, vendor number, payee name and address, discount
             2   terms, invoice amount, invoice date, and invoice receipt date.
                 Records additional shipping and other charges to adjust the payment
             3   amount, if they are authorized.
                 Adjusts the asset or expense recorded with the liability if the authorized
                 payment (based on the invoice) is different from the amount accrued
                 (based upon receipt and acceptance) using contract information and any
             4   increase is within agency tolerances.
                 Automatically adjusts the obligation amount and edits for funds
             5   availability to cover increases.
                 Automatically updates the funds control and budget execution balances to
                 reflect changes in the status of undelivered orders and expended
             6   appropriations, as well as changes in amounts.
             7   Provides for proper processing of payment confirmations and follow-ups.                7

                 JFMIP Requirements Not Referenced in Any Document
                 System allows for accruals of contracts or other items that cross fiscal
             1   years.
                 System separately identifies amounts that would be eliminated when
             2   preparing intra-agency and interagency consolidations.
                 Supports multiple pre-final closings to accommodate incremental
             3   adjustments and closings.
                 Provides for a year-end rollover of appropriate system tables into the new
             4   fiscal year, under the control of an authorized system administrator.
                 Provides for reconciliation of all open accounting period (prior month,
                 current month, prior fiscal year, and current fiscal year) balances to their
             5   respective subsidiaries through on-line queries and reports.
                 System provides for designated authorities to establish and modify the
                 level of fund control using elements of the classification structure,
             6   including object class, program, organization, project, and fund.
                 System provides for designated authorities to establish and modify the
                 system’s response (either reject transaction or provide warning) to the
             7   failure of a funds availability edit for each transaction type.




                                                                                                            A1
ED's Implementation of FMSS Oracle Federal Financials Phase II and III   Final Report            ED/OIG A11-C0007


ATTACHMENT A

               Provides the capability to identify payees who receive 1099s, including
             8 1099Cs.
               Comparison of the agency’s payment schedule and disbursing office’s
             9 accomplished payment schedule.                                                            9

                 JFMIP Requirements Partially Tested
                 System maintains historical data to produce comparative financial reports
             1   for management use.
                 Prepares trial balances and other supporting information needed for
                 external reports and financial statements, including consolidated
             2   statements.
                 Provides for on-line notification of funds availability prior to the
                 distribution of lower level funding and the processing of commitment,
             3   obligation, or expenditure transactions.
             4   Supports the timely recording of transactions.
                 Records the financial impact of all transactions that affect the availability
                 of funds, such as commitments, liquidations, obligations, and
             5   expenditures.
                 Updates all appropriate accounts to ensure that the system always
                 maintains and reports the current status of funds for all open accounting
             6   periods.
                 Adjusts available fund balances as reimbursable orders are accepted.
                 (Note: In the case of reimbursable orders from the public, an advance
             7   must also be received before additional funding authority is recorder).
                 Records an accrued liability upon receipt and acceptance of goods and
                 services and properly identifies them as capital asset, expense, prepaid
             8   expense, or construction.
                 Invoices are recorded through keyboard entry by a user or through an
             9   electronic interface with vendors in an electronic commerce arrangement.
                 Provides the capability of splitting an invoice into multiple payments on
                 the appropriate due dates when items on the invoice have different due
           10    dates.
                 Records discount terms and automatically determines whether taking the
                 discount is economically justified as defined in the Treasury Financial
           11    Manual, Volume I, section 6-8040.
                 Provides information about each payment to reflect the stage of the
                 scheduling process that the payment has reached and the date each step
                 was reached for the following processing steps: payment scheduled,
                 schedule sent to appropriate disbursing office, and payment issued by
           12    appropriate disbursing office.




                                                                                                             A2
ED's Implementation of FMSS Oracle Federal Financials Phase II and III   Final Report          ED/OIG A11-C0007


ATTACHMENT A

                 Updates payment information when confirmation is received from the
                 disbursing office, including the paid schedule number, check numbers or
                 trace numbers, and date, amount of payment, and payment method (check
           13    or EFT).
                 Posts transactions to SGL in accordance with the transaction definitions
           14    established by the core financial system management function.
                 System will selectively generate required transactions as needed by the
           15    year-end closing procedures.
                 System determines funds availability on adjustments to obligations or
           16    based on whether the funds cited are current, expired, or cancelled.
                 Allows commitment documents to be entered into the core financial
                 system on-line and from multiple locations, as well as through interfaces
           17    with other systems.
                 Maintains information needed to support Internal Revenue Service (IRS)
                 1099 and W-2 reporting, including TIN and payee type (e.g., sole
           18    proprietorship, partnership, and corporation).
                 Allows multiple payment addresses and/or bank information for a single
           19    payee.
                 Access previously entered information and/or record additional
                 information necessary to automatically determine the due date and
                 amount of vendor payments in accordance with OMB Circular A-125,
           20    based on invoices, receiving reports, and contracts or purchase orders.
                 Establishes payables and makes payments on behalf of another agency,
           21    citing the other agency’s funding information.                                      21

                 JFMIP Requirements Lacking Completed Test Documentation
                 Provides the capability to process, track, and control prior fiscal year
             1   adjustment transactions.
             2   Edit and validation routines used for Funds Availability Editing
             3   Checks commitment transactions against available funds.
                 Includes adequate controls to prevent the recording of commitments that
             4   exceed available balances
                 Supports recording obligations or expenditures that exceed available
                 balances and produce a report or otherwise provide a method that allows
             5   management to review the cause of this overobligation condition.
                 Provides the capabilities and controls for authorized users to override
             6   funds availability edits.
                 Provides automatic real-time notification to users of transactions failing
                 the funds availability edit and place the rejected transactions in an error
             7   file and/or suspense account for corrective action.




                                                                                                           A3
ED's Implementation of FMSS Oracle Federal Financials Phase II and III   Final Report           ED/OIG A11-C0007


ATTACHMENT A

                 Checks available funds for obligating documents (including Amendments
                 to obligating documents resulting in a change to dollar amounts or to the
             8   classification structure.
                 Checks available funds when the expenditure exceeds the obligating
                 document due to quantity or price variances within tolerances, additional
             9   shipping charges, etc.
                 Checks available funds for commitments and obligations incurred in
           10    support of reimbursable agreements.
                 Maintains information related to each commitment document, including
                 amendments. (Note: At a minimum, the system must capture requisition
           11    number, accounting classification structures, and estimated amounts.)
                 Provides for modifications to commitment documents, including ones
                 that change the dollar amount or the accounting classification structure
           12    cited.
           13    Edit and validation routines used for Vendors.
                 Maintains payee information that includes data to support obligation,
           14    accounts payable, and disbursement processes.
           15    Supports payments made to third parties that act as agents for the payee.            15


                                                                                        TOTAL         52




                                                                                                            A4
ED's Implementation of FMSS Oracle Federal Financials Phase II and III   Final Report   ED/OIG A11-C0007

ATTACHMENT B




                                                                                                     B1
ED's Implementation of FMSS Oracle Federal Financials Phase II and III   Final Report   ED/OIG A11-C0007

ATTACHMENT B




                                                                                                     B2
ED's Implementation of FMSS Oracle Federal Financials Phase II and III   Final Report   ED/OIG A11-C0007

ATTACHMENT B




                                                                                                     B3
ED's Implementation of FMSS Oracle Federal Financials Phase II and III   Final Report   ED/OIG A11-C0007

ATTACHMENT B




                                                                                                     B4
ED's Implementation of FMSS Oracle Federal Financials Phase II and III   Final Report   ED/OIG A11-C0007

ATTACHMENT B




                                                                                                     B5
ED's Implementation of FMSS Oracle Federal Financials Phase II and III   Final Report   ED/OIG A11-C0007

ATTACHMENT B




                                                                                                     B6
ED's Implementation of FMSS Oracle Federal Financials Phase II and III   Final Report   ED/OIG A11-C0007

ATTACHMENT B




                                                                                                     B7
ED's Implementation of FMSS Oracle Federal Financials Phase II and III   Final Report   ED/OIG A11-C0007

ATTACHMENT B




                                                                                                     B8